Introduction to Data Wrangling

Summary

Much of the world's data cannot be easily processed in Excel or other spreadsheets. It is either too inaccessible, too messy, too unstructured, too varied, or too large for elementary handling. This course is designed to equip students with core tools to start down the path towards becoming data scientists. We cover the basic data structures and file formats, and then move into classes and SQL. Basics of Pandas, NumPy, and matplotlib follow from that. We introduce the fundamental building blocks of data manipulation. We also demonstrate how to translate simple Excel commands into more powerful languages like SQL and Python, as well as how to build on the existing open-source libraries.

Associated project work

Students solve several simple programming exercises in Python, to demonstrate the proficiency for more advanced courses.

Students will gain experience with Python-based data wrangling technologies to extract insights from a structured, web-API-based dataset. Students will learn the fundamental building blocks of data extraction, manipulation, and aggregation via Pandas DataFrames and good Python programming practice.

Python

Jupyter notebooks and the kernel Variables Functions Logic and program flow Iteration Whitespace Matters Putting it all together

Python Built-in Data Structures

Python list Python dict (a.k.a. dictionaries) Python set Python tuple Comprehensions

Dealing with Strings in Python

The string data structure Unicode and Byte Strings Basic string processing StringIO in Python Regular expressions

Object-oriented programming

Everything is an Object Defining a Python Class Adding Attributes and Methods Inheritance Putting it all together...

Functions

Python functions basics Functions as first-class objects Recursion Closures Decorators

Exceptions

Catching general exceptions Handling success Doing something with the error Raising errors Exceptions and the call stack Reading traceback

Debugging

Errors and Exceptions NameError TypeError AttributeError KeyError Reading code critically

Python Data Formats

File Reading and Writing in more detail Reading and Writing, or more precisely, Reading OR Writing, but not Both Gzip Files Pickle Files JSON Files CSV Files XML Files

Pandas

Nouns (objects) in Pandas Verbs (operations) in Pandas Loading data Indexing and slicing data frames Indices in Pandas Adding and dropping data Filtering data Joining data Sorting by indices and columns Data transformations and function application String operations and transformations Pandas HTML data import example Pandas Timestamps DataFrames and Iterators

Data Aggregation in Pandas

Basic statistics and visualization Aggregating data by group DataFrame Resampling

SQL - Structured Query Language

SELECT - Getting information from the tables COUNT, SUM, and DISTINCT - Let SQL do work for you! WHERE, LIKE, and IN - Filtering the data ORDER BY - Sorting your outputs GROUP BY - Aggregating data JOIN - Putting tables together Conclusion

NumPy

NumPy Indexing Saving NumPy Data Files Relationship with Pandas Matplotlib Matplotlib and Pyplot Matplotlib plots from Pandas

Prerequisites

Elementary programming knowledge

The Data Incubator

Introduction to Data Wrangling

Summary

Associated project work