Data analysis with Python using pandas and NumPy

Data analysis with Python is the foundational skill of every data scientist and AI practitioner. Python's ecosystem — particularly pandas and NumPy — makes it possible to load, clean, transform, and analyse datasets of any size with concise, readable code.

This lesson teaches you the core data analysis workflow: from reading raw data files to producing clean, structured datasets ready for visualisation or machine learning.

Essential Python libraries for data analysis

Three libraries form the backbone of Python data analysis:

• NumPy — fast numerical computing with arrays. Handles math operations on large datasets efficiently. Everything else is built on top of NumPy.
• pandas — the workhorse. DataFrames let you work with tabular data (rows and columns) like a spreadsheet, but with the power of Python. Loading CSVs, filtering rows, grouping, merging, pivoting — pandas does it all.
• Jupyter Notebook — an interactive environment where you write code, see results immediately, and document your analysis. The standard tool for exploratory data analysis.

Install all three with: pip install numpy pandas jupyter. You are ready to analyse data in minutes.

The data analysis workflow with Python

A typical data analysis with Python follows these steps:

1. Load data: df = pd.read_csv('data.csv')
2. Inspect: df.head(), df.info(), df.describe()
3. Clean: handle missing values (df.dropna() or df.fillna()), fix data types, remove duplicates
4. Transform: create new columns, apply functions, merge datasets
5. Aggregate: df.groupby('category').mean() — summarise by groups
6. Export: df.to_csv('clean_data.csv') or feed directly into a model

The key insight: data analysis is iterative. You rarely get it right in one pass. Jupyter notebooks let you experiment, backtrack, and refine as you discover the shape of your data.

Practical data analysis techniques

Master these pandas operations and you can handle 90% of real-world data tasks:

• Filtering: df[df['age'] > 30] — select rows matching a condition
• Sorting: df.sort_values('revenue', ascending=False)
• Grouping: df.groupby('region')['sales'].sum()
• Merging: pd.merge(df1, df2, on='id') — join datasets like SQL
• Pivot tables: df.pivot_table(values='sales', index='month', columns='product')
• Apply: df['name'] = df['name'].apply(str.lower) — transform columns

Practice on real datasets from Kaggle or government open data portals. Hands-on repetition is the fastest way to internalise these patterns.

When evaluating data analysis, start with the fundamentals: what problem it solves, who it serves, and how it compares to the alternatives buyers consider first.

Start with the fundamentals. Define who you are serving, the single outcome you want, and the budget you are comfortable committing before you compare any options.

A practical rule is to favour clarity over cleverness. The choice that is easiest to understand, spell, remember and act on tends to win over the long run.

Context changes the answer. What is optimal for a fast-moving startup is rarely optimal for an established brand, so weigh your own stage, market and ambitions.

Avoid the common mistakes: rushing the decision, copying competitors blindly, and ignoring the total cost of ownership beyond the very first year of use.

Think in terms of trade-offs rather than a single perfect answer. Every option trades memorability, cost, reach and flexibility in a slightly different mix.

Document your reasoning. Writing down why you chose one path makes it far easier to revisit the decision later and to explain it to partners and investors.

Test with real people. A quick check with a handful of your actual customers will surface problems that no amount of internal debate ever reveals.

Plan for growth. The option that fits today should still make sense when your audience, catalogue or team is several times larger than it is right now.

Finally, revisit the decision periodically. Markets, tools and best practices move quickly, and a choice that was ideal last year may deserve a fresh look today.

When evaluating data analysis, start with the fundamentals: what problem it solves, who it serves, and how it compares to the alternatives buyers consider first.

Start with the fundamentals. Define who you are serving, the single outcome you want, and the budget you are comfortable committing before you compare any options.

A practical rule is to favour clarity over cleverness. The choice that is easiest to understand, spell, remember and act on tends to win over the long run.

Context changes the answer. What is optimal for a fast-moving startup is rarely optimal for an established brand, so weigh your own stage, market and ambitions.

Avoid the common mistakes: rushing the decision, copying competitors blindly, and ignoring the total cost of ownership beyond the very first year of use.

Think in terms of trade-offs rather than a single perfect answer. Every option trades memorability, cost, reach and flexibility in a slightly different mix.

Document your reasoning. Writing down why you chose one path makes it far easier to revisit the decision later and to explain it to partners and investors.

Test with real people. A quick check with a handful of your actual customers will surface problems that no amount of internal debate ever reveals.

Plan for growth. The option that fits today should still make sense when your audience, catalogue or team is several times larger than it is right now.

Finally, revisit the decision periodically. Markets, tools and best practices move quickly, and a choice that was ideal last year may deserve a fresh look today.

When evaluating data analysis, start with the fundamentals: what problem it solves, who it serves, and how it compares to the alternatives buyers consider first.

Start with the fundamentals. Define who you are serving, the single outcome you want, and the budget you are comfortable committing before you compare any options.

A practical rule is to favour clarity over cleverness. The choice that is easiest to understand, spell, remember and act on tends to win over the long run.

Context changes the answer. What is optimal for a fast-moving startup is rarely optimal for an established brand, so weigh your own stage, market and ambitions.

Avoid the common mistakes: rushing the decision, copying competitors blindly, and ignoring the total cost of ownership beyond the very first year of use.

Frequently asked questions

Do I need to know Python before learning data analysis?

You need basic Python: variables, loops, functions, and lists. If you can write a for-loop and call a function, you know enough to start with pandas.

Is pandas good enough for big data?

Pandas handles millions of rows on a modern laptop. For billions of rows, use Dask (pandas-compatible) or PySpark. Start with pandas — most real-world datasets fit.