Working with Datasets: Sources, Cleaning & Feature

The difference between a toy project and a real AI system is the data. Working with datasets — finding them, cleaning them, and engineering features from them — is where data scientists spend most of their time and where the biggest performance gains come from.

This lesson teaches you the practical dataset skills that separate beginners from professionals.

Working: Where to find quality datasets

Free, high-quality datasets are everywhere:

Kaggle — the largest dataset repository. Thousands of datasets with community notebooks showing analysis approaches.
Hugging Face Datasets — primarily NLP and multimodal datasets. Easy to load: from datasets import load_dataset
UCI Machine Learning Repository — classic ML benchmark datasets used in academic papers
Google Dataset Search — search engine specifically for datasets
Government open data — data.gov (US), data.gov.uk (UK), and equivalents worldwide
APIs — Twitter, Reddit, financial data (Yahoo Finance), weather data. Build custom datasets from live sources.

Start with Kaggle: it has datasets for every skill level, competitions to test your models, and community solutions to learn from.

Data cleaning essentials

Real data is always messy. Master these cleaning operations:

Handle missing values — drop rows, fill with mean/median/mode, or use model-based imputation
Fix data types — dates stored as strings, numbers stored as text, categorical encoding
Remove duplicates — df.drop_duplicates() — check for near-duplicates too
Handle outliers — investigate, do not blindly remove. Use IQR method or z-scores.
Validate ranges — ages cannot be negative, percentages cannot exceed 100
Standardise formats — consistent date formats, country names, currency codes

Create a data cleaning checklist and run it on every new dataset. Systematic cleaning prevents bugs that surface much later in the pipeline.

Feature engineering for better models

Feature engineering creates new input variables that help the model learn:

Date features — extract year, month, day of week, hour, is_weekend from timestamps
Text features — word count, sentence count, average word length, sentiment score
Interaction features — multiply or divide related columns (price_per_sqft = price / area)
Binning — convert continuous values to categories (age groups: 18-25, 26-35, etc.)
Target encoding — replace categorical values with their mean target value (careful with leakage)
Lag features — for time series: previous day's value, 7-day rolling average

Feature engineering is where domain knowledge shines. A feature that captures real-world meaning (revenue_per_employee, days_since_last_purchase) often improves models more than algorithm tuning.

When evaluating working, start with the fundamentals: what problem it solves, who it serves, and how it compares to the alternatives buyers consider first.

Start with the fundamentals. Define who you are serving, the single outcome you want, and the budget you are comfortable committing before you compare any options.

A practical rule is to favour clarity over cleverness. The choice that is easiest to understand, spell, remember and act on tends to win over the long run.

Context changes the answer. What is optimal for a fast-moving startup is rarely optimal for an established brand, so weigh your own stage, market and ambitions.

Avoid the common mistakes: rushing the decision, copying competitors blindly, and ignoring the total cost of ownership beyond the very first year of use.

Think in terms of trade-offs rather than a single perfect answer. Every option trades memorability, cost, reach and flexibility in a slightly different mix.

Document your reasoning. Writing down why you chose one path makes it far easier to revisit the decision later and to explain it to partners and investors.

Test with real people. A quick check with a handful of your actual customers will surface problems that no amount of internal debate ever reveals.

Plan for growth. The option that fits today should still make sense when your audience, catalogue or team is several times larger than it is right now.

Finally, revisit the decision periodically. Markets, tools and best practices move quickly, and a choice that was ideal last year may deserve a fresh look today.