📖 Chapter 7 of 12 · AI for New Learner- Senior 1 · Class 9
Chapter 7: Working with Real Indian Data
In this chapter
- Pari from Coimbatore — the data cleaner.
- Pandas — your superpower for working with CSV files.
- Where to find FREE real Indian datasets (data.gov.in, mandi prices, IndiaAI).
- Cleaning messy data — missing values, wrong types, outliers.
- Why ‘data work’ is 80% of every real AI job.
The Big Idea
Most beginners think AI is about clever algorithms. Real engineers know: AI is about DATA. A simple model with great data beats a fancy model with messy data — every single time. This chapter teaches you the unglamorous skill that pays the bills at every Indian AI company: reading, cleaning, and understanding real CSV files.
Did you know?
India’s data.gov.in portal has over 6 LAKH free datasets covering everything from mandi prices to rainfall to airline delays. The Open Government Data Platform is one of the world’s largest — all free for students to download and build AI on.
Builder Profile: Pari
Coimbatore, Tamil Nadu · Pari, age 14
Pari lives in Coimbatore. Her father runs a small textile workshop. Last year she decided to build an AI that predicts which day of the week has the highest power-cut risk — so her dad can plan production. Sounds easy.
She found the data on the Tamil Nadu Electricity Board site. 10,000 rows. She opened it in Python and froze. Half the rows had MISSING values. Some had ‘NA’, some ‘NULL’, some nothing. Dates were in three formats. Some power values were in MW, some in kW. The data was a mess.
She spent two weeks just CLEANING it. Then five minutes building the model. The model achieved 76% — useful enough that her father changed his production schedule. “Most of my work,” Pari says, “is scrubbing data. Like a doctor washing hands. Boring. Critical. Repeat daily.”
Pandas — your data sidekick
Five commands handle 80% of data work: pd.read_csv('file.csv') loads a CSV into a DataFrame; df.head() shows the first 5 rows; df.describe() gives mean, min, max and count for every numeric column; df.dropna() drops rows with missing values; df.fillna(0) fills missing values with 0 (or any value).
💻 CODE LAB Reading Indian mandi prices
import pandas as pd
# Read the CSV (downloaded from data.gov.in)
df = pd.read_csv('mandi_prices.csv')
# Peek at the first 5 rows
print(df.head())
# Summary statistics
print(df.describe())
# How many rows? How many missing?
print('Rows:', len(df))
print('Missing:', df.isna().sum().sum())
WHY THIS MATTERS
Most beginners are stuck because they can’t find data. Now you have NO excuse — 6 lakh free Indian datasets are one click away on data.gov.in, indiaai.gov.in, Kaggle, Bhashini Open Data, the UCI ML Repository and Hugging Face. Pick ONE that interests you (cricket, music, mandi prices) and download today.
✍ WORKED EXAMPLE Clean a Mandi Price Dataset
Problem: mandi_prices.csv has 5,000 rows. Some prices are missing (‘NA’), some dates are ‘dd-mm-yyyy’ and some ‘yyyy/mm/dd’, and some prices are in rupees, some in paise. Walk through the cleaning steps.
- INSPECT — run df.head() and df.info(). Find missing values, weird formats, types.
- FILL or DROP missing — for a price column, dropping is safer than filling with 0: df = df.dropna(subset=['price']).
- STANDARDISE dates — convert everything to ISO: df['date'] = pd.to_datetime(df['date'], errors='coerce').
- STANDARDISE units — if 50% of values look 100× too big, they’re probably in paise: df.loc[df.price > 10000, 'price'] /= 100.
- OUTLIERS — onion at Rs 50,000/kg? A type error. Drop or flag: df = df[df.price < 5000].
- SAVE the clean version — df.to_csv('mandi_clean.csv', index=False). NEVER overwrite the original.
Show answer
✓ Answer: 5,000 messy rows → about 4,200 clean rows ready for training. Took Pari about 10 hours.
Solved illustrations
1. Why do real AI engineers spend 80% of time on data?
Show answer
Answer: Because the model code is the same 20 lines for everyone — sklearn, pandas — but YOUR data is unique. Cleaning, formatting, joining, validating data is where the real intelligence happens. Garbage data → garbage AI.
2. What is the difference between df.dropna() and df.fillna(0)?
Show answer
Answer: dropna() removes any row with missing values. fillna(0) keeps the row but replaces missing values with 0. For prices, dropping is safer (a missing price filled with 0 corrupts the average). For an ‘optional comments’ column, filling is fine.
3. What’s an ‘outlier’ and how do I spot them?
Show answer
Answer: An outlier is a value far from the others — usually a typo. If 99% of mandi prices are Rs 20-200 and one says Rs 50,000, that’s an outlier. Use df.describe() to see min/max — any extreme value deserves investigation.
4. Why save the CLEAN file separately?
Show answer
Answer: Two reasons. (1) Cleaning takes hours; you don’t want to redo it. (2) NEVER edit the original — if you find a mistake later, you need to go back. Pari’s rule: the raw_data/ folder is read-only forever.
5. Pari spent 2 weeks cleaning and 5 minutes modelling. Is that normal?
Show answer
Answer: Yes — completely. The first project on any new problem is mostly data work. By the second project on the same data you have a clean pipeline and modelling takes hours. By the third, you’re shipping models in a day.
Questions — Chapter 7
A. Pandas commands
- To read a CSV file in pandas, use: pd.('file.csv').
- To see the first 5 rows of df, call: df.().
- To drop rows with missing values: df.().
- India’s largest open-data portal is at .
- Real AI engineers spend roughly % of their time on data work.
B. Data thinking
- Why does dropping missing values sometimes ruin a dataset?
- Name 3 free Indian datasets you could download today.
- What is the FIRST command you run after loading a new CSV?
- Why never edit the original raw data file?
- Pick a problem you care about. Where would you find data for it?
Ship-It Challenge
Download 1 dataset from data.gov.in this week. Load it with pandas. Print df.head(). Done.
Check your understanding — Chapter 7
Tap an answer to check it instantly. These are for practice — nothing is saved or sent.
Q1. Why do real AI engineers spend about 80% of their time on data?
- They are slow
- The model code is the same few lines for everyone, but YOUR data is unique — cleaning it well is where the real work is
- Data is more fun than models
Show answer
B. The model code is the same few lines for everyone, but YOUR data is unique — cleaning it well is where the real work is
Q2. What is the difference between df.dropna() and df.fillna(0)?
- They are identical
- dropna() removes rows with missing values; fillna(0) keeps the row but replaces missing values with 0
- Both delete the whole table
Show answer
B. dropna() removes rows with missing values; fillna(0) keeps the row but replaces missing values with 0
Q3. What is an outlier and how do you spot one?
- A perfect value
- A value far from the others (often a typo) — spot it with df.describe() by checking min/max
- The most common value
Show answer
B. A value far from the others (often a typo) — spot it with df.describe() by checking min/max
Q4. Why should you save the CLEANED file separately and never edit the original?
- To use more disk space
- Cleaning takes hours you don't want to redo, and you must keep the raw data so you can go back if you find a mistake
- There is no reason
Show answer
B. Cleaning takes hours you don't want to redo, and you must keep the raw data so you can go back if you find a mistake