📖 Chapter 2 of 13 · AI for New Learner- Senior 2 · Class 10
Chapter 2: From Idea to Dataset
In this chapter
- Saira from Lucknow — how a real teen data scientist thinks.
- Where to find FREE Indian datasets (data.gov.in, Kaggle, AI Kosh).
- Pandas in 12 lines — read, clean, explore data.
- What ‘data cleaning’ actually means — missing values, outliers, wrong types.
- EDA: Exploratory Data Analysis — looking before you leap.
The Big Idea
Real ML engineers spend 70% of their time on data — finding it, cleaning it, understanding it. Only 30% on the model itself. The Indian AI community has gifted you something incredible: open datasets on EVERY topic — agriculture yields, election results, hospital records, language corpora — all FREE at data.gov.in. This chapter teaches you how to hunt and prepare them.
Did you know?
data.gov.in has over 5 LAKH free datasets, all released by Indian government departments. The AI Kosh portal under IndiaAI Mission adds curated ML-ready datasets for Indian researchers. Both are free and need no registration for download.
Builder Profile: Saira
Lucknow, UP · age 14
Saira wanted to build an AI that warns farmers near her grandfather’s village in Barabanki about wheat pest outbreaks. She knew the IDEA. She did not have the DATA. Most teens stop here. Saira did not.
She opened data.gov.in. Searched ‘agriculture pest’. Found a UP state dataset of pest reports for 2018-2023. 12,000 rows. CSV format. Free. She opened it in Google Sheets — half the values were missing, dates were in three formats, and the column ‘pest_severity’ had typos like ‘high’ and ‘Hi’ and ‘HIGH’ mixed together.
Most teens would have given up. Saira wrote a 15-line pandas script that fixed everything in 2 minutes. Standardised dates. Filled missing values with the column mean. Made all severities lowercase. By the end of one Saturday, she had clean data ready to train a model. “The hardest part of ML,” she now tells her juniors, “is BEFORE the model. Master pandas first.”
📊 DATASET HUNT Top 4 Indian Dataset Sources
- Open Govt Data (data.gov.in) — Govt of India open data — 5+ lakh datasets across agriculture, health, elections, transport.
- AI Kosh (indiaai.in) — ML-ready datasets curated by the IndiaAI Mission: faces, languages, traffic, health.
- Kaggle Indian (kaggle.com) — Search ‘India’ for community datasets on Bollywood, IPL, COVID-19, mandi prices.
- UCI ML Repository (archive.ics.uci.edu) — Global classic ML datasets — Iris, Wine, Adult Income — used worldwide.
💻 CODE LAB Pandas — Read, Clean, Explore
import pandas as pd
# 1. READ
df = pd.read_csv('crops.csv') # any CSV file
print(df.shape) # rows, columns
print(df.head()) # first 5 rows
print(df.info()) # types + missing counts
# 2. CLEAN
df = df.dropna() # drop missing rows
df['severity'] = df['severity'].str.lower() # standardise text
df['date'] = pd.to_datetime(df['date']) # fix dates
# 3. EXPLORE
print(df.describe()) # mean, min, max
print(df.groupby('region').size()) # rows per region
⚠ FAILURE ANALYSIS
Problem: df = pd.read_csv('data.csv') gave UnicodeDecodeError.
Why: Your CSV has Hindi/Tamil text in some cells. Default pandas reads as UTF-8, which fails on some Indian-language encodings.
Fix: df = pd.read_csv('data.csv', encoding='utf-8-sig'). Or try encoding='latin-1' if that fails.
Solved illustrations
1. What does EDA mean?
Show answer
Answer: Exploratory Data Analysis. Before you train any model, you LOOK at your data — shape, types, distributions, missing values, outliers. Pandas .describe() and .info() are your starting tools. Five minutes of EDA saves hours of debugging later.
2. What is a ‘CSV’ file?
Show answer
Answer: Comma-Separated Values. A text file where each line is one row and commas separate columns. Excel can save and open them. They are the most common ML data format. Most datasets on data.gov.in are CSV.
3. Why drop missing values?
Show answer
Answer: Most ML models cannot handle NaN (Not a Number) cells. Two strategies: (a) drop those rows with df.dropna() or (b) fill them with df.fillna(df.mean()). Choose carefully — if 50% of rows are missing, dropping kills your dataset.
4. What is an ‘outlier’?
Show answer
Answer: A value far outside the normal range. E.g. a wheat yield of 50,000 kg per acre in a column where most are 50-200. Could be a real anomaly OR a data-entry typo. Always investigate outliers before training — they pull means and skew models.
5. Is web-scraping data legal?
Show answer
Answer: It depends. data.gov.in, Kaggle, UCI are all free to use. Scraping social media or paid sites can violate Terms of Service and Indian IT laws. Stick to officially-released open datasets. India’s DPDP Act 2023 protects user data — respect it.
Questions — Chapter 2
A. Code reading
- What does df.shape return? Why is it useful?
- What is the difference between df.head() and df.tail()?
- df.dropna() vs df.fillna(0) — when do you use each?
- What does df.groupby('city').size() do?
- Write 1 line to read ‘mandi_prices.csv’ into pandas.
B. Short answers
- What is EDA? Why do real engineers do it first?
- Name 4 sources of free Indian datasets.
- List 3 things that make a dataset ‘dirty’.
- What does the DPDP Act 2023 say about scraping personal data?
- Pick a problem you care about. Name a dataset you could use. (Be specific.)
Ship-It Challenge
Download ONE real Indian dataset this week. Read it in pandas. Save the screenshot.
Check your understanding — Chapter 2
Tap an answer to check it instantly. These are for practice — nothing is saved or sent.
Q1. What does EDA stand for and why do it first?
- Extra Data Adding
- Exploratory Data Analysis — you LOOK at your data (shape, types, missing values, outliers) before training anything
- Every Day Answers
Show answer
B. Exploratory Data Analysis — you LOOK at your data (shape, types, missing values, outliers) before training anything
Q2. Why do we drop or fill missing values before training?
- It looks tidier
- Most ML models cannot handle NaN cells; you either drop those rows or fill them — but choose carefully if many rows are missing
- Missing values are illegal
Show answer
B. Most ML models cannot handle NaN cells; you either drop those rows or fill them — but choose carefully if many rows are missing
Q3. Is web-scraping data always legal in India?
- Yes, anything online is free
- No — official open data (data.gov.in, Kaggle, UCI) is fine, but scraping social media or personal data can violate ToS and the DPDP Act
- Scraping is never allowed
Show answer
B. No — official open data (data.gov.in, Kaggle, UCI) is fine, but scraping social media or personal data can violate ToS and the DPDP Act
Q4. You get a UnicodeDecodeError reading a CSV with Hindi text. What's the fix?
- Delete the Hindi rows
- Read it with encoding='utf-8-sig' (or try 'latin-1') so pandas decodes the Indian-language characters
- Convert the file to an image
Show answer
B. Read it with encoding='utf-8-sig' (or try 'latin-1') so pandas decodes the Indian-language characters