Mini Project: Exploratory Data Analysis on Kaggle Dataset

Two people are given the same dataset and a week. The first produces a notebook with forty charts: every column histogrammed, every pair scattered, a correlation heatmap with sixty rows. It is thorough, it is technically correct, and nobody reads past the third cell.

The second produces six charts and one sentence: "Customers who contact support twice in their first month churn at three times the base rate, and we can identify them by day 30." That sentence is worth something. Somebody can act on it.

The gap between those two notebooks is not skill with Pandas. Both used the same functions. The difference is that the second person started with a question and treated every chart as an attempt to answer it, while the first mistook coverage for analysis. This project is your chance to practise the second habit deliberately, on data nobody has pre-cleaned for you.

What you are building

A single notebook that takes a real, messy public dataset and ends with a defensible finding. The deliverable is not "an exploration". It is a document that a reader who has never seen the data can follow from question to evidence to conclusion in ten minutes.

Choosing a dataset

The dataset choice determines how interesting your project can possibly be. Spend twenty minutes on it rather than five.

That last row is the one people underrate. If you cannot tell whether a house price of £12,000 is a typo, a garage sale or a different currency, you cannot clean the column — you can only guess. Familiarity with the subject is what turns a rule into a judgement.

Avoid the famous teaching datasets. Iris, Titanic and the built-in housing sets have been cleaned and analysed thousands of times; there is nothing left to find, and a reader recognises them instantly. Housing sales, retail transactions, flight delays, energy consumption, football matches, air quality, bike hire and public health records all work well.

Write the question before you write any code

A good question names a specific outcome and a plausible driver, and could turn out either way.

Write your question at the top of the notebook and leave it there. Every chart that does not help answer it or a sub-question of it should be deleted before you submit. This is the discipline that turns forty charts into six.

Stage 1: first contact with the data

1import pandas as pd, numpy as np2import matplotlib.pyplot as plt, seaborn as sns34sns.set_theme(style="whitegrid", context="notebook")5pd.set_option("display.max_columns", 60)67df = pd.read_csv("data/raw/dataset.csv")89audit = pd.DataFrame({10    "dtype": df.dtypes,11    "missing_pct": (df.isna().mean() * 100).round(1),12    "n_unique": df.nunique(),13    "example": df.iloc[0],14})15print(df.shape)16print(audit.sort_values("missing_pct", ascending=False))17print("exact duplicate rows:", df.duplicated().sum())

Read that audit table hunting for four specific things: a numeric column stored as text, str or object (something non-numeric is hiding in it), a column where n_unique equals the row count (an identifier, useless for analysis), a column where n_unique is 1 (constant, drop it), and anything above 60% missing (you cannot fill your way out of that).

Then read twenty actual rows. Not head() — df.sample(20), so you see the middle of the file rather than whatever happened to be sorted to the top. Almost every dataset has a surprise that only shows up when a human looks at raw values: a category that changes spelling halfway through, a date column with two different formats, a numeric column where 12% of values are exactly zero.

Stage 2: clean, and record what you did

The cleaning code is the easy half. The half that matters is the log, because a cleaning decision without a stated reason is indistinguishable from a mistake.

1log = []2original_n = len(df)34def note(action, reason, before, after):5    log.append({"action": action, "reason": reason,6                "rows_before": before, "rows_after": after,7                "rows_lost": before - after})89n = len(df)10df = df.drop_duplicates()11note("drop exact duplicates", "same record ingested twice", n, len(df))1213# convert to numbers first: a range filter on text either fails or compares wrongly14df["price"] = pd.to_numeric(df.price.astype(str).str.replace(r"[£$,]", "", regex=True),15                            errors="coerce")1617n = len(df)18df = df[df.price.between(1000, 5_000_000)]19note("filter price to 1k–5m", "values below 1k are placeholder zeros; above 5m are data errors", n, len(df))2021df["city"] = df.city.str.strip().str.title()22df["built_year"] = df.built_year.fillna(df.groupby("district").built_year.transform("median"))2324print(pd.DataFrame(log))25print(f"retained {len(df)/original_n:.1%} of original rows")

That final percentage is your safety check. If cleaning removed more than about 15% of your rows, stop and find out which rule did it. Losing half your data to a single filter is not cleaning, it is a change of dataset, and any finding afterwards describes only the rows that survived.

Every row you drop is a claim that those rows do not belong in your analysis. Be able to defend each claim, and know how many rows it cost.

Stage 3: one variable at a time

1num = df.select_dtypes(np.number)2stats = num.describe().T3stats["skew"] = num.skew()4stats["zeros_pct"] = (num == 0).mean() * 1005print(stats.round(2))67cols = [c for c in num.columns if c != "id"][:6]8fig, axes = plt.subplots(2, 3, figsize=(14, 7))9for ax, col in zip(axes.ravel(), cols):10    sns.histplot(df[col].dropna(), bins=40, kde=True, ax=ax)11    ax.set_title(col, fontsize=10)12fig.tight_layout()1314for col in df.select_dtypes(include=["object", "str"]).columns:15    print(f"\n{col}: {df[col].nunique()} levels")16    print(df[col].value_counts(dropna=False).head(6))

You are looking for three shapes, and each one is a lead worth following. A second hump means two populations are mixed together, and there is probably a column that separates them. A spike at a single value means a default or placeholder is masquerading as data. A hard wall at a round number means truncation — a cap on what the collection system would record.

Stage 4: relationships

1corr = num.corr()2mask = np.triu(np.ones_like(corr, dtype=bool))3fig, ax = plt.subplots(figsize=(10, 8))4sns.heatmap(corr, mask=mask, cmap="coolwarm", center=0, vmin=-1, vmax=1,5            annot=True, fmt=".2f", ax=ax)67# the pairs worth investigating, listed rather than eyeballed8pairs = corr.where(~mask).stack().dropna().sort_values(key=abs, ascending=False)9print(pairs.head(10))1011print(df.groupby("district")["price"].agg(["count", "median", "std"])12        .sort_values("median", ascending=False))1314print(pd.crosstab(df.property_type, df.has_garden, normalize="index").round(3))

Two cautions worth building into your habits here. Pearson correlation only detects straight-line relationships — a perfect U-shaped relationship reports approximately zero — so scatter the pairs that matter rather than trusting the heatmap alone. And always print the group count next to the group mean: a district with 6 sales and a striking median is not a finding, it is a small sample.

Then run the check that separates a real analysis from a superficial one — does your headline hold inside every subgroup?

1overall = df.groupby("near_station").price.median()2print(overall)34by_segment = df.groupby(["property_type", "near_station"]).price.median().unstack()5print(by_segment)

If the overall relationship is positive but reverses inside every property type, the aggregate is being driven by which property types happen to sit near stations, not by the stations. That reversal is one of the most interesting things you can find, and it is invisible unless you deliberately look.

Stage 5: state the finding, then try to break it

Write your findings as numbered statements, each carrying a number and a caveat:

1. Flats within 500m of a station sell for a median £412/sq m against £361 for   those further away — a 14% premium (n = 3,204 and 5,881).2. The premium does not exist for houses (£388 vs £391, n = 1,102 and 4,455).   It appears to be specific to flats.3. 31% of listings have no floor area recorded, and those listings have a 22%   lower median price. Area is not missing at random, so any per-square-metre   figure describes only the listings that reported it.

Then attack your own conclusion before someone else does. Does it survive if you remove the top 1% of prices? Does it hold in each year separately, or only in the year with the most rows? Is there a third variable — district wealth, property age — that would explain both sides of the relationship? A finding that survives three honest attempts to break it is worth reporting; one you never tested is a hypothesis wearing a conclusion's clothes.

A quick baseline model is a useful final check, not because you need a model but because it tells you whether your chosen variable carries independent signal:

1from sklearn.model_selection import cross_val_score2from sklearn.ensemble import RandomForestRegressor3from sklearn.inspection import permutation_importance45X = pd.get_dummies(df[features], drop_first=True)6y = df.price78model = RandomForestRegressor(n_estimators=200, random_state=42)9print(cross_val_score(model, X, y, cv=5, scoring="r2").mean().round(3))1011model.fit(X, y)12perm = permutation_importance(model, X, y, n_repeats=10, random_state=42)13print(pd.Series(perm.importances_mean, index=X.columns)14        .sort_values(ascending=False).head(10).round(4))

If the variable at the centre of your finding ranks near the bottom of permutation importance, other variables explain the same thing better, and your finding may be a shadow of theirs.

How this will be judged

Two things reliably separate strong submissions from adequate ones, and neither is technical. The first is that the strong ones report something that surprised them and explain how they checked it — evidence of actual investigation rather than a template applied. The second is that they say what the data cannot tell you. A limitations section is not an admission of weakness; it is the clearest possible signal that you understand your own analysis.

Before you submit, restart the kernel and run the notebook top to bottom on a clean session. Code that works only because of a variable you defined an hour ago and then deleted is code that does not work, and it is the single most common reason a submitted analysis cannot be reproduced by the person reading it.