Handling Missing & Outlier Data

A hospital dataset has 8,400 patients and a column called blood_pressure. About 12% of it is empty. You do the standard thing — fill the blanks with the column average — and move on.

df["blood_pressure"] = df["blood_pressure"].fillna(df["blood_pressure"].mean())

The model you build afterwards is noticeably worse than expected, and nobody can work out why. Here is why. The readings were missing because those patients were too unwell to sit up for the measurement. Missingness was not random noise; it was the strongest single predictor of poor outcomes in the whole dataset. By filling every blank with 128/82 — a healthy average — you did not recover information. You erased it, and replaced it with a claim that these were the most ordinary patients in the file.

That is the shape of the problem. Missing values and outliers are not blemishes to be scrubbed off before the real analysis. They are data about how the data was produced, and how you handle them decides what your results can possibly mean.

Three Reasons Data Goes Missing

Statisticians split missingness into three mechanisms. The names are ugly but the distinction is the entire lesson, because it determines whether your fix is harmless or catastrophic.

The blood pressure example is MNAR, and MNAR is the one that ruins analyses quietly. You cannot detect it from the data alone — the values you would need to test the hypothesis are exactly the ones you do not have. You detect it by asking someone who knows how the data was collected.

Ask "why is this blank?" before asking "what should I put here?" The first question has a factual answer and the second one does not until you have it.

Seeing the pattern

You can rule MCAR in or out. If missingness in one column correlates with values in another, it is not random.

1import pandas as pd23# How much is missing, and where4miss = pd.DataFrame({5    "missing": df.isna().sum(),6    "pct": (df.isna().mean() * 100).round(1),7}).query("missing > 0").sort_values("pct", ascending=False)8print(miss)910# Do blanks in one column travel with blanks in another?11print(df.isna().corr().round(2))1213# Do rows with a missing value differ from rows without?14flag = df["blood_pressure"].isna()15print(df.groupby(flag)[["age", "days_in_hospital", "num_medications"]].mean())

That last comparison is the important one. If patients with a missing reading average 71 years old and 9 days in hospital, while those with a reading average 54 years and 3 days, the missingness carries information. You have just proved it is not MCAR.

Deletion: When Throwing Rows Away Is Right

Deletion is honest. It makes no claims about values you do not have. Its cost is sample size and, if the missingness is not MCAR, bias — because the rows you delete are systematically different from the ones you keep.

1# Drop rows missing anything at all - usually too aggressive2df.dropna()34# Drop rows only where a genuinely essential field is absent5df.dropna(subset=["patient_id", "outcome"])67# Drop columns that are mostly empty8df.dropna(axis=1, thresh=int(0.5 * len(df)))

A workable rule of thumb, applied per column rather than globally:

That last row deserves emphasis. When a column is 70% empty, replacing it with a single binary column had_value often outperforms the imputed original, because the binary column is true information and the imputed values are mostly fabrication.

Imputation: Filling Blanks Without Lying Too Much

Mean, median and mode

The simplest fills. Median for skewed numeric data, mean for roughly symmetric numeric data, mode for categorical.

df["age"] = df["age"].fillna(df["age"].median())df["city"] = df["city"].fillna(df["city"].mode()[0])

Understand precisely what this does to your data. Suppose ages are 24, 31, 38, 45, 52 with three blanks. The median is 38, so you now have 24, 31, 38, 38, 38, 38, 45, 52. The mean does not move — but the standard deviation drops from 11.1 to 8.4, because you added three values with zero deviation. Every correlation involving age is now weaker than the truth, and any confidence interval you compute is too narrow.

Simple imputation always shrinks variance and weakens correlations. It buys you complete rows by making the data look more certain than it is.

Group-based imputation

Usually a strict improvement, and cheap. Instead of one global median, use the median within a meaningful group.

1df["salary"] = df.groupby(["department", "seniority"])["salary"].transform(2    lambda s: s.fillna(s.median())3)4# Some groups may be entirely empty - catch the leftovers5df["salary"] = df["salary"].fillna(df["salary"].median())

If graduate salaries centre on £28,000 and director salaries on £95,000, a global median of £41,000 is wrong for everyone. The group median is wrong for nobody in particular. That fallback line matters: a group with no observed values at all yields NaN from the median, and the blanks survive unnoticed.

KNN imputation

Find the rows most similar to the one with the gap, across all the other columns, and average their values for the missing field.

1from sklearn.impute import KNNImputer23num_cols = ["age", "bmi", "blood_pressure", "cholesterol"]4imputer = KNNImputer(n_neighbors=5, weights="distance")5df[num_cols] = imputer.fit_transform(df[num_cols])

KNN works on distance, so scale matters enormously. If income ranges to 100,000 and bmi to 40, income dominates the distance calculation completely and BMI is effectively ignored. Standardise before imputing, or the neighbours you find are not the similar rows — they are the rows with similar incomes.

Iterative imputation (MICE)

The most principled of the common methods. It treats each column with gaps as a regression target predicted from all the others, and cycles round repeatedly until the estimates stabilise.

1from sklearn.experimental import enable_iterative_imputer  # noqa: F4012from sklearn.impute import IterativeImputer34imputer = IterativeImputer(max_iter=10, random_state=42)5df[num_cols] = imputer.fit_transform(df[num_cols])

It preserves relationships between variables far better than column-wise fills, because it uses them. It is slower, and it assumes the missingness is MAR — that everything needed to predict the blank is present in the other columns.

Keeping the fact of the gap

Whatever you fill with, record that you filled. One extra column, and it costs nothing:

for col in ["blood_pressure", "cholesterol"]:    df[f"{col}_was_missing"] = df[col].isna().astype(int)# ... then impute

In the hospital example this single flag would have rescued the analysis. The imputed reading is a fiction; blood_pressure_was_missing = 1 is a fact, and it is the fact that predicts the outcome.

Finding Outliers

An outlier is a value far from the rest. That is a statistical statement, not a verdict — it says nothing about whether the value is wrong. A billionaire in an income survey is a genuine outlier; a person recorded as 340 years old is a data-entry error. Both are far from the mean. Only one should be removed.

Z-score

Measures how many standard deviations a value sits from the mean:

1import numpy as np23z = (df["income"] - df["income"].mean()) / df["income"].std()4outliers = df[z.abs() > 3]

The catch is circular: mean and standard deviation are themselves dragged by the outliers you are hunting. Take the values 10, 12, 11, 13, 12, 1000. The mean is 176 and the standard deviation is 403, so the z-score of 1000 is only 2.04 — under the threshold of 3. The outlier hid itself by inflating the yardstick. Z-scores also assume a roughly bell-shaped distribution; on skewed data they flag the entire long tail.

IQR — the robust default

Uses quartiles, which do not move when extreme values change.

1def iqr_bounds(s, k=1.5):2    q1, q3 = s.quantile(0.25), s.quantile(0.75)3    iqr = q3 - q14    return q1 - k * iqr, q3 + k * iqr56lo, hi = iqr_bounds(df["income"])7mask = (df["income"] < lo) | (df["income"] > hi)8print(f"{mask.sum()} outliers, bounds [{lo:,.0f}, {hi:,.0f}]")

On the 10, 12, 11, 13, 12, 1000 example: Q1 = 11.25, Q3 = 12.75, IQR = 1.5, upper bound = 15. The value 1000 is caught immediately. Raise k to 3.0 when you only want the truly extreme.

Multivariate outliers

Some points are unremarkable in every column and absurd in combination. Age 12 is normal. Income £180,000 is normal. Together they are a data error, and no single-column method will find it. Two model-based detectors handle this:

1from sklearn.ensemble import IsolationForest2from sklearn.neighbors import LocalOutlierFactor34X = df[["age", "income", "spend"]].dropna()56iso = IsolationForest(contamination=0.02, random_state=42)7df.loc[X.index, "iso_outlier"] = iso.fit_predict(X) == -189lof = LocalOutlierFactor(n_neighbors=20, contamination=0.02)10df.loc[X.index, "lof_outlier"] = lof.fit_predict(X) == -1

Isolation Forest works by randomly splitting the feature space and counting how few splits it takes to isolate a point — odd points get isolated fast. Local Outlier Factor compares a point's local density with its neighbours', so it catches points that are unusual for their own region even if they sit inside the global range.

Deciding What to Do About Them

Once found, four options. Choosing between them requires knowing why the value is extreme.

1# Cap at the 1st and 99th percentiles2lo, hi = df["income"].quantile([0.01, 0.99])3df["income_capped"] = df["income"].clip(lo, hi)45# Log transform for right-skewed positive data (log1p handles zeros)6df["income_log"] = np.log1p(df["income"])78# Keep, but let the model know9df["is_high_income"] = (df["income"] > hi).astype(int)

The decisive question is what the outlier represents. In fraud detection, credit risk, network intrusion or equipment failure, the outliers are the target — removing them removes the entire signal you were hired to find. That mistake is made surprisingly often, because "clean the data" is treated as a step that happens before anyone asks what the analysis is for.

Never remove an outlier you cannot explain. If you can say "this is a decimal-point error in the price field", delete it. If all you can say is "it is far from the mean", you are deleting evidence.

A Cleaning Pass You Can Defend

Put the decisions in one place, log every change, and compare before with after.

1def clean(df, numeric_cols, flag_cols=(), cap_at=(0.01, 0.99)):2    log = {"rows_in": len(df)}3    out = df.copy()45    # 1. Record missingness before touching anything6    for col in flag_cols:7        out[f"{col}_was_missing"] = out[col].isna().astype(int)89    # 2. Drop rows missing a field the analysis cannot proceed without10    out = out.dropna(subset=["patient_id", "outcome"])11    log["rows_dropped_essential"] = log["rows_in"] - len(out)1213    # 3. Impute numerics with the group median, then a global fallback14    for col in numeric_cols:15        n_before = out[col].isna().sum()16        out[col] = out.groupby("department")[col].transform(17            lambda s: s.fillna(s.median())18        )19        out[col] = out[col].fillna(out[col].median())20        log[f"{col}_imputed"] = int(n_before)2122    # 4. Cap rather than delete extreme values23    for col in numeric_cols:24        lo, hi = out[col].quantile(list(cap_at))25        n_capped = int(((out[col] < lo) | (out[col] > hi)).sum())26        out[col] = out[col].clip(lo, hi)27        log[f"{col}_capped"] = n_capped2829    log["rows_out"] = len(out)30    return out, log313233clean_df, log = clean(df, ["age", "bmi", "blood_pressure"],34                      flag_cols=["blood_pressure"])35for k, v in log.items():36    print(f"{k:32s} {v:,}")3738comparison = pd.DataFrame({39    "before_mean": df[["age", "bmi"]].mean(),40    "after_mean":  clean_df[["age", "bmi"]].mean(),41    "before_std":  df[["age", "bmi"]].std(),42    "after_std":   clean_df[["age", "bmi"]].std(),43}).round(2)44print(comparison)

Watch the standard deviation column in that comparison. A drop of a few percent is the expected cost of imputation. A drop of 30% means you have flattened the data, and every relationship you measure afterwards will look weaker than it truly is.

What This Means When You Build Something

Fit your imputation on training data only. Every method here learns parameters — a median, a set of neighbours, a regression — and if those are learned from the full dataset, information from your test rows leaks into your training rows and your validation score becomes optimistic fiction. In scikit-learn this means putting the imputer inside a Pipeline:

1from sklearn.pipeline import Pipeline2from sklearn.impute import SimpleImputer3from sklearn.preprocessing import StandardScaler4from sklearn.ensemble import RandomForestClassifier56pipe = Pipeline([7    ("impute", SimpleImputer(strategy="median")),8    ("scale", StandardScaler()),9    ("model", RandomForestClassifier(random_state=42)),10])11pipe.fit(X_train, y_train)   # medians computed from X_train alone

And keep a written record of what you did and why: which columns were imputed, by what method, how many outliers were capped, which rows were dropped and on what grounds. When someone asks in six months why the average is what it is, that record is the difference between an answer and a shrug. The handling of missing and extreme values is a chain of judgement calls, and judgement calls are only defensible when they are written down.