Ad click prediction: calibration, metrics and data at scale

Prompt: "Design the click prediction system for ads on a social platform."

Every other problem in this course produces a ranking. This one produces a number that sets a price, and that single difference reorganises the whole design.

This lesson follows that difference through the first three steps. The framing becomes calibrated classification; the metrics split into two families that must never be confused; and the data — three billion labelled examples a day, a 1.4% positive rate, and labels that arrive days late — needs sampling that silently breaks calibration unless you correct for it.

The clarifying questions

1. What is the auction mechanism? Second-price, first-price, or something else? What does the advertiser bid on — an impression, a click, or a conversion? This determines whether the click probability is used as a ranking key or as a multiplier on money.
2. What is the latency budget? Ads are scored inside a page load that is already happening. Expect tens of milliseconds, not hundreds — a far tighter constraint than any other case study here.
3. How many candidate ads per request? After targeting filters, hundreds to a few thousand.
4. What is the objective — clicks, conversions, or advertiser value? Conversions are what advertisers want and their labels arrive days late (see Data at scale).
5. What is the volume? It sets the training infrastructure and the retraining cadence.

Note the relationship to the ad click event aggregation case study in System Design Interview, which designed the ad click event aggregation pipeline — counting clicks correctly for billing. That lesson solved the accounting; this one solves the prediction. They meet at the impression log.

The constraints we design against

Invented. The platform is Cobalt, the social network from Section 5.

Framing it as machine learning

• Input: a (user, ad, context) triple. User features, ad and advertiser features, page context, and their crosses.
• Output: P(click) — a calibrated probability between 0 and 1.
• Objective: minimise log loss, so the predicted probabilities are correct in value, not only in order.

Task type: binary classification producing a calibrated probability.

Why calibration, and not ranking

This is the framing decision that defines this case study, and it deserves to be made explicit and early.

In a typical second-price auction ranked by expected value, each candidate ad is scored by:

expected value per thousand impressions  =  bid × P(click) × 1000

The winner pays a price derived from the runner-up's expected value divided by the winner's predicted click probability. So P(click) appears in the denominator of what the advertiser is charged. It is not a sorting key. It is an input to arithmetic about money.

Three concrete consequences of miscalibration:

• Over-prediction inflates expected value across the board. Ads win auctions they should have lost, advertisers are charged more than the impression was worth, and delivery over-runs budgets.
• Under-prediction loses auctions to competing demand, under-delivers advertiser campaigns, and leaves platform revenue on the table.
• Differential miscalibration — accurate for one ad category and inflated for another — distorts the marketplace, systematically advantaging some advertisers. Harder to detect and the most damaging.

This is why the headline offline metric is log loss, not ROC-AUC. Log loss is a proper scoring rule: it is minimised only by predicting the true probability, so it penalises a confident wrong value in a way an ordering metric cannot.

The baseline first

Rung 0: historical click-through rate per (ad creative, placement) pair, smoothed toward the campaign average, then toward the global average, in proportion to how much data exists. No model, no features beyond identity. It is calibrated by construction, since it is an observed frequency, and it will beat a badly-built model.

Its failure is that it cannot generalise. A brand-new creative has no history, and the smoothing falls back to a global average that ignores everything about who is being shown it.

Rung 1: logistic regression on a moderate set of features with hand-built crosses. Fast, calibrated by construction because log loss is its native objective, and genuinely competitive at this task. The model lesson is honest about how competitive.

Metrics: discrimination versus calibration

Two families of metric, measuring two different things, and confusing them is the failure this step exists to prevent.

Discrimination is whether the model separates clicks from non-clicks — whether clicked impressions score higher than unclicked ones. Measured by ROC-AUC.

Calibration is whether the predicted values are numerically right. Measured by log loss and by the calibration plot.

A model can be excellent at one and poor at the other. That is not a corner case here; it is the routine result of the negative downsampling in Data at scale, which improves training and systematically inflates every probability.

Normalised entropy is worth defining because it is the number this industry actually reports: the model's log loss divided by the log loss of a model that always predicts the overall base rate. A value of 1.0 means the model is no better than predicting the average; lower is better. It normalises away shifts in the base click rate, so numbers stay comparable week to week as traffic mix changes — which raw log loss does not.

The calibration plot

The diagnostic that reveals what no single number does.

Bucket predictions by predicted probability — 0.00–0.01, 0.01–0.02, and so on, or by decile. For each bucket, plot the mean predicted probability on the x-axis against the observed click rate on the y-axis. A perfectly calibrated model traces the diagonal.

Reading the shapes:

• Above the diagonal — the model under-predicts. Observed clicks exceed predictions.
• Below the diagonal — over-prediction. The most common failure after downsampling.
• S-shaped — over-confident at both extremes. Predictions are too close to 0 and 1.
• Diagonal in the low buckets, diverging in the high ones — good on typical traffic, wrong exactly where the money is, since high-probability impressions are the valuable ones.

Always produce this plot segmented: by ad category, by advertiser size, by placement, by country, by device, and by user tenure. A globally calibrated model can be badly calibrated in every segment, with the errors cancelling in aggregate. The segmented version is where real problems appear. The monitoring lesson shows the plot with a drifted curve alongside a healthy one.

Online metrics

The predicted-to-actual ratio is the cheapest and most valuable production metric in this system. Sum the predicted click probabilities over an hour of traffic; divide by the actual click count. A well-calibrated model gives 1.00. Any drift shows within an hour and needs no labelled evaluation set.

The guard metrics are not decoration

An ads system optimising revenue alone will increase ad load, favour attention-grabbing creative, and push toward placements that interrupt. Each raises short-term revenue and degrades the product that generates the impressions.

Ad hide rate, report rate, and organic session length are release gates for the same reason that "not interested" rate is one in Section 6. The long-run measure is advertiser retention and user retention together, and neither shows up in a two-week experiment.

Data at scale

Three billion labelled examples a day, a 1.4% positive rate, and labels that arrive days after the prediction. Each of those creates a specific problem.

The volume, and what to do with it

3 billion impressions a day is roughly 1.1 trillion a year. Nobody trains on all of it, and nobody needs to.

• Positives are precious. At 1.4%, a day yields about 42 million clicks. Keep all of them.
• Negatives are abundant and nearly free of information individually. Sample them.
• Recency matters enormously. A model trained on last week beats one trained on last quarter in this domain, because creatives, campaigns, and user interests turn over fast. Weight recent data heavily or restrict the window entirely.

A workable training set: all positives from the last 30 days — about 1.26 billion — plus negatives downsampled 1 in 10, about 8.9 billion, giving roughly 10 billion rows rather than 90 billion.

Negative downsampling and the correction it requires

Downsampling makes training tractable and makes the model's output probabilities wrong. The correction is a formula, it is exact, and forgetting it is one of the most common serious mistakes in this problem.

Keep negatives with probability w (say w = 0.1, a 1-in-10 sample). The training data now has ten times the true positive rate. The model's output p' is on that inflated scale.

Recover the true probability with:

p = p' / ( p' + (1 − p') / w )

A worked example. Suppose w = 0.1 and the model outputs p' = 0.12 on the downsampled scale:

p = 0.12 / ( 0.12 + 0.88 / 0.1 )  = 0.12 / ( 0.12 + 8.8 )  = 0.12 / 8.92  = 0.0135

The true probability is about 1.35%, not 12%. Without the correction the model over-predicts by roughly nine times, every auction is mispriced, and — critically — ROC-AUC is completely unchanged, because the transformation is monotonic. Every ranking metric on the dashboard looks perfect while the system overcharges every advertiser.

Delayed conversion labels

Clicks arrive in seconds. Conversions — a purchase, a signup, an install — arrive over hours or days, with a reporting window of up to 7 days.

That creates a genuine problem. An impression from two hours ago that has not converted might be a negative, or it might be a positive whose conversion has not happened yet. Treating it as negative teaches the model that recent impressions do not convert, which is false and which biases the model against exactly the traffic it is currently seeing.

Four approaches, and naming the trade-off is what matters:

1. Wait for the full window. Correct, and it makes the freshest 7 days of data unusable. For a domain where recency is everything, that is a large cost.
2. Model the delay. Train a second model predicting the conversion delay distribution, and use it to weight uncertain negatives: an impression 2 hours old with a typical delay of 3 days is weakly negative, not strongly negative. More complex and it recovers the recent data.
3. Two-stage labelling. Train on clicks (fast labels) for the recency-sensitive component and on conversions (slow labels) for a correction layer updated less often.
4. Importance weighting with a fallback. Use a short window (say 24 hours) with a weight correcting for the known fraction of conversions that arrive later, estimated from the historical delay distribution.

The recommendation for a first system: option 1 for the conversion model, option 3 overall — predict clicks with fresh data, correct to conversions with a slower model. Then invest in option 2 if conversion optimisation becomes the primary product.

Selection bias

You only observe clicks on ads you showed. Ads that lost every auction generate no data, so the model never learns whether they would have performed. New advertisers are systematically disadvantaged, which is both a modelling problem and a marketplace problem — a platform where new advertisers cannot get started loses its supply of new demand.

Mitigations: log the auction score and win probability so offline evaluation can be reweighted; give new ads a guaranteed exploration allocation (see Monitoring and follow-ups); and use an optimism bonus for high-uncertainty ads so they get shown enough to be measured.