Ad click prediction: sparse features and the model ladder

The feature space here is the largest and sparsest in the course: tens of millions of distinct identifiers, and the interactions between them carry most of the signal. How you represent those identifiers, and how you compute the counting features without leaking the label, matters more than the model on top.

The model half of this lesson is the honest progression, including the part where the simple models stay competitive far longer than anyone expects.

The cardinality problem

One-hot encoding user ID alone would require a billion-column vector. Two techniques make this tractable, and they compose.

Feature hashing

Embeddings for high-cardinality features

After hashing, each bucket index maps to a learned vector — typically 8 to 32 dimensions for ad and user identifiers. Similar advertisers drift toward similar vectors, so an advertiser with few impressions borrows structure from similar ones.

Embedding tables dominate the model's parameter count. A hashed user table of 2²⁴ rows at 16 dimensions is 268 million parameters; the dense layers on top might be under a million. That imbalance drives the training architecture: shard the embedding tables across parameter servers and replicate the dense layers, as noted in Step 5: training.

Cross features

The signal is in the interactions. "This user" and "this ad category" each carry modest information; "this user and this ad category" carries a lot.

• Explicit crosses. Concatenate two feature values into one new categorical, then hash it: (user_region × ad_category), (device_type × placement), (hour_of_day × ad_category). These are what make logistic regression competitive at this task, and constructing the right ones is skilled manual work.
• Learned interactions. A neural model with embeddings can learn interactions through explicit interaction layers — factorisation-machine style dot products between every pair of embeddings, or attention over feature fields. This is the main thing a deep model buys over logistic regression here.

Counting features, and the leakage trap

Counting features are the most predictive family available and the most dangerous:

• Click-through rate of this creative over the last 1 hour, 24 hours, 7 days.
• Click-through rate of this (user, advertiser) pair.
• Number of times this user has seen this creative today.
• Click-through rate of this advertiser in this placement.

They are strong because they summarise exactly what the model is trying to predict. That is also why they leak.

The trap, concretely: a training pipeline computes creative_ctr_24h with a nightly batch job over calendar days. For an impression at 09:00 on Tuesday, the "last 24 hours" window as computed by that job runs to the end of Tuesday — so it includes the click or non-click of that very impression, plus every other impression of that creative later that day.

On a low-volume creative with 40 impressions, the impression being predicted is 2.5% of its own feature. The model learns to read the answer out of the feature. An invented but very plausible result: offline ROC-AUC of 0.87 against 0.74 online, and nobody can reproduce the offline number.

The fixes:

• Point-in-time computation. Every counting feature is computed strictly from events before the impression timestamp. This is what a feature store's offline store with valid-from timestamps exists for.
• Explicit lag. Compute the feature as of one hour before the impression, so a small pipeline error cannot cross the boundary.
• Exclude self. Where a per-entity aggregate is unavoidable, subtract the current impression's own contribution.
• The smell test. If a feature's importance is far higher than anything else and offline metrics jumped after adding it, assume leakage before assuming success.

The model: rung 1, logistic regression with crosses

With the features in place, climb the ladder honestly.

A linear model over hashed sparse features and hand-built crosses.

Why it works so well here. The feature space is enormous and sparse, and with billions of training rows a linear model over hundreds of millions of sparse features has plenty of capacity — capacity in a sparse linear model comes from the number of features, not from depth. Log loss is its native objective, so it is calibrated by construction. It trains in a single pass with online learning, so it can be updated continuously. Its coefficients are inspectable. Serving is a sparse dot product measured in microseconds.

Its limit. It cannot discover interactions. Every cross must be constructed by hand, and with hundreds of feature fields the useful crosses number in the thousands. Finding them is slow skilled work that never finishes.

Rung 2: gradient-boosted trees

Strong on the dense, aggregate part of the feature space — counting features, rates, ratios, context. It finds interactions automatically, exactly as in Section 7.

Its limit here. Trees handle very high-cardinality categoricals poorly. A tree cannot usefully split on a 60-million-value creative ID, and one-hot encoding it is impossible. So trees are excellent on half of this feature space and unusable on the other half.

A historically successful hybrid worth naming: use the trees as a feature transformer. Train a tree ensemble on the dense features, then treat the leaf each example falls into as a categorical feature fed to the logistic regression alongside the sparse features. The trees discover the dense interactions; the linear model handles the sparse identities. This is a real production pattern and a strong thing to describe.

Rung 3: deep models with embeddings

A neural network with embedding tables for the sparse identifiers, an explicit interaction component, and dense layers on top.

What it buys:

• Embeddings mean sparse identifiers generalise. An advertiser with 200 impressions borrows from similar advertisers rather than being a coefficient fitted on 200 rows.
• Interaction layers discover crosses that nobody hand-built, across all feature pairs.
• One model can carry several heads — click, conversion, engagement — sharing a representation (see the news feed model lesson).
• Continuous training over a stream is natural.

What it costs:

• Serving cost rises by an order of magnitude or more against a sparse linear model, against a 20 ms budget for 800 candidates.
• Calibration is no longer free. Deep models trained with log loss are often over-confident, and an explicit recalibration step is usually required.
• Training infrastructure is substantially heavier: sharded embedding tables, parameter servers, and a much longer path from idea to measured result.

The honest note

The gap between a well-engineered logistic regression with good crosses and a modern deep model on this task is real but modest — typically a few percent of relative log loss improvement, not a transformation. What produces large gains in click prediction is, in rough order of impact:

1. Better and fresher features, especially counting features computed correctly.
2. More recent training data and a faster retraining cadence.
3. Correct calibration.
4. Model architecture.

A candidate who says "I'd start with logistic regression with crosses, get the counting features and the calibration right, and move to a deep model when feature engineering stops paying" is describing how these systems are actually built. A candidate who opens with an architecture and never mentions calibration or feature freshness is not.

Training

• Loss: log loss, weighted for downsampling, with the correction from Data at scale applied to outputs.
• Optimiser: an adaptive per-parameter method, because sparse features have wildly different update frequencies — a rare creative ID is updated a handful of times while a placement ID is updated billions of times, and a single global learning rate serves neither.
• Regularisation: L2 on the dense layers; L1 on sparse features to drive unused ones to zero and keep the model small.
• Split by time, always, with a gap. Random splitting leaks in this domain more severely than almost anywhere else, because the same creative appears thousands of times a day and a random split puts its afternoon impressions in training and its morning ones in test.