Model, training, serving and monitoring: steps 4 to 7 of the framework

The correct first model is almost always embarrassingly simple, and saying so is what separates a strong candidate from a well-read one. This lesson covers the second half of the framework: choosing the model, training it, serving it inside a latency budget, and noticing when it quietly stops working.

The steps are uneven in weight. Training is the shortest step in the framework and still has four decisions worth stating. Serving carries the rest of the course: six of the ten case studies serve predictions with the same two-stage shape drawn here. And monitoring is the step almost everyone runs out of time for, even though it is scored.

Step 4: choosing a model

The baseline ladder

Climb it in order, and state out loud what each rung buys.

Rung 0 — a heuristic, no model. Most popular items in the last 24 hours. Most recent posts. Items from creators the user follows. This is not a placeholder; it is the number every model must beat, and on some problems it is genuinely hard to beat. It also gives you a working end-to-end system in a day, which means the data pipeline, the serving path, and the metrics all exist before any modelling starts.

Rung 1 — logistic regression. Fast to train, fast to serve, interpretable coefficients, well-calibrated probabilities out of the box. With hand-built cross features it remains competitive on very large sparse problems, which is why Section 8 (Ad Click Prediction) treats it as a serious option rather than a warm-up.

Rung 2 — gradient-boosted decision trees. For tabular features of moderate dimension, this is the default winner. It handles mixed types and missing values, needs almost no feature scaling, is robust to outliers, and trains on a single machine for most realistic dataset sizes. Sections 7 and 11 recommend it as the final answer, not the baseline.

Rung 3 — neural networks. Necessary when raw unstructured input has to be understood (images, video, audio, free text), when very high-cardinality categorical features need learned embeddings, when several tasks should share a representation, or when the serving architecture requires embeddings — which is what makes retrieval possible at all.

The decision table

Two-tower models, explained once

The pattern appears in Sections 2, 4, 6 and 9, so here is the plain version.

A two-tower model has two separate networks. One encodes the query side — a user, or a search query. The other encodes the item side — a video, a product, a listing. Each produces a vector of the same length. The score is the dot product of the two vectors.

Why the shape matters: because the towers never look at each other, item vectors can be computed in advance for the whole catalogue and stored in an index. At request time you compute one user vector and find its nearest neighbours. That is how ten million items get reduced to a few hundred in under 30 milliseconds.

The cost of the shape: because the towers never look at each other, the model cannot learn interactions between a specific user feature and a specific item feature. "This user watches cooking videos, and this is a cooking video" has to be captured through the geometry of the space rather than as an explicit feature crossing. That is precisely what the second-stage ranking model is for, and it is why every large recommender has two stages rather than one.

Step 5: training

Training has four decisions in it worth stating: the loss, the batch, the search, and the split.

Choosing the loss

The loss encodes what "wrong" means. Match it to the output shape.

The interview-relevant point is not memorising the table. It is being able to say why: "log loss, because the downstream auction consumes the probability itself, so I need it calibrated and not merely correctly ordered."

Batching and distributed training, briefly

Batch size trades gradient noise against hardware efficiency. Larger batches give smoother gradients and better hardware utilisation, but past a point they generalise slightly worse and need a scaled-up learning rate with a warm-up period.

Distributed training in one paragraph: data parallelism puts a copy of the model on each worker, splits the batch across them, and averages the gradients — this is what you use, and it works until the model no longer fits on one device. Model parallelism splits the model itself across devices and is only needed for very large models. For large embedding tables the common arrangement is a hybrid: the embedding table is sharded across parameter servers, while the dense layers are replicated. Two sentences on this is the right depth unless the interviewer digs.

Hyperparameter search

Random search beats grid search for the same budget, because most hyperparameters do not matter much and grid search spends its budget varying them anyway. Bayesian methods beat both when each training run is expensive. Say which you would use and why the budget makes it the right call.

The split that matters here: split by time

This is the part to get right.

Random train/test splitting is standard practice everywhere else and is wrong for almost every problem in this course. If your data spans January to June and you split at random, the training set contains June and the test set contains January. The model is being tested on the past using knowledge of the future.

A worked example. Nimbus trains a watch-prediction model on six months of logs, splitting at random. Offline ROC-AUC: 0.88. Deployed, it reaches 0.79 in the first week and drifts down from there.

The cause: a large fraction of the signal is per-video popularity, and with a random split a video's own future views are in the training set while some of its past views sit in test. The model has memorised which specific videos did well — an entirely useless skill for tomorrow's uploads.

The fix: train on January–April, validate on May, test on June. The gap between random-split and time-split scores is itself informative. If it is large, your model depends on item-specific memorisation and will degrade quickly between retrainings.

Related refinements worth one sentence each: for a problem with strong geography effects, hold out an entire region rather than a random sample (Section 3). For a problem where a single user contributes thousands of rows, split by user so the same user is never in both sets.

Step 6: serving

This is the step that carries the rest of the course. Six of the ten case studies serve predictions with the same two-stage shape, and the diagram below is the one they re-skin.

Batch or online

Decide this first, because everything downstream depends on it.

Hybrids are the norm and worth saying out loud: precompute embeddings and heavy aggregates in batch, compute the final score online. Section 9 precomputes nearest neighbours entirely. Section 8 must score inside an auction and cannot precompute anything.

The two-stage pattern

The problem: a user needs 20 items chosen from 10 million, in under 200 milliseconds. Scoring 10 million items with a good model is impossible — at even 0.1 milliseconds each that is 1,000 seconds.

The resolution is to use two models of different cost at different scales.

• Stage 1 — retrieval (candidate generation). A cheap model reduces 10,000,000 to about 1,000. It must have high recall — losing the right item here means no later stage can recover it — and it uses few features, because it runs against everything. Implemented as approximate nearest neighbour search over precomputed item embeddings, plus non-model sources like "recently uploaded by creators you follow".
• Stage 2 — ranking. An expensive model scores those ~1,000 candidates (or a filtered ~500) with rich features, including cross features between the user and each item. It optimises ordering quality, measured with nDCG.
• Stage 3 — re-ranking. Business rules the model does not encode: remove already-seen items, enforce diversity so five items from one creator do not fill the slate, demote integrity-flagged content, apply freshness boosts.

The cost changes by roughly four orders of magnitude between stages, which is what makes the budget work.

The components under the diagram

Approximate nearest neighbour (ANN) search. Finding the 1,000 vectors closest to a query vector among 10 million, without comparing against all 10 million. It trades a little accuracy for a large speed-up — you might retrieve 95% of the true top-1,000 in 8 milliseconds instead of 100% in 400 milliseconds. The visual search serving lesson compares the methods properly.

Embedding store / vector index. Holds item vectors and serves the ANN queries. Rebuilt or incrementally updated as the catalogue changes; the rebuild cadence is a real design decision because a nightly rebuild means new items are invisible for up to a day.

Feature store (online). The low-latency half of the feature store from the features and leakage lesson. One lookup for user features, one batched lookup for candidate features. The batched candidate lookup is usually the largest single slice of the latency budget.

Model server. Loads a versioned model and answers scoring requests, usually with batching across candidates and often on accelerated hardware.

The latency budget

Numbers below are invented for a 300 ms p99 target and are the shape to reproduce, not values to memorise.

Building the table live in the interview is the point. When the interviewer says "now use a model that takes 300 ms", you can answer from the table: it does not fit, so either the candidate set shrinks, the model is compressed or distilled, or the scoring moves off the request path into a precomputed cache.

Step 7: monitoring and the feedback loop

Once the model is serving, the danger changes. Models do not fail loudly. They keep returning confident predictions that are increasingly wrong, and nothing in the system objects.

Two kinds of drift

Concrete versions, both invented:

• Data drift. Nimbus launches in a new country. The share of non-English titles goes from 4% to 22% in three weeks. The model's text features were fitted almost entirely on English. Nothing about "what makes a video watchable" changed; the inputs did.
• Concept drift. A shopping app's click model was trained through November. In January, the same "50% off" badge on the same product predicts far fewer clicks, because post-holiday users are price-insensitive browsers rather than gift buyers. The inputs look identical. The meaning changed.

The distinction matters because the fixes differ. Data drift is often fixed by retraining on recent data or by broadening the feature preparation. Concept drift needs retraining and usually a shorter retraining cadence, because it will happen again.

Detecting silent degradation

Monitor in four layers, cheapest and fastest first.

1. Input distribution. Per-feature mean, variance, null rate, and cardinality against a reference window. Population stability index or a Kullback–Leibler divergence gives a single number per feature. Alert on the feature, not on the aggregate — one broken feature is invisible in an average.
2. Prediction distribution. The histogram of scores the model produces. If the mean predicted click probability moves from 0.031 to 0.048 overnight with no product change, something upstream broke. This is the fastest real signal you have, because it needs no labels.
3. Delayed label metrics. Once outcomes arrive — minutes for clicks, days for conversions, never for some things — compute the real metrics on recent traffic.
4. Calibration. For any model whose probability is consumed as a number, plot predicted against observed rates by bucket. Calibration usually drifts before ranking quality does, which makes it an early warning. The ad click monitoring lesson treats this as the primary alarm.

Also monitor the boring things that cause most incidents: feature freshness (is the online store still being written to?), model version in production, null rates, and inference latency.

Retraining

Three triggers, and you should name which one you would use:

• Scheduled — daily, weekly. Predictable, easy to operate, sometimes wasteful.
• Triggered by drift — retrain when a monitor crosses a threshold. Efficient, but needs the monitoring to be trustworthy first.
• Continuous / near-real-time — the model updates from a stream. Necessary where trends move in hours (Section 8), operationally demanding everywhere else.

Deploy through shadow mode — the new model scores live traffic and its output is logged but not used, so you compare against production on identical inputs — then a canary at 1–5% of traffic with automatic rollback on guard metrics, then a ramp. Keep the previous model loaded and make rollback a configuration change, not a deployment.

The feedback loop

This is the failure nobody sees coming.

The model's output becomes the data that trains its replacement. Nimbus's ranker puts video A at position 1 and video B at position 30. A gets 40× the impressions. Tomorrow's training data shows A with far more engagement — not because A is better, but because A was shown. The next model ranks A higher still.

Within a few retraining cycles the catalogue has collapsed onto whatever the first model happened to favour. Good items that were never shown have no engagement history and stay invisible. The offline metrics improve throughout, because the model is getting better at predicting a world it created.

The mitigations, and you should name at least two:

• Log the propensity — the probability with which each item was shown — so offline evaluation can be reweighted to undo the selection.
• Keep a randomised holdout. A small slice of traffic gets randomly ordered or randomly injected candidates. Expensive in short-term engagement, and the only source of unbiased data you will have.
• Explore deliberately. Give new and under-shown items a boost, or use a bandit policy that values uncertainty. The video recommendation monitoring lesson covers exploration versus exploitation.
• Hold out a population. Keep a small user group on an older or simpler model as a long-run reference for what the system is doing to behaviour over months.