Video recommendations: serving, feedback loops and exploration

This lesson covers the request path, its latency budget, and the caching decisions that make 180,000 feed requests per second affordable. Most of the savings come from one principle: compute in advance anything that does not depend on the current request.

It then turns to what happens after launch. The failure modes of a recommender are slow, self-reinforcing, and invisible in the metrics the system optimises, which is why they dominate the follow-up questions.

What is precomputed and what is live

The design principle: compute in advance anything that does not depend on the current request.

Why user embeddings can be stale

This is the design decision worth defending in detail, because the instinct is to compute the user vector on every request.

A user's long-term interests move slowly. Someone who watched cooking, cycling, and history videos for six months will still be interested in those in four hours. Recomputing that vector on every request is expensive — a forward pass through the user tower plus fetching the full history — and buys almost nothing.

What does change fast is the session: what they watched in the last ten minutes, what they skipped, what they searched for. That is real-time signal and it must not be stale.

The resolution is to split the representation:

• Slow user vector, from long-term history, recomputed every 4–6 hours in batch and cached. This is what feeds the ANN retrieval.
• Fast session vector, from the current session's events, computed live and cheaply — an average of the embeddings of the last few items engaged with.
• Combined at request time, typically by concatenation or a weighted sum, before the ANN query.

The cost saving is substantial: 400 million users refreshed every 5 hours is about 22,000 embedding computations per second in batch, against 180,000 per second if done live — an eight-fold reduction, and the batch version uses hardware far more efficiently.

Add an on-demand refresh trigger: if a user's behaviour in this session diverges sharply from their cached vector, recompute. That catches the case the staleness argument misses — a user whose interests genuinely changed today.

The latency budget

Invented, against a 250 ms p99 target.

Two points to make about this table. The six candidate sources run in parallel, so the cost is the slowest, not the sum — and the slowest is the ANN query. And feature hydration for 1,000 candidates is comparable to the ranking model itself, which is why every filter that can run before hydration does.

Degradation

Feed requests must not fail. Define what happens when each component is slow or down:

• ANN index unavailable → serve from the other five candidate sources. Quality drops, the feed still fills.
• Ranking model timing out → return candidates ordered by the retrieval score. Noticeably worse, still coherent.
• Feature store slow → hydrate a reduced feature set with defaults for the rest, and use a model trained to tolerate missing features.
• Everything down → serve a cached trending list by country and language.

Each fallback is a step down in personalisation and never a blank page. Stating the degradation ladder is a strong wrap-up move, and it directly reuses the reliability patterns from System Design Interview.

Caching

• Candidate sets per user for 5–10 minutes, so infinite-scroll pages 2 and 3 reuse the retrieval work. This is the highest-value cache in the system, because a scrolling session makes several requests.
• Item features, aggressively — they are shared across all users.
• Never cache the final ranked list across requests, because it must exclude what the user has already been shown in this session.

The feedback loop, concretely

Serving is the easy half of running this system. The hard half is that its failures are slow, self-reinforcing, and invisible in the metrics it optimises.

Step 7: monitoring and the feedback loop described the mechanism. Here is what it looks like on this system.

Day 1: the ranker slightly prefers cooking videos for a user who watched two. Day 7: three of their twenty daily slots are cooking. Day 30: twelve are. Day 90: the user's watch history is overwhelmingly cooking, the model's confidence is very high, and the user has not been offered anything else in weeks.

Nothing broke. Each step was locally correct — the model predicted engagement well and was right. The system has narrowed a person's exposure to a fraction of what they might have liked, and the metrics say it is doing an excellent job, because within the narrowed set the predictions are accurate.

The same mechanism runs at catalogue scale. Videos that get shown accumulate engagement data, which makes the model confident about them, which gets them shown more. Videos that were never shown stay unshown regardless of quality.

Measuring it

You cannot fix what you do not measure, and the standard metrics are blind to this.

Set targets on these, not only on engagement. A weekly catalogue coverage floor is a concrete, enforceable constraint.

Exploration versus exploitation

The formal frame for the fix.

Exploitation is showing what the model believes is best. Exploration is showing something uncertain to find out. Pure exploitation is optimal for the next impression and terrible over months, because the model stops learning about anything it has stopped showing.

Practical mechanisms, in order of increasing sophistication:

1. Fixed exploration slots. Reserve 1–2 of every 20 positions for under-shown items. Crude, effective, trivially implementable, and the right first answer.
2. Epsilon-greedy. With small probability, replace a slot with a random eligible item.
3. Upper confidence bound. Score items by predicted value plus a bonus proportional to the model's uncertainty about them. Principled — items get shown until the system knows whether they are good — and it needs a usable uncertainty estimate, which neural models do not give for free.
4. Thompson sampling. Sample from the posterior over each item's value and rank by the sample. Elegant, and it needs the same uncertainty machinery.

Start with fixed slots. The measurable cost is a small short-term engagement drop; the benefit is a catalogue that stays alive and a model that keeps learning.

Cold start

New users have no history. Options in order: use context (country, language, device, and the referring content if they arrived from a link); ask directly during onboarding, which is cheap and surprisingly effective; serve popularity-by-demographic; and adapt fast — the first three or four engagements should move the feed noticeably, which means the session vector from the serving lesson must be weighted heavily when the long-term vector is empty.

New videos have no engagement history. The two-tower item vector comes from content features, so they are retrievable immediately, which is the main advantage over matrix factorisation. For ranking, substitute creator-level priors for missing item-level statistics — a creator whose videos are reliably well-watched is evidence about their newest one. And give new items a guaranteed impression allocation so the data exists to rank them properly. Without that guarantee, a new video's ranking features stay empty forever.

Engagement, wellbeing, and what the design must include

The metrics lesson established that engagement metrics can rise while the product gets worse. That is not a monitoring afterthought; it produces concrete requirements in the running system:

• A satisfaction survey running continuously on a sampled slice, reported alongside engagement at every release review.
• Explicit negative signals as a release gate, with a stated threshold: a model that raises "not interested" rate beyond a set margin does not ship regardless of watch time.
• Session-length discounting in the score, so the system does not profit from a user's inability to stop.
• A long-run holdout population on a satisfaction-weighted model, because no two-week A/B test can measure a three-month effect.
• User controls that actually feed back — "not interested", "don't recommend this creator", topic preferences — wired into both training and re-ranking. A control that does nothing is worse than no control.

These are design requirements with implementation cost, and treating them as such is what the interview is testing.