Video recommendations: candidate generation and multi-objective ranking

The two stages each have one job. Candidate generation reduces 50 million eligible videos to about a thousand, in 35 milliseconds, without losing the one the user would have loved. Ranking then gets a thousand candidates, seventy milliseconds, and the freedom to use every feature that retrieval could not afford.

This lesson builds both. The retrieval half is about coverage — several sources, each covering a failure of the others. The ranking half is about precision on a small set — cross features, several predictions per item, and a re-ranking pass for what the model cannot see.

Candidate generation: three approaches, in order of sophistication

Collaborative filtering. "Users who watched what you watched also watched X." Implemented naively as item-item similarity: for each pair of videos, count how many users engaged with both, normalised by their individual popularity. Precompute the top 100 similar videos for each item, and at request time take the union over the user's recent watches.

It requires no model training in the usual sense, it is fully explainable, and it works remarkably well for users with history. It fails completely for new items — an item nobody has co-watched has no neighbours — and it scales poorly, because the item-item matrix is quadratic in catalogue size (though sparsity and truncation make it tractable in practice).

Matrix factorisation. Represent the user-item engagement matrix as the product of two smaller matrices: one row of numbers per user, one per item, so that their dot product approximates engagement. With 400 million users, 50 million items, and 64 dimensions, that is two matrices totalling around 29 billion numbers — large but far smaller than the 20 quadrillion cells of the full matrix.

It learns latent structure — a dimension might end up meaning "cooking content" without anyone naming it — and it produces embeddings, which means approximate nearest neighbour search applies. Its limitation is that it uses only the interaction matrix: it cannot use item content, user demographics, or context, so a brand-new video has no vector at all.

Two-tower neural retrieval. The recommended answer, and the generalisation of matrix factorisation.

• The user tower takes user features (watch history, affinities, demographics) plus context and produces a vector.
• The item tower takes item features (creator, topic, duration, text embedding of the title, thumbnail embedding) and produces a vector in the same space.
• Score is the dot product.

Because the towers are separate, item vectors are precomputed for the whole catalogue and indexed for approximate nearest neighbour search. Because both towers take features rather than only IDs, a brand-new video gets a reasonable vector from its content alone — which matrix factorisation cannot do, and which is the main reason to prefer it.

The cost of the shape, as Step 4: choosing a model said: the towers never see each other, so no user-item cross features are possible. Ranking handles those.

Training the two-tower model

Positives are engaged (user, item) pairs. Negatives are in-batch — every other item in the batch — plus sampled random catalogue items. The loss is a softmax over similarity scores, the same contrastive shape as in the visual search training lesson.

One correction specific to recommendation: sampled softmax with a popularity correction. In-batch negatives are drawn from the engagement distribution, so popular items appear as negatives far more often than random. Without correction, the model over-penalises popular items. Subtracting the log of each item's sampling probability from its score fixes it. This is a specific, checkable detail that demonstrates real familiarity.

Multiple candidate sources

No single source should fill the candidate set. Production systems union several, each covering a different failure:

Union, deduplicate, filter out already-watched — roughly 1,000 unique candidates reach ranking.

The exploration source is a deliberate cost. It will lower short-term engagement and it is the only thing preventing the catalogue collapse described in Step 7: monitoring and the feedback loop. Treat its budget as a product decision, and defend it as such.

Ranking: what changes at this stage

Those thousand candidates now reach the ranker.

The candidate set is small enough that per-item cost rises by four orders of magnitude — from microseconds to a tenth of a millisecond. That budget buys three things retrieval could not have:

1. Cross features. Computed per (user, item) pair: the user's affinity for this creator, how many of this creator's videos they have finished, days since they last watched this topic, whether their friends engaged with this video. These are the highest-signal features in the system and they are quadratic to precompute, so they can only exist once the candidate set is small.
2. A heavier model. A deep network with embeddings and explicit feature interaction, rather than a dot product.
3. Multiple predictions per item. Retrieval produces one score. Ranking can produce a dozen.

The ranking model

A shared trunk with several output heads:

• Input: user features, item features, context features, and cross features, with high-cardinality categoricals passed through embedding tables.
• Trunk: several dense layers producing a shared representation.
• Heads: one per objective — P(watch > 30 s), predicted watch fraction, P(like), P(share), P(subscribe), P(not interested), P(report).

Each head is trained on its own label with its own loss, over the shared trunk. Rare heads — subscribe, report — borrow representation from common ones, which is the multi-task benefit described in the harmful content training lesson.

Gradient-boosted trees remain a legitimate alternative here and would be competitive on the tabular features alone. The reason to go neural is the embeddings: video IDs, creator IDs, and topic IDs have millions of distinct values, and trees handle that badly while embedding tables handle it natively. If the feature set were purely aggregate statistics, trees would likely win — which is exactly the situation in Section 7 (Event Recommendation System), where they do.

Combining the heads into one score

Each item now has seven or eight numbers. The feed needs one.

score = w₁·P(watch>30s) · f(predicted_watch_fraction)      + w₂·P(like) + w₃·P(share) + w₄·P(subscribe)      − w₅·P(not_interested) − w₆·P(report)

Three things about this formula, then the pointer:

• The weights are product policy, not statistics. No dataset says how much one report is worth in units of likes. Someone decides, and the decision is defensible or it is not.
• The negative weights are typically much larger in magnitude than the positive ones, because explicit negative signals are rare and deliberate.
• The watch-fraction transform f is concave, so the difference between 20% and 40% watched matters more than between 80% and 100%. This is the discounting from the metrics lesson.

Section 10 owns this. How the heads share a trunk without negative transfer, how the weights are set and re-tuned, how to handle heads whose predictions live on different scales, and how to know when a weight change has broken something — all of that is in Predicting several things at once and the news feed model lesson. This section uses the formula and points there rather than repeating it.

Calibration, briefly

The heads' outputs are used inside a weighted sum, which means their scales must be comparable. If P(like) is systematically inflated relative to P(share), the weights are silently doing something other than what was intended.

For a pure ranker this would not matter — only the ordering does. Here it does, because combining several probabilities is arithmetic on their values. Check calibration per head, and recalibrate after retraining. Section 8 (Ad Click Prediction) develops calibration properly.

The re-ranking pass

The ranking model optimises expected value per item independently. It has no concept of the slate as a whole, so it will happily return eight videos from the same creator on the same topic, each individually the best available.

Re-ranking applies rules the model does not encode:

Implement diversity greedily: take the top item, then repeatedly take the next-highest item that does not violate a constraint. It costs a few milliseconds and it is the difference between a feed that feels curated and one that feels stuck.