Video search: serving, caching and monitoring

The full request path has to accommodate two retrievers and a ranking model in 300 milliseconds. This lesson builds that path and its latency budget, then turns to what happens after launch.

Search degrades in ways users notice long before dashboards do, because the failures live in the tail. Fresh uploads, spam and evaluation cost are the three problems that follow you into production, and they are also where most of the follow-up questions come from.

The path

1. Query understanding — normalise, spell-correct, detect language, classify intent, and compute the query embedding. Some of this is cached; head queries hit a cache almost always.
2. Parallel retrieval — lexical and semantic fired simultaneously, each returning ~500.
3. Merge and deduplicate — union of both sets, roughly 800 unique videos.
4. Filter — remove blocked, deleted, region-restricted, and age-gated content. Cheap predicate checks, done before anything expensive.
5. Feature hydration — fetch features for the surviving candidates from the online feature store in one batched call.
6. Ranking — score all candidates with the ranking model.
7. Re-ranking — diversity (not five videos from one channel), freshness boost for time-sensitive queries, and spam demotion.
8. Response — top 20 with thumbnails and metadata.

The latency budget

Invented, for a 300 ms p99 target.

Two things to say about this table in an interview. Retrieval runs in parallel so it costs the max, not the sum — that is worth 25 ms and it demonstrates you are reading the path rather than adding a column. And feature hydration for 800 candidates rivals the model itself, which is why the filter step comes before it: every candidate removed early is a feature fetch saved.

The ranking model

Over ~800 candidates with a 70 ms budget, a gradient-boosted tree ensemble or a moderately sized neural network both fit. Features fall into four groups:

• Query features — length, language, intent class, frequency band.
• Video features — age, duration, channel authority, historical engagement rate, quality signals.
• Query–video match features — BM25 score, semantic score, title match fraction, transcript match count, whether the channel name matches the query. These carry most of the signal.
• Context features — device, time of day, country, and (if personalised) the user's watch history affinity with this channel and topic.

Note that the match features can only be computed after retrieval, for this specific query–video pair. That is precisely what the retrieval stage cannot afford, and it is the concrete reason the two-stage split exists.

Caching

Search caches better than most problems in this course, because query distribution is heavily skewed — a small number of head queries account for a large share of traffic.

Personalisation and caching pull against each other: a fully personalised page cannot be shared between users. The usual resolution is to cache the candidate set and the non-personalised ranking, then apply a light personalised re-rank on top. That keeps most of the cache benefit and most of the personalisation benefit.

Short time-to-live values matter for freshness. A 15-minute cache on a query about breaking news means results are up to 15 minutes stale, which may be unacceptable — so make the time-to-live depend on the query's detected time-sensitivity.

What to monitor

Once the path is live, the question becomes how you find out it is failing — and where.

The tail is where search quality actually lives, and it is invisible in aggregates. Sample tail queries deliberately for human review — a few hundred a week, rated by the panel — as a standing quality check.

Fresh and trending content

The hardest structural problem in video search. A video uploaded 90 seconds ago about a major news event has:

• No engagement history, so every engagement-derived feature is zero or missing.
• No transcript yet, if speech recognition is queued.
• No semantic vector yet, if encoding is queued.

Yet it may be the single most relevant result for thousands of queries right now.

The mitigations, worth naming as a set:

1. Lexical retrieval is the freshness path. Title and description are indexed within a minute. Dense retrieval cannot match that, which is a second, independent reason to run both.
2. Query time-sensitivity detection. Classify whether a query is about something happening now — by comparing current query volume against its historical baseline. A query whose volume spikes 50× in an hour is about something new. Apply a strong recency boost for such queries and none for evergreen ones.
3. Prior-based features. For a brand-new video, substitute channel-level historical engagement for the missing video-level features. A channel that reliably produces well-watched videos is evidence about its newest one.
4. Exploration. Give new videos a small guaranteed impression allocation so they generate the data that makes them rankable at all.

Spam, clickbait, and manipulation

Search is adversarial. Creators optimise for it, and some optimise dishonestly:

• Keyword stuffing in descriptions and tags. Cap the contribution of the description field, and weight title and transcript above it.
• Misleading titles and thumbnails. Detected by the gap between click-through rate and post-click watch time. High clicks with very short watches is the clickbait signature.
• Engagement manipulation. Purchased views and likes. Detected by anomalous engagement patterns rather than by the search system itself, but the signal must reach ranking.
• Content mismatch. The video does not contain what the title claims. The transcript is the check — if the title says "harissa chicken" and the transcript never mentions either word, the match is suspect.

Handle these as ranking demotions, not removals, except where content policy applies. A demotion is reversible and errs toward leaving legitimate content findable; a removal is not. That preference for the reversible action under uncertainty is the same principle Section 5 develops for moderation.

Evaluating without an expensive panel

Human relevance panels are the honest measure and cost around 1,000 rater-hours per full evaluation. Four cheaper approaches worth naming:

1. Interleaving. Mix results from two rankers into a single list and see which ranker's results get clicked more. Far more sensitive than an A/B test — it detects differences with roughly an order of magnitude less traffic, because the comparison is within-user rather than between-user.
2. Counterfactual evaluation from logs. With logged propensities, estimate how a new ranker would have performed on past traffic. Cheap, and it only works where the new ranker's choices overlap the old one's.
3. Reformulation as a label. If users searching X frequently reformulate to Y and then click, the videos clicked after Y are relevant to X. This mines relevance labels from failures at no cost.
4. A small standing panel on the tail only. Spend the human budget where automated signals are weakest.