Video search: lexical and semantic retrieval and visual content understanding

There are two ways to find candidate videos for a query. They fail on opposite things, which is why production systems run both. That is the first half of this lesson.

The second half moves past text about videos to the video itself — speech, on-screen text, frames — and to the cost calculation that decides how much of it Nimbus can afford. At 400 hours of uploads a minute, that calculation shapes the design more than any modelling choice.

Lexical retrieval

An inverted index maps each term to the list of documents containing it. The query "leaky tap repair" looks up three short lists and intersects them. Decades of engineering have made this extremely fast — tens of milliseconds over hundreds of millions of documents on modest hardware.

BM25 scores each matching document. It rewards three things:

• Documents where the query term appears more often (with diminishing returns — the tenth occurrence adds far less than the second).
• Rare terms over common ones, so "harissa" counts for far more than "the".
• Shorter documents, on the reasoning that a term in a 6-word title is more central than the same term in a 400-word description.

What it does well: exact matches, rare terms, proper nouns, product names, and anything the user typed because they already know it exists. Fully interpretable — you can point at exactly why a document matched. New videos are searchable the moment they are indexed, with no model involved.

What it cannot do: match "leaky tap" to "dripping faucet". Zero shared terms means zero score. Synonyms, paraphrases, and cross-language matching are all invisible to it.

Semantic retrieval

A two-tower model (Step 4: choosing a model) encodes the query and the video into the same vector space. Relevance is the dot product. Retrieval is approximate nearest neighbour search over precomputed video vectors (Serving: approximate nearest neighbour search).

• Query tower: a text encoder over the query. Small and fast, because it runs on every request — a few milliseconds.
• Video tower: a text encoder over title, description, and transcript, optionally combined with visual features. Large and slow, and that is fine because it runs once per video, offline.

Training data: query–video pairs where the user clicked and watched a substantial fraction, with negatives from in-batch sampling plus mined hard negatives (videos that rank highly but were not engaged with). The loss is contrastive, exactly as in the visual search training lesson.

What it does well: paraphrase, synonym, cross-language, and vague or descriptive queries ("that song from the advert with the horses").

What it cannot do reliably: exact rare terms. A specific product model number or an unusual proper noun may have been seen a handful of times in training, and its vector will be approximately meaningless. Dense retrieval will confidently return something plausible and wrong, and unlike BM25 it cannot tell you why.

Why both, and how to merge

They are complementary, not competing. Run both in parallel — they are independent, so the latency cost is the slower of the two rather than the sum — and merge.

Merging. The two scores are not comparable: BM25 is unbounded and corpus-dependent, cosine similarity sits in a fixed range. Do not add them.

The robust method is reciprocal rank fusion: score each document by the sum over retrievers of 1/(k + rank), with k a small constant around 60. It uses only rank order, so no calibration between systems is needed. A video at rank 3 in lexical and rank 40 in semantic scores 1/63 + 1/100.

The alternative is to skip merging entirely: take the union of both candidate sets, deduplicate, and let the ranking model in the serving lesson sort out the ordering with both retrievers' scores as features. This is usually better, because the ranker has far more information than a fusion formula does. Reciprocal rank fusion is the right answer when the candidate set must be trimmed before ranking for cost reasons.

Visual content understanding

Everything so far has searched text about videos. The rest of this lesson is about searching the video itself, and about the cost calculation that decides how much of it you can afford.

What is inside a video, and what it costs to extract

All figures are invented and used to show relative magnitude and the shape of the decision, not as quotes.

The number that forces the design

Nimbus receives 400 hours of video per minute. That is:

• 24,000 hours per hour
• 576,000 hours per day

Transcribe everything at £0.15 per hour: £86,400 a day, about £31 million a year. Full temporal video encoding at £0.60 per hour: £345,600 a day, over £125 million a year.

Those numbers do not survive contact with a budget, and stating them is what turns "we would also use video content" into a design decision.

The resolution: process selectively

Not every video deserves the same treatment. Tier by expected search value.

The promotion rule is the interesting part: a video moves up a tier when it accumulates impressions or watch time. This is lazy evaluation applied to a machine learning pipeline — spend the expensive processing on content that has demonstrated it will be searched for.

The failure mode is a cold-start trap. A video that is never surfaced never earns processing, so it stays unsearchable and is never surfaced. Break the cycle with an exploration budget: process a random sample of Tier 3 and give newly-processed videos a temporary retrieval boost, which is the same exploration idea as in the video recommendation monitoring lesson.

Frame sampling

Encoding every frame is unnecessary. A 30-frames-per-second hour of video is 108,000 frames, and consecutive frames are nearly identical.

Three strategies:

1. Uniform sampling. One frame every 2 seconds: 1,800 frames per hour. Simple, and it wastes effort on static scenes while under-sampling fast-cut sequences.
2. Shot-boundary sampling. Detect scene cuts by frame-to-frame difference, take one representative frame per shot. A typical hour might contain 300–800 shots. Better coverage per unit cost, and this is the recommendation.
3. Keyframe-only. Use the compressed video's own keyframes. Nearly free, since no decoding of intermediate frames is needed, at the cost of less control over what gets sampled.

Combine per-frame vectors into one video vector by averaging, or keep a small set of shot vectors so a query can match a specific moment — which enables jumping to the relevant point in a long video, a real product feature and a good thing to raise.

Transcripts as text, not as a separate modality

The simplest and most effective use of speech recognition output is to treat the transcript as another text field. Index it in the inverted index alongside the title, with a lower weight because it is long and noisy. Feed it to the video tower as additional text.

No new modality, no fusion architecture, large gain. Compare this with harmful content detection, where genuine multimodal fusion is unavoidable because the harm often exists only in the combination of image and text.

The caveat worth naming: speech recognition quality varies sharply by language, accent, audio quality, and background noise. Transcript-derived recall will be systematically worse for some languages and accents. Track transcript-driven recall by language as a quality metric, because otherwise the system silently serves some users worse than others.