Visual search: ANN serving, monitoring and follow-ups

The model is done. Now find the 100 nearest vectors out of 40 million, 3,000 times a second, in under 30 milliseconds.

That serving problem is the first half of this lesson. The second half is what happens once the system is live — what drifts, how new listings arrive, where safety and personalisation belong — which is the material for the last five minutes of the interview and for the follow-up questions that always come.

Why exact search does not fit

Exact nearest neighbour search compares the query vector against every catalogue vector. With 256 dimensions and 40 million items that is about 10.2 billion multiply-add operations per query. Even at a highly optimised billion operations per second per core, one query occupies ten core-seconds. At 3,000 queries per second you would need tens of thousands of cores for search alone.

The whole latency budget for retrieval was 25 ms (see Step 6: serving). Exact search misses it by roughly three orders of magnitude.

The trade is explicit and it is the point: give up a few percent of correctness, gain three orders of magnitude of speed.

The three methods to know

Inverted file index (IVF). Cluster all vectors into, say, 16,384 groups and store the centroid of each. At query time, compare against the 16,384 centroids, pick the nearest few (the nprobe parameter), and search exhaustively only inside those. Searching 8 of 16,384 clusters examines roughly 0.05% of the catalogue. Tuning is one dial: more probes, more recall, more latency. The failure mode is a query near a cluster boundary, whose true neighbours sit in a cluster you did not probe.

Product quantisation (PQ). A compression scheme, usually combined with IVF. Split each 256-dimensional vector into 32 chunks of 8 numbers, and replace each chunk with the ID of the nearest of 256 learned representative chunks. A vector that took 1,024 bytes now takes 32 bytes — a 32× reduction, taking the 41 GB index to about 1.3 GB. Distances are computed against the compressed codes using a small lookup table. It costs recall, because the compression is lossy, and it is what makes a large index fit in memory on a modest number of machines.

Hierarchical navigable small world (HNSW). A layered graph. Each vector is a node connected to its near neighbours; upper layers hold fewer nodes with longer-range links. Search enters at the top, greedily walks toward the query, and descends layer by layer — like finding a house by taking the motorway, then the main road, then the street. Typically the best recall-at-latency of the three. The costs are memory (the graph edges add substantially on top of the vectors) and update behaviour: deletions are handled with tombstones and the graph degrades until it is rebuilt.

The trade-off, plotted

Figures below are illustrative, chosen to show the shape of the trade-off on a catalogue of this size. Benchmark on your own data before committing.

Read the shape, not the values. Every method has a dial that trades latency for recall, the curves flatten hard above about 95% recall, and product quantisation buys memory by giving up a few points of recall at the same latency.

Choosing for Attic

Recommendation: IVF with product quantisation, sharded by category across eight machines, rebuilt nightly, with new listings appended to a small uncompressed HNSW index searched in parallel and merged. The small index keeps today's listings findable within minutes; the large one carries the catalogue.

That two-index pattern — a big periodically-rebuilt index plus a small live one — is the standard answer to "how do new items become searchable immediately", and it recurs in Sections 4, 6 and 9.

What drifts here

With the system live, monitoring starts. Three kinds of drift matter for Attic, and the first is specific to embedding systems.

Embedding drift across retraining. The one that surprises people. Retrain the encoder and every vector moves — the new model's space is not the old model's space. Query vectors from the new model against an index built by the old model return nonsense. The index must be rebuilt in full on every encoder change, and query and index versions must be pinned together and switched atomically. Rebuilding 40 million embeddings is hours of accelerator time and a real operational cost, which is why encoder retraining is quarterly rather than nightly.

Catalogue drift. Attic's mix shifts with season and with seller behaviour. If listings photographed against plain backgrounds go from 20% to 60% because a new seller tool crops images automatically, the encoder is now seeing a different input distribution than it was trained on.

Query drift. Users learn what the feature is good at and ask it for that. The query distribution narrows over time, which makes online metrics look better while coverage quietly gets worse.

Monitor: mean and variance of the embedding norm, mean nearest-neighbour distance for a fixed probe set of query images, and the zero-click rate segmented by category. That last one is the cheapest early warning available.

Adding new items without retraining

The strength of the embedding framing. A new listing is encoded once — a single forward pass, a few milliseconds — and appended to the live index. No retraining, no class list to extend. End-to-end, a new listing should be searchable within minutes.

Filtering unsafe and unwanted results

Similarity is not the only requirement. A visual search must not return prohibited items, adult content, or listings that violate policy, and "the embedding put it near the query" is not a defence.

The correct place for this is a filter after retrieval and before display, driven by a separate classifier — the subject of Section 5 (Harmful Content Detection) — plus a blocklist. Retrieve 200, filter, show 100. Do not attempt to teach the similarity encoder to avoid unsafe items; you would be asking one model to satisfy two objectives that have nothing to do with each other, and the failure would be silent.

Personalising the ranking

Retrieval by visual similarity is identical for every user. Personalisation belongs in the re-rank: score the 100 candidates with a small model using visual similarity, the user's category and price history, distance to the seller, listing freshness, and seller quality.

Note the pattern from Step 6: serving — cheap and general first, expensive and personal second. Section 6 (Video Recommendation System) makes this the centrepiece.