Course Content
Machine Learning System Design Interview
11 sections · 33 lessons
Visual search: framing, metrics and training data
Prompt: "Design a visual search system. A user taps an image and gets visually similar items back."
This is the first case study, so it runs slowly and names each of the seven steps from the framework as it reaches them. Later case studies move faster. This lesson covers the first three steps — framing, metrics and data — which together decide what the encoder will actually learn.
The clarifying questions
The word doing all the damage in the prompt is similar. Ask what it means before anything else.
- Similar by what? Same object category? Same colour and pattern? Same style? Same physical product? These are four different systems. On a furniture marketplace, "another photo of this exact chair" and "chairs that would look good in the same room" need different training data.
- Image only, or image plus text? If the user can add "in oak" to a photo, the problem becomes multimodal and looks more like Section 4 (YouTube Video Search).
- How large is the catalogue, and how fast does it change? Ten thousand items is a different system from forty million. A marketplace where items sell and vanish within days is different from a stable product catalogue.
- Latency budget and surface. Is this a full-screen search result page, or a strip that loads under an item?
- What happens after? Are we optimising taps on results, or purchases?
The answers we will design against
Invented, and fixed for the rest of this case study. The product is Attic, a second-hand furniture and homeware marketplace.
| Constraint | Value |
|---|---|
| Catalogue | 40 million active listing images |
| Churn | ~500,000 new listings a day; ~450,000 sold or expired |
| Query volume | 3,000 visual searches per second at peak |
| Latency budget | 200 ms p99 for the result page |
| Definition of similar | Same object type and visual character — a buyer looking for "this kind of thing" |
| Success | The user taps a result, and eventually messages the seller |
Framing it as machine learning
Following the framing step, state input, output, and objective explicitly.
- Input: one query image, plus optional filters (price range, distance, category).
- Output: a ranked list of 40 listing IDs from the catalogue.
- Objective: maximise the chance the user taps a result and contacts the seller.
The task type is retrieval, then a light re-rank — not classification.
That distinction is worth defending, because "classify the image, then show other items of that class" is the obvious wrong answer and an interviewer may offer it to you.
Classification fails here for three reasons. First, the label set is unbounded: new product types appear constantly and a fixed class list cannot cover a second-hand marketplace. Second, class membership is too coarse — every one of the 180,000 listings tagged "chair" is equally "similar" under classification, which is no ordering at all. Third, adding a new listing must not require retraining, and with a classifier every new class does.
Retrieval by embedding similarity fixes all three. The model learns a function from image to vector such that visually similar images land near each other. A new listing is encoded once and inserted into the index. No retraining, no class list, and a real-valued distance that gives you an ordering.
The baseline first
Before any encoder: take the query image's listing category and colour histogram, and return recent listings in the same category sorted by colour distance. No training, one afternoon of work, and it establishes the whole serving path and the metrics.
It will be mediocre — colour histograms treat a blue sofa and a blue rug as near-identical — but you now have a number to beat, and if it turns out to capture 70% of the available value, that is worth knowing before spending three months on an encoder.
Metrics: measuring a similarity nobody logs
Similarity has no natural log entry. Nobody clicks a button labelled "these two chairs are alike", which makes this the hardest metric problem in the course.
Offline metrics
Offline evaluation needs a set of query images each paired with a set of known-similar catalogue images. Two ways to build one, and both are imperfect.
Human-judged similarity sets. Show an annotator a query image and 20 candidates, and ask them to mark each as very similar, somewhat similar, or not similar. Expensive — at roughly 90 seconds per query image, a 5,000-query evaluation set is over 100 annotator-hours — and noisy, because "somewhat similar" means different things to different people. Expect inter-annotator agreement on the middle grade to be poor.
Behaviour-derived sets. Treat "images a user viewed in the same session and then messaged the seller about" as similar. Free and plentiful, but it inherits whatever the current system showed. Items nobody was ever shown never appear as positives.
Use both: the behavioural set for iteration speed, and a smaller human set as the honest check, because the behavioural set cannot tell you about the items your system never surfaces.
Given an evaluation set, the metrics are the ranking metrics from the offline metrics lesson:
| Metric | Why it is here |
|---|---|
| recall@k (k = 100) | Retrieval-stage metric. Does the ANN index return the right items at all? |
| mAP | Overall ordering quality with binary relevance. The headline offline number |
| nDCG@20 | Used with the human set, where relevance is graded rather than binary |
| Embedding query time at fixed recall | Not a quality metric, but it belongs on the same dashboard — a 2% recall gain that costs 60 ms is a loss |
Online metrics
| Metric | Layer | What it tells you |
|---|---|---|
| Click-through rate on the result grid | Immediate | Are results attractive at thumbnail size? |
| Position of first click | Immediate | Is the ordering right, or is the good result at rank 12? |
| Seller contact rate per visual search | Session | The real target — did the search produce intent? |
| Visual searches per user per week | Retention | Is the feature earning its place? |
| Guard: zero-click rate | Immediate | The fraction of searches with no interaction at all — the clearest signal of bad results |
The zero-click rate is the guard metric this system needs. Click-through rate can rise because results became more eye-catching at thumbnail size while being less relevant — a brightly lit photo of the wrong chair beats a dim photo of the right one. Zero-click rate catches the case where the whole slate is wrong.
Where offline and online disagree here
Two causes specific to visual search.
Thumbnail effects. The model scores full-resolution images; the user sees 180-pixel thumbnails on a phone. Distinctions the embedding captures may be invisible at that size, and photo quality — lighting, background clutter, framing — drives clicks in ways no similarity metric measures.
Availability. A perfect match that sold yesterday is a bad result. Offline evaluation against a static snapshot never sees this; online it shows up as clicks that lead nowhere.
Data and labels: where the similarity pairs come from
The encoder learns whatever your training pairs tell it to learn. This step decides the quality of the entire system, which is why it comes before the model.
Three sources of similarity pairs
Co-engagement. Two listings viewed in the same session, or two listings a buyer messaged about, are treated as a similar pair. Free and available in volume — Attic logs roughly 12 million sessions a week. The bias is severe: it only pairs items that were shown together, which the current ranking decided.
Human annotation. Annotators mark pairs as similar or not against a written rubric. Accurate, slow, expensive. Best spent on evaluation and on hard cases rather than on bulk training data.
Self-supervision from augmented pairs. Take one image, produce two randomly altered versions of it, and declare them a positive pair. No labels needed at all, and unlimited volume. This is the default starting point and it works startlingly well.
The recommendation: pretrain with self-supervision on the full catalogue, then fine-tune on co-engagement pairs, and hold out human annotation for evaluation. That ordering gets a usable model without waiting on an annotation budget.
What augmentation decides
Self-supervision has one dial and it controls everything. The augmentations you apply define what the model is told to ignore.
| Augmentation | Tells the model to ignore | Right for Attic? |
|---|---|---|
| Random crop and resize | Framing and zoom | Yes — sellers photograph from any distance |
| Brightness and contrast jitter | Lighting conditions | Yes — living-room photos vary wildly |
| Horizontal flip | Left–right orientation | Yes for furniture; wrong for text or logos |
| Rotation up to 15° | Camera tilt | Yes |
| Grayscale conversion | Colour entirely | No — buyers care about colour |
| Heavy blur | Fine texture | No — wood grain and fabric weave are the signal |
Applying grayscale augmentation would teach the model that a brown leather sofa and a grey fabric sofa are the same item. That is a modelling decision disguised as a data-loading parameter, and it is a good thing to say aloud in an interview.
Building triplets
The classic training unit is a triplet: an anchor, a positive that should be near it, and a negative that should be far.
- Anchor: a listing photo.
- Positive: an augmented version, or another photo of the same listing, or a co-engaged listing.
- Negative: another listing.
The negative is where the quality is. A random negative is nearly always a completely different object — a lamp against a sofa — and after a few thousand steps the model finds those trivial. Gradients go to zero and training stops improving while the loss looks fine.
Hard negatives decide the system
A hard negative is a listing that is not the right answer but looks like it. For an oak dining chair anchor: a different oak dining chair with a slightly different back. For a navy two-seat sofa: a navy three-seat sofa.
Mining them, cheapest first:
- In-batch negatives. Within a training batch of 512 images, every non-matching image serves as a negative. Free, and gives you 511 negatives per anchor.
- Same-category mining. Draw negatives only from the anchor's own category, so the model must distinguish chairs from chairs rather than chairs from lamps.
- Model-mined negatives. Every few epochs, run the current model over the catalogue and pull each anchor's nearest neighbours that are not positives. The most effective and the most expensive, and the one that destabilises training if overdone — some "hard negatives" are genuinely similar items you had no label for, and punishing the model for finding them teaches it something false.
A workable mix: in-batch negatives as the base, roughly 10–20% mined hard negatives, refreshed every few epochs.