Course Content
Machine Learning System Design Interview
11 sections · 33 lessons
Similar listings: cold start, precomputed serving and monitoring
A newly-listed property has no session history, so it has no embedding — and it is exactly when a host most needs bookings. The first half of this lesson gives new listings a place in the space until behaviour takes over.
The second half is serving, where the decision is unusual: almost everything can be precomputed, and knowing when that stops being true is the design judgement. The lesson closes with what drifts, including the fact that what counts as "similar" changes with the calendar.
Why cold start matters commercially, not only technically
A new host lists a property. If it never appears in similar-listings modules or search results, it gets no views, so it accumulates no session data, so it never gets an embedding, so it never appears. The listing is invisible, the host concludes the platform does not work, and they leave.
That is the same self-fulfilling failure as Section 6's new videos and Section 8's new creatives, with a sharper consequence: on a marketplace, the loss is supply, and supply is what makes the product worth using.
At an invented ~40,000 new Roost listings a week, this is a continuous flow, not an edge case.
Deriving an initial embedding from attributes
The technique: learn a mapping from attributes to embedding space using listings that have both.
- Take the roughly 4 million listings that have well-trained behavioural embeddings.
- Build an attribute vector for each: property type, bedrooms, capacity, price band, amenity flags, geohash of location, review score, host type, photograph count.
- Train a small model — a gradient-boosted tree per output dimension, or a shallow network — mapping attributes to the behavioural embedding. This is a regression problem with 32 outputs.
- For a new listing, run its attributes through that model to produce an initial embedding.
The result is a listing placed roughly where similar existing listings sit. Not as good as a learned one — it cannot know that this particular flat is the one people pick when the barn is booked — and far better than nothing.
A simpler alternative worth stating because it is often nearly as good: average the behavioural embeddings of the k most similar existing listings by attributes, weighting by similarity, with geography weighted most heavily. Three lines of code, no model to maintain, and on this problem it captures a large share of what the learned mapping does. Recommend the simple version first and the learned mapping if measurement shows it is worth the maintenance.
The geographic prior
Location dominates substitutability here, so the strongest single component of a cold-start embedding is the location of nearby listings. A new listing 200 metres from an existing cluster of well-embedded listings should start very close to that cluster's centroid, adjusted by price band and property type.
A practical construction: initialise as a weighted average of (a) the centroid of listings within 500 m, weighted 0.5; (b) the centroid of listings in the same market and price band, weighted 0.3; and (c) the attribute-model prediction, weighted 0.2. Those weights are invented and would be tuned; the ordering reflects that geography carries the most signal.
How quickly to replace it
The interesting question, and the answer is gradual rather than binary.
Blend the attribute-derived and behaviour-derived embeddings by how much behavioural evidence exists:
embedding = α · behavioural + (1 − α) · attribute_derivedwhere α = n / (n + k)with n the number of sessions containing the listing and k a smoothing constant — say 50. At 10 sessions, α = 0.17 and the attribute prior dominates. At 50, α = 0.5, an even blend. At 500, α = 0.91 and behaviour has taken over. The transition is smooth and there is no threshold to argue about.
This is the same shrinkage idea used for category affinities in the event recommendation features lesson and for the hierarchical fallbacks in the ad click monitoring lesson. It recurs because it is the right answer to "I have a little evidence and a decent prior".
Guaranteeing exposure
The blend gives a new listing a reasonable position. It does not guarantee it gets shown, and without impressions n never grows.
Reserve one of the 12 module slots for a listing with fewer than a threshold number of sessions, subject to it passing the hard filters and having an attribute-derived embedding reasonably close to the anchor. The cost is a small drop in module click-through rate. The benefit is that new supply enters the behavioural data within days rather than months.
Measure the cost honestly — it is real — and defend it as a marketplace investment rather than pretending it is free.
Precompute or search live
With every listing — new or old — holding an embedding, the serving decision is unusual: almost everything can be precomputed, and knowing when that stops being true is the design judgement.
| Precomputed neighbour lists | Live approximate nearest neighbour search | |
|---|---|---|
| What is stored | Top 200 neighbours per listing, as a list | The embedding index |
| Storage | 6M × 200 × ~12 bytes ≈ 14 GB | 6M × 32 × 4 bytes ≈ 768 MB |
| Latency | ~2 ms — a key-value lookup | ~15 ms — an index query |
| Personalisation | None; identical for every user | The query vector can blend anchor and session |
| Freshness | As stale as the last batch job | Immediate for anything in the index |
| Operational complexity | Low | Medium |
The recommendation: precompute, with a live path for the session-aware case.
Precomputation wins here because the anchor is a listing, not a user. There are 6 million anchors, which is a tractable number to enumerate, and the neighbour list for a given anchor is the same for everyone before personalisation. A nightly batch job computing the top 200 neighbours for every listing is a few hours of work and turns the request path into a key-value lookup.
Contrast with Section 6, where the anchor is a user — 400 million of them, each with a vector that changes within a session. Enumerating those is not possible, so live retrieval is mandatory. The distinction is worth stating: precompute when the query space is small and enumerable; search live when it is not.
Where the live path earns its place: when the module should reflect the current session rather than only the anchor. Averaging the embeddings of the last few listings viewed and querying with that produces a noticeably better module for users deep in a session. Serve the precomputed list by default and the session-aware query for sessions past a few views.
The request path
- Look up the anchor's precomputed 200 neighbours. ~2 ms.
- Filter hard constraints: availability on the user's dates, guest capacity, price ceiling, instant-book if the user filtered on it. This typically removes 60–80% of the candidates on a dated search — which is why the precomputed list holds 200 and not 12.
- Re-rank the survivors with a small model over embedding similarity, price difference from the anchor, review score, distance, and the user's session context.
- Apply diversity: at most 2 from one host, and a spread of price points.
- Return 12.
Step 2 is the one to size correctly. If availability filtering removes 80% of 200 candidates, 40 remain and there is enough to rank. If a user has unusual dates and it removes 95%, only 10 survive and the module is thin — so define a fallback: widen to the next 200 neighbours, or fall back to the same-market baseline from the framing lesson.
The filter-before-rank ordering is the same principle as in the event recommendation serving lesson.
Refresh cadence
| Artefact | Cadence | Reason |
|---|---|---|
| Embedding model retrain | Weekly | Behaviour shifts slowly; retraining moves the whole space |
| Precomputed neighbour lists | Weekly, with the model | Must be rebuilt together or the lists reference the old space |
| New-listing attribute embeddings | Hourly | New supply must become visible quickly |
| Availability data | Streaming | Showing a booked listing is a visible failure |
| Re-ranking model | Weekly | Cheap to retrain, tracks seasonal shifts |
The atomicity requirement repeats from the visual search monitoring lesson: the embedding model and the neighbour lists are one artefact. Deploy a new model against old lists and the module returns coherent nonsense.
Seasonality in what "similar" means
A property of this domain that most recommendation problems do not have.
A ski chalet and a lakeside cabin in the same valley are substitutes in July and not in February. A city apartment near a conference centre substitutes for a different apartment near the same centre during conference season and for something else entirely in August. What counts as similar genuinely changes with the calendar.
Three responses, in increasing order of effort:
- Season as a re-ranking feature. Keep one embedding space, and let the re-ranker use the month and the listing's seasonal booking pattern. Cheapest, and it captures a good share of the effect.
- Separate embeddings per season. Train on summer sessions and winter sessions separately, and switch by the user's requested dates. Doubles the training and storage, and it directly represents the phenomenon.
- Date-aware training. Include the requested date range as context in the model. Most principled, most complex.
Start with option 1 and measure whether the seasonal error justifies option 2. In most markets it will not; in strongly seasonal ones it will.
Monitoring
- Coverage: the share of listings appearing in at least one module per week. Falling coverage means the embedding space is collapsing onto popular listings.
- New-listing time-to-first-impression. The supply-side health metric. It should be hours, not weeks.
- Post-filter candidate count distribution. How often does availability filtering leave fewer than 12? This is the leading indicator of a thin module.
- Embedding space stability across retrains. Compare each listing's neighbour set before and after. A large shift means either a genuine behaviour change or a training problem, and you want to know which before the lists go live.
- Module click-through and booking rate, split by market size.