Course Content
Machine Learning System Design Interview
11 sections · 33 lessons
People You May Know: the model, serving, privacy and fatigue
Five hundred candidates, a rich set of hand-computable graph features, and a genuine question about whether the sophisticated option earns its cost. That is the model half of this lesson.
The rest is the running system, and it asks three questions: what to precompute, who is being harmed, and which signals may be used at all. On this problem the privacy answer shapes the design as much as the latency answer does.
Gradient-boosted trees on graph features
The strong baseline, and — as in Section 7 — very possibly the final answer.
The feature vector per candidate pair:
- Structural: mutual connection count, Adamic–Adar, Jaccard, preferential attachment, number of two-hop paths, whether a three-hop path exists, degree of each endpoint.
- Attribute: same employer, overlapping employment dates, same school with overlapping years, same city, distance, same industry.
- Behavioural: profile views in each direction, co-appearance in search results, messages exchanged, shared group membership.
- Temporal: account ages, days since either changed employer, recent connection activity for each.
- Recipient-side: invitations received recently by the candidate, their acceptance rate, their recent dismissal behaviour. These are the features that let the model learn the guard constraint rather than only having it imposed.
- Negative history: whether this suggestion has been dismissed before, and how many times.
Why trees suit this, and the reasoning is the Section 7 reasoning:
- The features are tabular, engineered, and heterogeneous in scale.
- The signal lives in interactions — mutual connections crossed with same-employer crossed with account age — which trees find without being told.
- Missing values are pervasive (no employer listed, no school, no location) and are handled natively.
- Training is fast, so iteration is fast.
- Feature attributions are available, which matters when someone asks why a particular suggestion appeared.
Trained with binary cross-entropy on "invitation sent and accepted within 30 days", with a second head or a second model on "accepted and interacted with", combined as in Predicting several things at once.
Graph neural networks
The sophisticated option. Worth explaining plainly and then assessing honestly.
A graph neural network learns a vector for each node by repeatedly aggregating information from its neighbours. Round one: each node's vector summarises its own attributes. Round two: each node's vector summarises its attributes plus its neighbours'. Round three: two hops out. After k rounds, a node's vector encodes the structure and attributes of its k-hop neighbourhood. Link prediction then scores a pair by combining their two vectors.
What it can capture that hand-built features cannot:
- Structural patterns nobody thought to engineer — particular neighbourhood shapes that predict connection.
- Attribute and structure jointly, rather than as separate feature columns.
- Higher-order structure beyond the pairwise measures.
- A node representation reusable across other tasks on the same graph — job recommendation, content ranking, search — which can be the strongest argument for building one.
What it costs at this scale:
- Training infrastructure. A 900-million-node graph does not fit in one machine's memory. Training requires neighbourhood sampling, graph partitioning, and distributed coordination — substantially more engineering than a boosting library and a feature table.
- Serving. Node embeddings must be precomputed and refreshed. The graph changes by 40 million edges a day, so embeddings go stale, and a full refresh is expensive.
- Supernodes again. Aggregating over a node with 25,000 neighbours needs sampling, and the sampling strategy becomes a significant hyperparameter.
- Debuggability. When a suggestion is wrong, a tree model can tell you which features drove it. A graph neural network cannot, easily — and on a system with a safety constraint, being able to explain a decision has real value.
- Iteration speed. Hours to days per experiment against minutes.
The honest assessment. For this problem at this scale, the incremental gain of a graph neural network over well-engineered graph features plus gradient-boosted trees is real but modest. The hand-built features — Adamic–Adar, Jaccard, overlapping employment dates — already capture the dominant structure, because the dominant structure in a social graph is simple: triangles close.
Recommendation: build trees on graph features. Revisit graph neural networks when one of three conditions holds — the feature engineering has genuinely plateaued and been measured to have done so; the node embeddings would serve several products, which changes the cost calculation entirely; or the graph is heterogeneous enough (members, companies, schools, skills, posts as different node types with different edge types) that hand-engineering every meaningful path becomes impractical.
That third condition is the strongest argument in practice, and it is worth stating: on a professional network the graph really does have many node and edge types, and enumerating useful paths through them by hand does stop scaling.
Serving: precompute or score live
With the model chosen, the serving question is what to precompute.
| Precompute in batch | Score live | |
|---|---|---|
| What is stored | Top ~200 scored suggestions per member | Candidate sets and features |
| Latency | ~5 ms lookup | ~120 ms |
| Storage at 900M members | Large but bounded — a few hundred bytes per member | Small |
| Freshness | Up to a day stale | Current |
| Context sensitivity | None | Can use the current session |
Recommendation: precompute, with a live overlay. Batch-score the top 200 candidates per member daily. Serve from that list with three live adjustments applied at request time:
- Remove anyone the member has since connected with, dismissed, or blocked, and anyone who has blocked them.
- Boost candidates related to the current session — someone whose profile was viewed minutes ago is a much stronger suggestion than the batch score reflects.
- Apply the volume cap from the metrics lesson, using a live counter of recent unsolicited invitations received by each candidate.
Precomputation works here for the same reason as Section 9: the graph changes slowly relative to how often the module is viewed, and there are a bounded number of enumerable query entities. Contrast with Section 6, where the user's state changes within a session and live retrieval is mandatory.
Trigger an immediate recompute for members whose graph changed materially — a new connection, an employer change, a profile update — since those are precisely the moments when suggestion quality matters most and the stale list is most wrong.
Refresh cadence
| Artefact | Cadence |
|---|---|
| Two-hop candidate sets | Daily, incremental for changed members |
| Model scores | Daily |
| Model retraining | Weekly to monthly — behaviour changes slowly |
| Dismissal and block lists | Streaming — must be immediate |
| Received-invitation counters | Streaming — the volume cap depends on them |
| Session-context boosts | Live |
The two streaming rows — dismissal and block lists, and received-invitation counters — are safety-critical. A dismissal that takes a day to take effect means a member sees a suggestion they explicitly removed, which is the most reliable way to make the feature feel broken.
Privacy constraints on signals
A real design constraint on this problem, and one an interviewer will probe.
| Signal | Consideration |
|---|---|
| Address-book uploads | Requires explicit consent, and it reveals information about people who never consented — a contact who is not a member, or a member who never uploaded anything |
| Profile views | Using them is fine; revealing them through a suggestion may disclose that someone was looking |
| Message metadata | Sensitive; content almost always off-limits |
| Location history | Highly sensitive; usually restricted to coarse city-level |
| Inferred relationships | Suggesting people from a context a member keeps separate — a support group, a medical community, a former life — can be a serious disclosure |
| Cross-product signals | Data from another product in the same company is often restricted by regulation and by the terms under which it was collected |
Two design consequences that belong in the answer rather than in a caveat:
Signals must carry provenance. Each feature is tagged with where it came from and what consent covers it, so the eligible feature set can differ by region and by the member's own settings without retraining a model per jurisdiction. This is the same one-model-many-configurations pattern as in the harmful content monitoring lesson.
Suggestions must not leak. A suggestion is an inference made visible. If A can deduce from being suggested B that B viewed their profile, uploaded their contacts, or is in some context-specific group, the feature has disclosed something. Test for this explicitly: for each signal, ask what a member could deduce from a suggestion driven by it, and drop or blend any signal that answers that question too precisely.
Fatigue
The same suggestion appearing repeatedly and being ignored is the most common complaint about this feature.
Handle it as a state machine per (member, candidate) pair rather than as a score adjustment:
- Shown, no action: demote progressively. A candidate shown 5 times without action should be heavily penalised, not marginally.
- Explicitly dismissed: remove permanently, or for a very long period. This is a stated preference and must be treated as one.
- Invitation sent, not accepted: remove the candidate from the sender's suggestions, and do not suggest the sender to the recipient either.
- Rotate. Even among high-scoring candidates, vary what is shown across sessions. A static list makes the feature feel dead.
Track impressions per (member, candidate) pair as a first-class monitored quantity. It is cheap and it directly measures the most common user complaint.
Monitoring
| Signal | What it detects |
|---|---|
| Acceptance rate by member tenure | New-member experience, which the aggregate hides |
| 99th percentile invitations received per member per week | The harm tail |
| Report and block rate on invitations from suggestions | The safety gate |
| Dismissal rate | Suggestion quality, fast and label-free |
| Suggestion repeat rate | Fatigue |
| Candidate set size distribution | Members with sparse graphs getting empty modules |
| Coverage — share of members appearing in anyone's suggestions | Whether the system serves the whole network or a well-connected core |