Nearby Friends: the publish-subscribe architecture, subscriptions and scaling

Nearby Friends: scope, scale and why the proximity design fails ended with a direction rather than a design: route each location update to whoever is interested, then throw it away. This lesson builds that architecture, then the subscription management that keeps it correct as people move, and then what breaks first at scale.

The architecture

Three components, and one genuinely open choice about how channels are keyed.

The components

Connection servers. Each client holds a WebSocket to one of roughly 100 servers (How the server pushes to the client compares the alternatives). The connection carries location updates upward and friend updates downward. The server is stateful in exactly the way the chat system's stateful problem described, and the same registry-and-reconnect machinery applies.

Location cache. An in-memory key-value store: loc:{user_id} → (lat, lng, timestamp) with a time-to-live of around 60 seconds — twice the update interval, so one dropped update does not make a friend vanish. Four hundred megabytes total, no durability, and self-healing after a total loss.

Publish-subscribe broker. Where updates are routed. Redis is the common implementation; any broker with cheap dynamic channel creation works, and the design should not depend on a specific product.

The flow

1. The client sends its position over the WebSocket every 30 seconds.
2. Its connection server writes the position into the location cache with a fresh time-to-live.
3. The server publishes the update to a channel.
4. Subscribers to that channel receive it. Each subscriber is a connection server acting on behalf of the users it holds.
5. The receiving server filters: is the publisher a friend of this user, and is the distance within the radius? If both, push a compact update to that client.

Step 5 happens on the connection server rather than the broker, because the server already holds each connected user's friend list in memory. The broker stays a dumb, fast router.

The open choice: what is a channel?

Option A — a channel per geohash cell. Every user publishes to the channel named by their current cell, and subscribes to their own cell plus its eight neighbours (the boundary fix from the geohash deep dive, reused here for exactly the same reason).

Good: subscription count is bounded at nine per user regardless of friend count, and the geographic scoping means a user in Delhi never receives traffic from Toronto.

Bad: a subscriber receives every update in their cell, including from the 49,998 people who are not their friends. In dense cells that is a great deal of wasted delivery — the hot-cell discussion below costs it.

Option B — a channel per user. Every user publishes to their own channel, and subscribes to each of their online friends' channels.

Good: perfectly selective. You receive updates only from friends, so no filtering waste at all.

Bad: subscription count is the friend count. Ten million active users × 200 friends = 2 billion subscriptions, and every friend coming online or going offline changes the subscription set.

Recommendation: start with Option A, the cell channel, because bounded subscriptions are easier to operate and geographic scoping is a natural shard key. Then fix its weakness rather than abandoning it: use a finer geohash precision in dense regions so no channel carries too much traffic, and fall back to per-user channels for users whose cell is pathologically busy. The scaling section below does the arithmetic that motivates this.

Managing subscriptions

Cell channels give bounded subscriptions. Keeping those subscriptions correct as people move is the operational work of this design.

The subscription set

On connecting, a client's server computes the user's current geohash cell and its eight neighbours, and subscribes to all nine. Nine rather than one, for exactly the reason in the geohash deep dive: a friend 200 m away can be in an adjacent cell, and subscribing only to your own cell would make them invisible.

Choose the precision so the cell is at least as large as the notification radius. With a 5 km radius, a precision-5 geohash (roughly 5 km cells) gives a 3×3 block spanning about 15 km — the radius is fully covered with margin.

Resubscribing on the move

When a user crosses a cell boundary, their nine-cell block shifts. Moving one cell east means unsubscribing from the three cells on the western edge and subscribing to three new cells on the eastern edge. Six channel operations, not eighteen — recompute the difference between the old and new sets rather than tearing everything down.

The cost of churn

Cost it for the worst realistic case: a car on a motorway at 100 km/h crossing 5 km cells.

5 km ÷ 100 km/h = one boundary crossing every 3 minutes

Six channel operations every three minutes from one fast-moving user is nothing. Now the whole population, assuming an average crossing every 10 minutes across a mix of stationary, walking, and driving users:

10,000,000 users ÷ 600 s = ~17,000 boundary crossings per second × 6 channel operations = ~100,000 subscribe/unsubscribe operations per second

Substantial, and small relative to the 333,000 publishes per second the broker is already handling. Churn is a real cost and not the dominant one — worth stating precisely rather than hand-waving in either direction.

Flapping at a boundary

A person sitting in a café exactly on a cell boundary will have their reported position jitter across it as GPS noise moves them a few metres. Each jitter triggers six channel operations, and a few thousand such users produce constant pointless churn.

Two standard fixes:

• Hysteresis. Do not switch cells until the user is a set distance past the boundary — say 200 m. Crossing costs a switch; hovering does not.
• Dwell time. Require the new cell to be the reported cell for two consecutive updates before acting.

Both trade a small amount of latency at genuine crossings for a large reduction in churn.

Where subscriptions actually live

One refinement worth mentioning: the subscriber is the connection server, not the user. A server holding 100,000 users in a city needs one subscription per distinct cell among them, not one per user. Because users cluster, that collapses a great many subscriptions into a few, and the server then multiplexes each received update to whichever of its local users care.

This is the same idea as connection multiplexing, and it converts a per-user cost into a per-server cost — an order-of-magnitude reduction with no change to behaviour.

Scaling and follow-ups

What breaks first, and the questions this problem attracts.

Sharding the broker

A single broker instance cannot absorb 333,000 publishes per second plus the resulting fan-out. Shard by channel name: the cell identifier hashes to a shard, and all traffic for that cell stays on one node. Geographic locality helps here — a user's nine cells will often hash to several shards, so a connection server maintains connections to all shards, which is acceptable at a few dozen nodes.

The hot cell

This is the failure the channel recommendation above was hedging against. Consider a dense cell — a stadium, a festival, a city square — with 50,000 people inside it.

50,000 users ÷ 30 s = 1,667 publishes per second in that one channel, delivered to every subscriber in the cell

If every one of those 50,000 users subscribes, that single cell produces 1,667 × 50,000 = 83 million message deliveries per second. One cell, and it exceeds the entire rest of the system.

Three responses, applied together:

1. Adaptive precision. Index dense regions at a finer geohash precision — precision 7 instead of 5 — so the stadium becomes hundreds of small channels instead of one. The subscription block grows to cover the same physical radius, which is the cost.
2. Fall back to per-user channels in dense cells. Option B from the architecture above, applied selectively where it wins: subscribe to your friends directly and ignore geography entirely. In a stadium, 200 friend subscriptions beats receiving 1,667 messages a second.
3. Cap and sample. Above a threshold, reduce the publish rate within the cell, accepting coarser update granularity where the feature is least useful.

The general lesson generalises well beyond this problem: a partitioning scheme based on physical space inherits the unevenness of physical space. The same failure appeared as the dense-cell problem in the proximity service's serving lesson, and as the hot-key problem in every sharding discussion in this course.

Users with very many friends

Someone with 5,000 friends imposes a large filter cost on their connection server for every update received. Since filtering is a set membership test against an in-memory friend list, it stays cheap. The real cost is on the outbound side: if many of those friends are nearby, one person can receive hundreds of updates per second. Cap the number of nearby friends displayed — a user cannot read a list of 300 anyway — and stop pushing beyond it.

Battery-aware update intervals

The client controls the update rate, and it should vary it:

• Stationary (no significant accelerometer activity, position unchanged): back off to one update every few minutes, or stop entirely.
• Walking: every 30 seconds.
• Driving: the position changes fast, but so does cell membership; every 15 to 30 seconds is usually enough for a feature with a kilometre-scale radius.
• Screen off / app backgrounded: reduce sharply or suspend, subject to the platform's background-execution rules, which differ between mobile operating systems and change between releases.

Adaptive intervals cut the 333,000 writes per second substantially, because at any moment most users are not moving.

Time-to-live and stale positions

The 60-second time-to-live is a correctness mechanism, not a cleanup detail. A user who loses signal should disappear from friends' lists rather than appear frozen at a location they left ten minutes ago. Showing a stale position as current is worse than showing nothing, because the user acts on it.

Privacy, which this feature demands

Not optional and not an afterthought:

• Explicit, revocable opt-in, with an off switch that takes effect immediately — which here means unsubscribing and deleting the cache entry, both of which are instantaneous.
• Per-friend granularity. Sharing with everyone is rarely what people want.
• Coarse rather than exact positions. Displaying "about 1 km away" instead of precise coordinates preserves the feature's value and removes most of its risk.
• No history. The time-to-live enforces this structurally: there is nothing to retain because nothing is written durably.
• Automatic expiry of sharing sessions, so a switch left on is not left on forever.

Legal obligations around location data differ substantially between jurisdictions and are evolving, so any specific compliance claim should be checked by someone with current regional expertise rather than asserted from a design document.