Google Drive: the sync architecture, conflicts and follow-ups

Google Drive: scope, scale and block-level storage settled what to send: only the blocks that changed. This lesson is how the pieces fit, how a device learns that something changed elsewhere, what happens when two devices change the same file, and the follow-ups the problem attracts.

The sync architecture

The components

Notice the ordering: blocks reach the block store before the metadata commit, so a stored version can never reference data that does not exist.

The metadata database is the source of truth. It holds the namespace: every file's path, size, timestamps, version number, owner, permissions, and ordered block-hash list. Sharded by user identifier, because every operation is scoped to a namespace. This is the database that must be right; the block store is content-addressed and therefore self-verifying.

The block store holds immutable blocks keyed by hash. Immutability is a gift: blocks are never updated, only added and eventually garbage-collected, so there is no cache invalidation and no write conflict at the block layer.

The notification service tells connected devices that their namespace changed. It carries a cursor, not content: "your namespace has advanced to change 8,412". The device then calls the metadata API to find out what actually changed.

That indirection matters. Notifications become tiny and idempotent — receiving the same cursor twice costs nothing — and a device that was offline for a week takes the same code path as one that missed a single change.

The client

Four parts, and they are worth naming because interviewers ask what runs locally:

1. Watcher — observes the local folder through the operating system's file-change notifications.
2. Chunker — splits changed files into blocks and hashes them.
3. Indexer — maintains a local database of file → block-hash list, so the client can compute a diff without re-reading everything.
4. Transfer queue — uploads new blocks and downloads missing ones, with retry and resume.

The upload sequence

1. Watcher sees report.pptx changed.
2. Chunker produces the new hash list; the indexer diffs it against the stored one. One new hash.
3. Client asks the API which of those hashes the server lacks.
4. Client uploads the missing block directly to the block store using a pre-signed URL (the same pattern as YouTube's upload path — bytes do not pass through the application tier).
5. Then, and only then, the client commits the new metadata: file version n+1 with the new hash list.
6. The metadata write appends to the user's change log; the notification service pushes the new cursor to the user's other devices.

Step 5 is deliberately last. Metadata is committed only after every block it references exists, so a partially uploaded file is never visible. If the client dies at step 4, the uploaded block is an orphan — wasteful, and cleaned up by a background collector — but no version ever references missing data. The object store's write path uses the same ordering for the same reason.

Long polling versus WebSockets

Both work. Long polling — the client issues a request the server holds open for 30 to 60 seconds — is the pragmatic choice here, because sync tolerates a few seconds of latency and long polling needs no persistent-connection infrastructure. WebSockets lower latency and are worth it if the product promises near-instant propagation. The chat system's lesson on server push compares them properly.

Conflict resolution

Two devices, both offline, both edit the same file. Both reconnect. This is the moment the design either loses data or does not.

Why it cannot be avoided

Offline editing means changes are made without coordination. Nothing on either device knew the other change existed. This is not an implementation gap — it is the definition of concurrent modification, and no amount of locking on the server helps, because neither device was talking to the server when it happened.

Detection is straightforward, though. Every file has a version number in the metadata database. A client submits an update as "version 7 → version 8". If the stored version is already 8, this update was computed against stale state, and the two changes are concurrent.

The three options

Last-write-wins. Accept whichever update arrives second and discard the first.

Cheap, and it silently destroys work. The user who edited on the train discovers their afternoon is gone with no notification and no recovery path. Against the non-functional requirement "no silent data loss", it is disqualified. Say so explicitly, because a surprising number of designs land here by default.

First-write-wins with a conflict copy. The first update to arrive becomes version 8. The second is rejected and its client writes the local version alongside as a new file — report (conflicted copy 2026-08-31, Priya's laptop).pptx.

Nothing is lost. The resolution is handed to the person best placed to do it. The cost is a duplicate file the user must reconcile by hand.

Operational transformation or conflict-free replicated data types. Represent the document as a sequence of operations rather than a blob, and define how concurrent operations combine so that every replica converges on the same result. This is what makes real-time collaborative editing work, where two people type in the same paragraph at the same moment.

The recommendation, and the honest reason

Recommend first-write-wins with a conflict copy for a file sync service.

The reason is not that merging is too hard to implement. It is that merging is not defined for most file types. A .psd, a .mp4, a compiled binary, a compressed archive — there is no meaningful way to combine two concurrent versions. A merge algorithm needs to understand the document's structure, and a file sync service treats files as opaque bytes by design.

A caveat on block-level sync worth knowing

Modern office formats are compressed archives. Changing one slide can alter the compressed byte stream well beyond the edited region, so the block-level saving from Block-level storage and deduplication is smaller for these formats than the clean 10× arithmetic suggests. It still helps — much of a large file is unchanged media — but the honest statement is that the saving depends on file format, and uncompressed or append-mostly files benefit most.

Follow-ups

Five extensions that come up on almost every run of this problem.

Sharing and permissions

Sharing breaks the assumption that a file belongs to one namespace. Two models:

• Copy the entry into the recipient's namespace. Fast reads — a user's file list is a single shard query — at the cost of writing to every recipient's namespace when permissions change. This is fan-out on write (Fan-out on write versus fan-out on read) applied to a file tree.
• Store an access control list on the file and resolve at read time. One write on a permission change, and a "shared with me" listing becomes a query across shards.

Recommend a hybrid, for the same reason as Section 13 (Design a News Feed System): fan out for ordinary shares, resolve at read time for a document shared with 50,000 people. Add inherited folder permissions and you also need a rule for what happens when a file moves between folders with different access — which is a product decision that must be made explicitly, because leaving it implicit produces security bugs.

Version history and its storage cost

With content-addressed blocks, a version is an ordered hash list. Keeping version n costs only the blocks that version introduced.

A 40 MB file edited 100 times, one 4 MB block changing each time:

Naive full copies: 100 × 40 MB = 4 GB

Block-level versions: 40 MB + (100 × 4 MB) = 440 MB

Roughly a 9× saving, and it improves as files grow. Bound it anyway with a retention policy — 30 days, or the last 100 versions — and garbage-collect blocks no live version references.

Garbage collection is the genuinely tricky part: a block may be referenced by many files across many users, so deletion requires reference counting or a periodic mark-and-sweep. Deleting a still-referenced block is unrecoverable data loss, so the safe pattern is to mark, wait out a grace period longer than any in-flight operation, and only then sweep.

Very large files

A 20 GB file at 4 MB blocks is 5,000 blocks and 5,000 hashes in one metadata record. Three adjustments: upload blocks in parallel with a bounded concurrency; store the block list separately from the file record so the record stays small; and support resumable transfer at block granularity, so a failure at 90% costs one block.

Quota enforcement

A user's used-storage figure is a sum over their namespace, and recomputing it on every upload is expensive. Keep a counter, updated asynchronously, and accept that it lags.

The consequence is honest and worth stating: a user can briefly exceed their quota. That is acceptable — a small overshoot costs a little storage — whereas blocking every upload on a strongly consistent counter costs latency on the hot path for every user. Enforce hard limits at a threshold above the nominal quota, and reconcile the counter periodically.

Cold storage tiering

Access to stored files is skewed the same way video is (YouTube's cost optimisations): recent files are opened constantly, files untouched for a year are opened rarely. Blocks not read for, say, 180 days move to a cheaper storage class with slower retrieval.

Two Drive-specific wrinkles. Deduplication means a block can be cold for one user and hot for another, so tiering decisions belong to the block, based on aggregate access, not to the file. And a cold block's first read is slow, so the client should show progress rather than appear frozen.