Google Drive: scope, scale and block-level storage

The prompt: "Design Google Drive." A file storage and synchronisation service: files live in the cloud, and a folder on every one of your devices mirrors them.

Attempt it for 45 minutes before reading. Try specifically to answer this: a user changes one paragraph in a 40 MB document. What exactly travels over the network? The quality of your answer to that question is the quality of your design.

This lesson answers that question. It scopes the problem, runs the estimate that exposes how wasteful whole-file sync would be, and then builds the storage idea that closes the gap: content-addressed blocks.

What this problem is actually about

It is tempting to treat this as "object storage with a user interface", and that reading produces a shallow answer. Object storage is Section 26 (Design an S3-like Object Storage). What makes Drive its own problem is synchronisation: many devices, each holding a local copy, each of which can change independently, some of them offline, all of which must converge on the same state.

Two hard parts follow: moving the minimum number of bytes when something changes, and deciding what to do when two devices change the same thing. Block-level storage, later in this lesson, handles the first; Conflict resolution handles the second.

The five questions

1. Which platforms? Web only is a much smaller problem — there is no local mirror, so there is no sync and no conflict. Desktop and mobile clients holding local copies are what create the interesting work. Assume desktop plus mobile plus web.

2. What is the maximum file size? This decides whether a file is a single object or a composition of parts. Assume a cap in the range of tens of gigabytes, which means multi-part handling is mandatory.

3. Is offline editing supported? If devices can change files while disconnected, conflicts are inevitable and the design must have an answer. If everything requires connectivity, the server can serialise all changes and the problem is far easier. Assume offline is supported — it is what users expect from a sync client.

4. Are sharing and permissions in scope? Sharing changes the data model, because a file's location in one user's tree is not the same as its location in another's. Scope it as a follow-up (see Follow-ups) and say why.

5. Is version history required? "Restore a previous version" is a common expectation, and it interacts directly with the block storage design below — with block-level storage, keeping a version costs only the blocks that changed.

Requirements and scale

Functional: upload and download files; sync a local folder across devices; see changes made on another device appear locally within seconds; work offline and reconcile on reconnection; restore previous versions; share a file with another user.

Non-functional: no silent data loss, ever — this is the requirement that outranks everything else; changes propagate to online devices within a few seconds; bandwidth used per change is proportional to the size of the change, not the size of the file; storage cost per user stays viable at consumer pricing.

The numbers

Invented figures for a consumer sync service. A day is 100,000 seconds (see Rounding aggressively and staying fast).

Total storage: 50M × 10 GB = 500 PB, before replication. At a replication factor covering durability requirements, the physical figure is a multiple of that — Section 26 (Design an S3-like Object Storage) does the durability arithmetic properly.

Metadata: 50M users × 1,000 files = 50 billion file records. At roughly 500 bytes each (path, size, timestamps, version, owner, permissions, and the list of block hashes) that is 25 TB of metadata. Too large for one database, and sharded by user identifier it partitions cleanly, because almost every query is scoped to one user's namespace.

Change rate: 10M × 20 = 200 million file modifications per day

200M ÷ 100,000 s = 2,000 modifications/s average, ~6,000/s at peak.

The observation that drives everything

Now the important part. Take the average modified file at 5 MB and ask what naive whole-file upload would cost:

200M modifications × 5 MB = 1 PB/day of upload traffic

Against 500 PB of total stored data, the platform would be re-uploading 0.2% of its entire corpus every single day, mostly to store data it already has.

Because the great majority of those modifications are small edits to existing files, not new files. A 40 MB presentation gets one slide changed. A 200 MB video project gets its metadata touched. A 2 MB spreadsheet gets one cell updated. Whole-file upload sends the entire file every time.

Connections

Clients need to hear about changes made elsewhere. If 3 million devices hold a long-lived connection at peak, that is a connection tier of the kind sized in the chat system's requirements lesson — around 30 servers at 100,000 connections each. Much smaller than a chat system, because a sync client tolerates a few seconds of delay in a way a conversation does not.

Block-level storage and deduplication

This is the deep dive and the one new idea this section owns: splitting files into content-addressed blocks so that only changed blocks move and identical blocks are stored once.

The mechanism

Instead of storing a file as one object, split it into blocks — say 4 MB each — and hash each block with a cryptographic hash function such as SHA-256. A file becomes:

report.pptx  →  [ h1, h2, h3, h4, h5, ..., h10 ]     (an ordered list of block hashes)

The metadata database stores the ordered hash list. The block store holds the actual bytes, addressed by hash. Two things follow immediately:

• Only changed blocks travel. The client hashes the file locally after an edit, compares the new hash list with the old, and uploads only the blocks whose hashes are new.
• Identical blocks are stored once. Two files sharing a block share the stored object, because they resolve to the same hash.

The saving, computed

A 40 MB presentation, split into 4 MB blocks, gives 10 blocks. The user edits one slide, changing bytes in one region:

A 10× saving at 4 MB blocks, 80× at 512 KB blocks. Smaller blocks mean better granularity and more metadata: at 512 KB, that 40 MB file needs 80 hashes instead of 10, and a 10 GB video needs 20,000. Block size is the tuning knob, and typical implementations sit in the low megabytes.

Apply the 4 MB figure to the traffic estimated above: 1 PB/day of whole-file upload becomes roughly 100 TB/day, and the difference is bandwidth nobody pays for and time nobody waits through.

The fixed-size block trap

Here is where the naive version fails, and it is worth feeling before the fix.

Fixed-size blocks work when an edit replaces bytes in place. They fail completely when an edit inserts bytes. Insert one byte at the beginning of a file and every subsequent byte shifts by one position. Every block boundary now falls at a different place in the content, every block hash changes, and the client re-uploads the entire file — after doing all the work of hashing it.

Content-defined chunking is the difference between "block-level sync works for overwrites" and "block-level sync works". Name it, because it is the detail that shows you have thought past the diagram.

Cross-user deduplication, and its privacy caveat

If blocks are addressed by content hash, two different users storing the same file resolve to the same blocks. One million users each storing the same 100 MB installer costs 100 MB, not 100 TB.

The saving is enormous and the caveat is real. If the client asks "do you already have block h?" and skips the upload when the answer is yes, then upload speed leaks information: an attacker who has a candidate file can learn whether anyone on the platform already stores it. For an ordinary document that is a mild leak; for a file whose possession is itself sensitive it is not.

Two mitigations, with an honest trade-off:

• Deduplicate server-side only. The client always uploads; the server discards a duplicate after receiving it. This closes the leak and gives up the bandwidth saving, keeping only the storage saving.
• Deduplicate within a user or organisation, not globally. Retains most of the practical saving (people duplicate their own files constantly) and removes the cross-tenant oracle.

Note also that per-user encryption at rest defeats cross-user deduplication entirely, because the same plaintext encrypts to different ciphertext under different keys. That is a genuine tension between two things users want, and saying so is better than pretending it resolves cleanly.