Designing a URL shortener, with the numbers

Almost every article about designing a URL shortener draws the same four boxes — client, app server, database, cache — connects them with arrows, and calls it a design. The boxes are not wrong. They are just not the part that requires any thought. What actually decides whether this system works is a handful of numbers and one genuinely interesting problem: how do you mint a short, unique code, quickly, on many machines at once, without any of them handing out the same one?

So let us build it properly. Requirements first, then arithmetic, then the code-generation problem, then the parts that only show up under load. Every number below is an assumption I am stating openly rather than a measurement — the point is the method, and you should redo the sums with your own traffic.

What the service actually has to do

Take a long URL, return a short one. Given the short one, send the visitor to the original. That is the whole product. Around it sit a few features that turn out to shape the design more than the core does: an optional custom alias so someone can have go.example.com/spring-sale instead of go.example.com/7Kd2a9, optional expiry, and click analytics.

The non-functional requirements matter more here than usual, because they pull in different directions:

• It is overwhelmingly read-heavy. A link is created once and followed many times. Treat writes and reads as two different systems with different budgets.
• The redirect must be fast. It sits in front of somebody else's page load. Every millisecond you spend is added to a journey the user did not ask to take.
• Links must not break. This is the one nobody writes down, and it is the strictest constraint in the system. A short link gets pasted into an email, printed on a poster, embedded in a document you will never see again. Once issued, a code can never be reassigned to a different target, and "we lost that row" is not a recoverable error.

That last point quietly rules out a few clever ideas later on, so it is worth fixing in mind now.

Do the arithmetic before you draw any boxes

Pick numbers, state them as assumptions, and follow them through. Say one million new links a day, and a read-to-write ratio of one hundred to one — modest for this kind of service.

Two things fall out immediately. The write path is trivial — fifty-eight writes a second is a single unremarkable database doing almost nothing. The read path is four orders of magnitude larger, and it is the only number that should influence your architecture. If you find yourself optimising link creation, you are optimising the wrong half.

Now storage. A row holds the short code, the original URL, a creation timestamp, an optional expiry, and an owner reference. Long URLs vary wildly, but a hundred bytes is a fair average. Call it two hundred bytes per row once you include index overhead:

1,000,000 links/day x 365 days x 5 years = 1,825,000,000 rows1,825,000,000 rows x 200 bytes           = 365 GB

Three hundred and sixty-five gigabytes over five years. That is a single machine with room to spare. This is the most useful outcome of doing the arithmetic: the honest answer is that one well-indexed database handles the storage, and the interesting engineering is entirely in the read path and the code generation. A design that opens with sharding has skipped the step that would have told it not to.

One more sum, because it determines how long your codes are. With sixty-two characters available:

62^5 =            916,132,83262^6 =         56,800,235,58462^7 =      3,521,614,606,208

Five characters does not cover 1.8 billion links. Six covers it with a factor of thirty in hand. Six characters it is — and notice that this is a decision derived from a number rather than picked because it looked tidy.

Three ways to mint a short code

This is the real design problem. There are three families of answer, and the differences between them are not cosmetic.

Hash the URL and truncate. Run the long URL through a hash, take the first few characters, encode them. It is stateless, which is genuinely attractive — any server can compute a code without coordinating with anything. Two problems. Truncating a hash to thirty-six bits makes collisions a routine event rather than a curiosity, so you need a read-before-write check and a retry strategy anyway, which throws away the statelessness you were buying. And hashing is deterministic, so the same URL always yields the same code. Sometimes that is a feature: you deduplicate. Usually it is a bug, because two customers shortening the same public URL now share a link, and one of them deleting it or reading its analytics affects the other.

Generate randomly and check. Pick six random characters, see whether the row exists, retry on conflict. Simple, and the codes are unguessable, which matters if a link might be sensitive. The cost is a read before every write, and — more subtly — collision probability climbs as the keyspace fills. At 1.8 billion rows out of 56.8 billion slots you are only three percent full, so retries stay rare. Ten times more traffic and that arithmetic starts to hurt.

Encode a counter. Keep a monotonic integer, encode it in base 62. No collisions at all, by construction — this is the only option where uniqueness is a property of the design rather than something you check for afterwards. Here is the encoding, which is the entire mechanism:

ALPHABET = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"BASE = len(ALPHABET)def encode(n):    if n == 0:        return ALPHABET[0]    out = []    while n > 0:        n, rem = divmod(n, BASE)        out.append(ALPHABET[rem])    return "".join(reversed(out))def decode(code):    n = 0    for ch in code:        n = n * BASE + ALPHABET.index(ch)    return n

Running that over a few values shows the shape, and confirms the round trip:

              0 -> 0        (1 chars)             61 -> Z        (1 chars)             62 -> 10       (2 chars)      1,000,000 -> 4c92     (4 chars) 56,800,235,584 -> 1000000  (7 chars)

The counter approach has one real drawback, and it is worth stating clearly rather than burying: sequential counters produce enumerable codes. Anyone can request 4c90, 4c91, 4c92 and walk your entire corpus, which leaks both your customers' links and your link volume to a competitor. If that matters — and for a business product it usually does — you keep the counter for uniqueness and break the ordering, either by permuting the integer with a reversible keyed transform before encoding, or by mixing in a few random characters. You get collision-free generation and unguessable output. What you must not do is assume a short code is a security boundary; if a link is genuinely sensitive, it needs authorisation, not obscurity.

Handing out ranges so servers never collide

A counter sounds like it forces every write through one place. It does not, and the fix is the same trick databases use for primary keys: hand out blocks. Each application server claims a range of ten thousand values, serves them from memory, and comes back for another block when it runs low.

class CodeAllocator:    """Claims blocks of counter values so each server mints codes locally."""    def __init__(self, store, block_size=10_000):        self.store = store            # atomic increment-and-return        self.block_size = block_size        self.next_id = 0        self.limit = 0    def next_code(self):        if self.next_id >= self.limit:            # One round trip per block, not per link.            end = self.store.increment_by(self.block_size)            self.limit = end            self.next_id = end - self.block_size        code = encode(self.next_id)        self.next_id += 1        return code

The shared counter is now touched once per ten thousand links rather than once per link — at fifty-eight writes a second, that is roughly one coordination call every three minutes. The trade is that a server crash abandons the rest of its block, so you burn unused codes. Given thirty times more keyspace than you need, that is a cost worth paying without a second thought.

The data model is one lookup

The redirect asks exactly one question: given this code, what is the target? That is a key-value lookup, and the schema should admit it. The short code is the primary key, and the read path touches nothing else — no join to a users table, no analytics table, no permission check.

This is worth being disciplined about, because features accrete. Someone wants the owner's plan tier shown on the redirect, someone else wants a per-link rule evaluated, and each addition puts another query in front of a page load. Keep the redirect path a single point lookup and let everything else happen elsewhere, asynchronously.

You will also want an index on owner and creation time so people can list and manage their links — but note that this index serves the dashboard, not the redirect, and the two have completely different performance requirements.

Why the redirect is a 302

This looks like trivia and is not. A 301 tells the browser the move is permanent, and browsers take that literally: they cache it, often aggressively, and stop asking you again. Which sounds efficient, and costs you two things you cannot get back.

First, your analytics. If the browser never contacts you, you never see the click. Your numbers quietly become a count of first-ever visits per browser rather than a count of clicks, and nobody notices until the reports are used for something that matters.

Second, your ability to change anything. Expiring a link, correcting a typo in a target, or disabling a link that now points somewhere malicious all become impossible for every browser holding the cached redirect. For a poster printed with your link, that is indefinite.

So most shorteners return 302, accept the extra request, and keep control. The exception is a link you are certain will never change and never needs measuring, which in practice is almost none of them. If you do use 301, set cache headers deliberately rather than letting the default decide for you.

Custom aliases and the reservation race

Custom aliases share a namespace with generated codes, which creates two problems.

The first is collision between the two schemes: a generated code could theoretically equal an alias someone already holds. Solve it by construction rather than by checking — put aliases and generated codes in ranges that cannot overlap, for instance by requiring generated codes to be exactly six characters and aliases to be seven or more, or by reserving aliases in a separate table consulted first.

The second is a genuine race. Two people submit spring-sale at the same moment, both requests check for existence, both find nothing, both insert. Checking and then inserting is not atomic, so the check proves nothing. The uniqueness constraint on the column is what actually enforces this: attempt the insert, catch the violation, and return "that alias is taken" to whoever lost. That pattern — let the database arbitrate rather than checking first — is the right instinct for any uniqueness problem under concurrency.

You also need a reserved list. Someone will try to register login, admin, api, or static, and if your routing puts aliases at the domain root, that alias will shadow a real route.

Counting clicks without slowing the redirect

The naive version increments a counter on the row during the redirect. At peak that is roughly six thousand writes a second against the hottest rows in your database, all to make a number in a dashboard slightly fresher. It is the single easiest way to turn a fast read path into a slow one.

Instead, get the redirect out of the way first and record the click afterwards. Emit an event — code, timestamp, referrer, coarse geography — onto a buffer, and let a separate consumer aggregate it. The redirect does not wait for that to succeed.

def redirect(code, cache, db, events):    target = cache.get(code)    if target is None:        target = db.lookup(code)          # single point lookup        if target is None:            return 404        cache.set(code, target, ttl=3600)    # Fire and forget: the visitor never waits on analytics.    events.emit_nowait({"code": code, "at": now()})    return 302, target

The consequence is that click counts become eventually consistent and slightly approximate. Say so in the product rather than pretending otherwise — "updated every few minutes" is an honest and entirely acceptable promise, and it buys you a redirect path that does one read and nothing else.

On caching: the hot set is what makes this cheap. Links skew hard toward recent — the ones being actively shared take almost all the traffic. Thirty days of links at these volumes is about six gigabytes, which fits comfortably in a cache tier, whereas the full five-year corpus at 365 GB does not. So cache recency, not everything, and let the long tail fall through to the database, where a point lookup on a primary key is fast anyway.

What gives way first

Under real load, the failures arrive in a predictable order, and none of them are the ones the box diagram suggests.

The first is a single link going viral. One code now takes a large share of your six thousand requests a second, and if your cache is sharded by key, that traffic lands on one node. Local in-process caching in front of the shared tier absorbs this well, precisely because a hot key is the case where every server wants the same answer.

The second is the analytics pipeline falling behind. It is decoupled, so it fails quietly — the redirects keep working while the queue grows and the dashboards drift. Monitor consumer lag and the age of the oldest unprocessed event, not just whether the consumer is running.

The third is expiry cleanup. If you delete expired rows with one large query, you will eventually run it against a table big enough to cause real trouble. Delete in bounded batches, and treat expiry as a filter on read as well, so a link is dead the moment it should be rather than whenever the sweeper next runs.

What to measure: redirect latency at the ninety-ninth percentile rather than the mean, cache hit ratio, the 404 rate — a rise means either dead links being shared or someone enumerating your keyspace — and the age of the oldest unprocessed analytics event. If those four are healthy, the system is doing its job, and you will notice the trouble before your users write to tell you about it.