Tokenization - Breaking Down Text

A language model is, underneath everything, a stack of matrix multiplications. Matrices multiply numbers. They do not multiply the word cat. So before a single parameter of the model gets involved, something has to turn a string of characters into a list of integers — and that translation step is where a surprising amount of a model's observable behaviour is decided.

This step is called tokenisation: splitting text into units (tokens) drawn from a fixed vocabulary, then replacing each unit with its integer ID. It sounds like plumbing. It is not. Tokenisation is why a model can spell most words but cannot reliably count the letters in strawberry, why the same paragraph can cost several times as much in Hindi as in English, and why adding a trailing space to your prompt can measurably degrade the answer.

The best way to understand it is to try the two obvious designs first and watch both of them break.

Attempt one: one integer per word

The natural first idea is to split on whitespace and punctuation and give every distinct word an ID. the is 1, cat is 2, sat is 3, and so on.

Now count how many IDs you need. A large English web corpus contains somewhere between one and two million distinct word forms once you include names, typos, URLs, product codes, hashtags and inflections. Suppose you cap it at 500,000 and throw the rest away.

Two things go wrong immediately, and both are fatal.

The vocabulary is enormously expensive. Every token in the vocabulary needs a row in the embedding matrix — a learned vector, typically 4,096 numbers wide in a mid-sized model. That is $500{,}000 \times 4{,}096 \approx 2.05$ billion parameters spent purely on the lookup table, before the model has a single attention layer. The output layer, which must produce a score for every possible next token, costs the same again.

Everything outside the vocabulary becomes a hole. The word unhappiness might not make the top 500,000. Under word tokenisation it becomes <UNK> — a single "unknown" token. The model receives no signal that it contains un-, happy and -ness, three pieces it knows perfectly well. It is told only "some word I have never seen". Every new product name, every misspelling, every rare surname collapses into the same undifferentiated blob.

Word-level tokenisation forces an impossible choice: an unaffordable vocabulary, or a model that goes blind exactly where language is most productive — at new and rare words.

Attempt two: one integer per character

Swap to the opposite extreme. Vocabulary = the characters. For English, roughly 100 symbols covers letters, digits and punctuation. The embedding table becomes trivial. Nothing is ever unknown.

Now the cost moves somewhere else: sequence length.

The sentence "The quick brown fox jumps over the lazy dog" is 9 words but 44 characters. Character tokenisation makes every sequence roughly 4–5 times longer. That matters enormously, because the attention mechanism inside a transformer compares every token to every other token — its cost grows with the square of the sequence length. Making sequences 5× longer makes attention roughly 25× more expensive.

Worse, the model has to spend its early layers relearning something we already know: that c, a, t in that order form a meaningful unit. Those are layers not spent on meaning.

The third row is what every production model actually uses. The insight is simple: let common words stay whole, and let rare words break into reusable pieces. The question is how to decide, automatically, which pieces are worth keeping.

Byte-Pair Encoding, worked by hand

Byte-Pair Encoding (BPE) answers that question with one greedy rule, applied over and over: find the most frequent adjacent pair of symbols in the corpus and merge it into a single new symbol. Repeat until you have as many symbols as you want.

Let us actually run it. Here is a toy corpus of four words with their counts:

low      × 5lower    × 2newest   × 6widest   × 3

Split every word into characters and append an end-of-word marker (written here as _) so the algorithm can tell a word-final t from a word-internal one:

l o w _        × 5l o w e r _    × 2n e w e s t _  × 6w i d e s t _  × 3

Now count every adjacent pair, weighted by word frequency:

Three pairs tie at 9. Break the tie by taking the first, e s, and merge it everywhere:

Merge 1: e + s → esl o w _          × 5l o w e r _      × 2n e w es t _     × 6w i d es t _     × 3

Recount. Now es t occurs 9 times and is the winner. Merge it. Then est _ occurs 9 times. Merge it. Then l o at 7, then lo w at 7. Five merges in, the state is:

Notice what BPE has discovered without being told anything about English: the suffix -est and the stem low. Nobody wrote a morphology rule. Frequency did the work.

The pay-off: a word it has never seen

The training corpus never contained the word lowest. Encode it anyway. Start from characters and apply the learned merges in the order they were learned:

l o w e s t _  → merge 1 (e+s):      l o w es t _  → merge 2 (es+t):     l o w est _  → merge 3 (est+_):    l o w est_  → merge 4 (l+o):      lo w est_  → merge 5 (lo+w):     low est_Result: ["low", "est_"]  — 2 tokens, both meaningful

This is the whole trick. An unseen word decomposes into pieces the model has rich, well-trained representations for. There is no <UNK>, no information loss, no cliff edge.

BPE gives you a fixed-size vocabulary that nonetheless covers infinite text, by trading token count for coverage: familiar words cost one token, unfamiliar ones cost several.

Byte-level BPE: closing the last hole

Plain BPE still has a gap. If your base alphabet is "characters seen during vocabulary training", an emoji or a rare Chinese character from outside that set is unrepresentable. Byte-level BPE fixes this by starting from the 256 possible byte values instead of characters. Any text in any script is a sequence of bytes, so every possible input is representable — worst case, one token per byte. GPT-2 introduced this, and it is the design behind the tokenisers of the GPT family and Llama 3 onwards. SentencePiece-based tokenisers (Llama 2, Gemma and others) reach the same guarantee a different way, with byte fallback: any character missing from the vocabulary is spelled out as its raw bytes. Either way, modern tokenisers have no <UNK>.

The base-256 alphabet is also why you sometimes see a token that looks like mojibake in a tokeniser dump: a single UTF-8 character can span 2–4 bytes, and a merge may cover only part of it.

The other two algorithms you will meet

BPE picks merges by raw frequency. Two alternatives pick them differently.

WordPiece (used by BERT and relatives) uses the same merge loop but a different scoring rule. Instead of choosing the most frequent pair, it chooses the pair that most increases the likelihood of the corpus under a unigram model — which works out to maximising

The denominator matters. Consider a pair like t + h: very frequent together, but t and h are also extremely frequent individually, so the ratio is unremarkable. Compare q + u: q almost never appears without u, so the ratio is huge. WordPiece merges qu eagerly and th reluctantly; BPE does the reverse. In practice WordPiece produces slightly more linguistically tidy pieces, marked with ## for word-internal position: playing → play, ##ing.

SentencePiece with a unigram model (used by T5, and by many multilingual models) works backwards. It starts with a deliberately oversized candidate vocabulary, then repeatedly removes the candidates whose deletion hurts corpus likelihood least, until it reaches the target size. Crucially it treats the input as a raw character stream including spaces — space is encoded as a visible marker, usually ▁ — so it needs no language-specific pre-tokenisation. That is why it is the default for languages such as Japanese and Thai, which do not put spaces between words.

Seeing it in code

1from transformers import AutoTokenizer23tok = AutoTokenizer.from_pretrained("gpt2")45text = "Tokenisation is unglamorous but decisive."6ids = tok.encode(text)78print(len(ids))                       # 109print(tok.convert_ids_to_tokens(ids))10# ['Token', 'isation', 'Ġis', 'Ġun', 'g', 'lam', 'orous', 'Ġbut', 'Ġdecisive', '.']11# (Ġ is how GPT-2's tokeniser displays a leading space)1213# Round-trip is exact14assert tok.decode(ids) == text

Two details in that output are worth pausing on.

First, the leading spaces. 'Ġis' — that is, ' is' — is a single token that includes the space before it. In byte-level BPE, space is not a separator that gets stripped — it is part of the token. This is the source of a real and easily-missed failure mode:

1tok.encode("The capital is")     # [464, 3139, 318]2tok.encode("The capital is ")    # [464, 3139, 318, 220]  ← trailing ' ' token34# The model was trained on text where ' Paris' follows ' is' directly.5# After a lone space token 220, it must now predict a continuation that6# starts *without* a space - a distribution it has seen far less often.

Never end a prompt with a trailing space. You have pushed the model onto a token boundary it rarely saw during training, and completion quality drops for no reason a user could ever guess.

Second, unglamorous — a perfectly ordinary word — costs four tokens ( un, g, lam, orous), while decisive costs one. That is not a judgement about importance. It is purely a statement about frequency in the tokeniser's training corpus.

What tokenisation determines about the model

It sets a large fraction of the parameter budget

The embedding matrix has one row per token, one column per model dimension. For GPT-2 small: $50{,}257 \times 768 = 38{,}597{,}376$ parameters — about 31% of the model's 124M total, spent entirely on the lookup table. For a model with a 128,256-token vocabulary and 4,096-dimensional embeddings: $128{,}256 \times 4{,}096 \approx 525$ million parameters for the input embedding, and the same again for the output projection if the two are not tied.

Growing the vocabulary shortens sequences (cheaper attention) but inflates the embedding and output layers, and makes the final softmax over all tokens more expensive. That trade-off is the reason vocabulary sizes cluster in the 32k–256k range rather than being pushed arbitrarily high.

It decides what your text costs

Roughly, English prose runs about 4 characters per token — around 0.75 tokens per word. But that average hides enormous variation:

That last row has a real consequence. If a tokeniser was fitted mostly on English text, a paragraph in a low-resource language can consume two to four times as many tokens as its English translation. The user pays more per API call, fits less into the context window, and gets a model whose per-token representations for that language are thinner. This is often called the tokenisation tax, and it is a genuine equity problem, not a rounding error.

Newer tokenisers shrink the tax by spending more of a larger vocabulary on other scripts. Measured on one Hindi sentence ("प्रौद्योगिकी ने हमारे जीवन को पूरी तरह बदल दिया है।") and its 7-token English translation: GPT-2's 50,257-token vocabulary needs 82 tokens, cl100k_base (about 100,000 tokens, the GPT-4 tokeniser) needs 54, and o200k_base (about 200,000 tokens, used from GPT-4o onwards) needs 14. The gap has narrowed from roughly 12× to 2× — which is also a reason vocabularies have grown from 32k towards 128k–256k. It has not disappeared, and it is worse still for lower-resource languages, so measure your own.

It explains the model's blind spots

Ask a model how many rs are in strawberry. It often gets this wrong, and the reason is structural rather than a lack of intelligence. The model never receives the letters. It receives two or three integer IDs standing for chunks like str, aw, berry. Asking it to count letters is like asking you to count the strokes in a Chinese character you only recognise as a whole shape. It can be done — through memorised knowledge about spelling — but it is indirect, and it fails in the ways indirect reasoning fails.

The same reasoning explains arithmetic difficulties. If a tokeniser splits 1234 as 12 + 34 but splits 1235 as 123 + 5, the model sees two numerically adjacent quantities as structurally unrelated inputs. Digit alignment for column-wise addition becomes something the model must learn to reconstruct rather than something it is handed. Newer tokenisers deliberately split digits into fixed groups to make this consistent. GPT-2 splits 1234567 as 123 + 45 + 67; cl100k_base and o200k_base pre-split any run of digits into chunks of at most three, left to right, giving 123 + 456 + 7; some SentencePiece models split every digit on its own. It is a change to the tokeniser, not the architecture, and it makes numbers far more regular for the model to learn.

What to do with this when you build something

Measure tokens, never characters. If you are budgeting a context window, batching documents, or estimating cost, run the real tokeniser over real samples of your real data. A rule of thumb calibrated on English news articles will mislead you badly on JSON payloads, medical codes or Vietnamese.

1from transformers import AutoTokenizer23tok = AutoTokenizer.from_pretrained("gpt2")45for label, s in [6    ("prose", "The committee reached a decision on Thursday."),7    ("code",  "def f(x):\n    return [i**2 for i in range(x)]"),8    ("id",    "order-8f14e45fceea167a5a36dedd4bea2543"),9]:10    n = len(tok.encode(s))11    print(f"{label:6s} {len(s):3d} chars  {n:3d} tokens  {len(s)/n:.1f} chars/token")

Keep prompts on clean boundaries. End on a word, a colon, or a newline — never on a bare space. If you are building few-shot prompts, keep the separator between examples byte-identical every time, so the model sees one consistent pattern rather than several near-misses.

Do not ask a model to do character-level surgery on text unless you give it a way out. "Reverse this string", "count the vowels", "does this word contain a double letter" are all fighting the tokeniser. Either accept the error rate, or have the model emit code that does the manipulation, where the operation happens on real characters rather than on IDs.

And when you swap a model for another, re-measure everything. A different tokeniser means different token counts, different costs, different effective context length, and different sensitivities — even if the prompt text is byte-for-byte identical.