How text models generate: tokens, transformers and decoding

Almost every text-producing system in this course works the same way: it predicts one token at a time, each prediction conditioned on everything written before it, appending as it goes. That one mechanism explains why generation is slow, why streaming exists, why long prompts cost money, and most of the serving design in Sections 2 through 5.

The loop, step by step

Give the model the text The lighthouse keeper walked to the. Internally:

1. The text is split into tokens and turned into numbers.
2. One forward pass through the network produces a score for every token in the vocabulary — typically 30,000 to 200,000 entries.
3. Those scores are turned into probabilities.
4. One token is chosen from that distribution (the decoding strategies at the end of this lesson cover how).
5. The chosen token is appended to the sequence, and the loop returns to step 2.

The distribution at step 3 might look like this — an illustrative example, not a real model's output:

Pick edge, append it, and the next forward pass runs on The lighthouse keeper walked to the edge. Then again. Then again.

Why this makes generation slow

Step 5 depends on step 4. You cannot compute the tenth token before you know the ninth, because the ninth is part of the ninth's input. Generation is therefore inherently sequential, and no amount of hardware removes that.

Put numbers on it. Suppose one forward pass takes 20 milliseconds. A 300-token answer takes 300 × 20 ms = 6 seconds, and there is no way to shorten it except by making each pass faster or producing fewer tokens.

Compare that with the prompt. The prompt's tokens are all known up front, so they can be processed in a single parallel pass — a 1,200-token prompt does not cost 1,200 steps. This split, called prefill (the prompt, parallel) and decode (the output, sequential), is the single most useful fact for reasoning about generative latency, and the lesson on the inference cost and latency budget builds a whole cost model on it.

The three consequences you will use constantly

• Stream the output. Perceived latency becomes time-to-first-token (often 200–500 ms) instead of time-to-completion (6 seconds). Nothing about the model changed; the user experience changed completely.
• Output length is the cost dial. Halving the answer length halves the decode time and roughly halves the decode cost. Prompt length is far cheaper per token than output length.
• Short-output features are viable where long-output features are not. Smart Compose (Section 2) generates five tokens and must respond in well under 100 ms. The chatbot in Section 4 generates hundreds and buys its time with streaming.

The analogy, and where it breaks

Autoregressive generation is often introduced as "phone keyboard autocomplete, but better". That gets the mechanism right and the implication wrong.

Where it breaks: a keyboard predicts from the last two or three words, while the model conditions on the entire preceding sequence, which is why it can hold a topic across paragraphs. And a second, more important break — the model has no plan. It does not decide on an ending and write towards it. Each token is chosen from what came before, and once emitted it cannot be revised. Coherence over a long answer is an emergent effect of conditioning on a long context, not the execution of an outline. That is why models can write themselves into a corner and then confidently continue out of it.

Transformers, at the level this round needs

The loop above runs on a transformer. You will not be asked to derive attention. You will be asked why a long context is expensive, why one architecture suits translation and another suits chat, and what happens when a conversation outgrows the window. Those three answers all come from one mechanism.

The analogy, then the puncture

Attention is usually explained as "the model decides what to look at". Useful opening: when processing the word it in the trophy did not fit in the suitcase because it was too large, positions corresponding to trophy get more weight than positions corresponding to suitcase.

Now puncture it, because this analogy causes real errors in interviews:

• It is not a decision, it is a blend. Nothing is selected and nothing is discarded. Every position contributes; some contribute more.
• The weights are not reasons. Attention maps look interpretable and are widely over-read. A high weight is not evidence that the model "used" that word to reach its answer, and papers going back years have shown attention weights and true feature importance can disagree. Do not offer attention maps as an explainability story.
• There are many attentions at once. A layer has multiple heads, and a model has many layers, so there is no single map to read.

Encoder-decoder versus decoder-only

Two arrangements, and the choice shows up in Sections 3, 4, and 6.

The practical difference: an encoder-decoder can build a rich bidirectional representation of the source before writing a word, which is worth a lot when the source is fixed and complete. A decoder-only model treats everything as one stream, which is simpler, scales well, and is why most general-purpose models are built that way.

Why long context is expensive

Every position attends to every other position, so the attention work grows with the square of the sequence length. Double the context and attention compute goes up roughly fourfold. Go from 2,000 tokens to 32,000 and the attention term is about 256 times larger.

Two honest qualifications, because this is where candidates overstate.

• At short lengths, attention is not the dominant cost — the dense layers are. The quadratic term only takes over once the sequence gets long.
• Memory tells a different story. The key-value cache (see The inference cost and latency budget) grows linearly with sequence length, and it is often the binding constraint before compute is. An illustrative figure: at roughly 0.5 MB of cache per token, a 16,000-token conversation holds about 8 GB of cache for that one request. Multiply by concurrent users and you see why long context is a capacity-planning problem, not a config flag.

Decoding strategies

At every step the model hands you a probability distribution over the whole vocabulary. Decoding is the rule for turning that distribution into one token, and it changes the character of the output more than most people expect — often more than swapping the model would.

The five rules you need

Greedy. Take the highest-probability token, every time. Deterministic, fast, and prone to loops, because whatever it repeats becomes strong evidence for repeating it again.

Beam search. Keep the k best partial sequences alive (say k = 4), extend each, keep the best k again, and return the highest-scoring complete sequence at the end. It optimises the whole sequence rather than each step, which matters when there really is a best answer. It costs k times the compute and it reliably produces safe, bland text.

Temperature. A dial applied before sampling. Below 1 it sharpens the distribution towards the likely tokens; above 1 it flattens it, giving unlikely tokens a real chance. Temperature 0 degenerates to greedy.

Top-k sampling. Keep only the k most likely tokens, renormalise, sample. Stops the long tail of nonsense from ever being drawn.

Nucleus (top-p) sampling. Keep the smallest set of tokens whose probabilities add up to p (say 0.9), renormalise, sample. Better than top-k because the set adapts: where the model is confident the set is tiny, and where it is genuinely uncertain the set is large.

The same prompt under each

Prompt: Write one sentence describing a lighthouse at dawn. Outputs below are invented for illustration, but the character of each is what you should expect.

Read down the column. Greedy and beam are correct and forgettable. The middle setting is what most writing products ship. The last is unusable in most products and exactly right in a few.

Choosing, and the trap

The rule is about whether the task has a best answer.

• Translation, summarisation, structured output, tool arguments, code: there is a right answer, or at least a narrow band of them. Use greedy or low temperature, sometimes beam search. Section 3 (Google Translate) uses beam search for exactly this reason.
• Creative writing, brainstorming, suggestion variety, anything where two users should not get the same output: sample, with temperature around 0.7–1.0 and nucleus sampling on.

The other knobs worth naming

Repetition and frequency penalties reduce the score of tokens already used, which suppresses loops at the cost of unnatural vocabulary if pushed hard. Stop sequences end generation at a marker rather than at the token limit. Maximum output tokens is your hardest cost control — it is the only setting that puts a ceiling on the bill for a single request.