Describe the Transformer architecture and how a forward pass produces an output?

What you need to know

A forward pass is one trip from token ids to logits. It is easiest to explain by following the shape of the data at each step.

The pass, step by step

ids       (B, T)             B = batch size, T = number of tokensx = E[ids] (+ positions)     (B, T, d_model)   embedding lookuprepeat N times:    x = x + Attn(Norm(x))    mixes information ACROSS positions    x = x + FFN(Norm(x))     transforms EACH position alonex = Norm(x)logits = x @ W_out           (B, T, V)         V = vocabulary size

• Tokenise. Text becomes integer ids, one per sub-word piece.
• Embed. Each id selects one row of an embedding table of shape (V, d_model).
• Add position. Attention on its own ignores order, so position is injected (added vectors, or RoPE rotations inside attention).
• N blocks. Attention lets each token gather information from others. The feed-forward network (FFN) then processes each token's gathered information.
• Output head. A final linear layer maps each d_model vector to V logits — one score per possible next token.

Real sizes

For a model like Llama 3 8B: d_model = 4096, 32 blocks, 32 query heads, a vocabulary of 128,256 tokens. A 100-token prompt therefore produces a logits tensor of shape (1, 100, 128256).

Training versus generation

In training, all T outputs are used: position 1 predicts token 2, position 2 predicts token 3, and so on, all in one parallel pass. A causal mask stops a position from seeing the answer. In generation, only the last position's logits matter; you pick a token, append it, and run again. That is why training is parallel but generation is sequential.

The 2017 paper had an encoder and a decoder. Today's chat LLMs are decoder-only: the same block stack with a causal mask, and no encoder.

A real-life example

A code-completion assistant sees the user type:

def is_even(n):    return n %

Suppose this becomes 12 tokens. The model builds a (1, 12, d_model) tensor, runs it through every block, and gets (1, 12, V) logits. Only row 12 is used: its highest-probability token is 2. The assistant appends it and runs again for ==, then 0.

If the engineer logs all 12 rows, rows 1–11 are the model's guesses for tokens it has already seen — useful for measuring loss, useless for completion. This is also why the KV cache exists: without it, each new token would recompute all earlier rows.

Follow-up questions to expect

• "Where is most of the compute?" — For short and medium contexts, in the matrix multiplications of the FFN and the Q/K/V/output projections. Attention's T × T part only dominates at long context.
• "Why is the whole sequence processed at once?" — Attention is matrix maths over all positions together, so the GPU does it in parallel. RNNs had to go one step at a time.
• "What does the final norm do?" — In pre-norm models the residual stream is never normalised inside the blocks, so one last norm keeps the scale sensible before the output head.