Course Content
Transformer Architecture Q&A
6 sections · 60 lessons
What is the embedding matrix, and how does its parameter count compare to the rest of GPT?
What you need to know
The numbers
| Model | V × d | Embedding params | Total | Share |
|---|---|---|---|---|
| GPT-2 small | 50,257 × 768 | 38.6M | 124M | ~31% |
| GPT-2 XL | 50,257 × 1,600 | 80.4M | 1,558M | ~5% |
| Llama 3.2 1B (tied) | 128,256 × 2,048 | 263M | ~1.24B | ~21% |
| Llama-3-8B (untied) | 128,256 × 4,096 | 525M × 2 | 8.03B | ~13% |
For Llama-3-8B the rest is the blocks: about 218M per layer (42M attention with GQA, 176M SwiGLU MLP), times 32 layers, about 6.98B.
Parameters are not compute
- Input embedding — a lookup. Almost zero FLOPs, however large.
- LM head — a real
d × Vmatmul, about2·d·VFLOPs per position that needs logits. At inference that is only the last position.
So the 525M input embedding of Llama-3-8B adds memory but nearly no compute. This is why Kaplan et al. (2020) fit scaling laws on non-embedding parameters, and why the 6·N·D training-cost estimate is more accurate with N excluding the input embedding.
What the rows learn
Rows of tokens used in similar contexts end up close together. That makes the matrix a useful first layer of meaning, but not a finished one: the blocks refine each token's vector heavily based on context.
Rare and unused rows
A row is only updated when its token appears (or, with tying, through the output softmax). Tokens that almost never appear in training can keep near-random rows — the cause of odd "glitch tokens" that made early models behave strangely.
A real-life example
A team designs a 1.5B-parameter model for ten Indian languages. A 50K vocabulary would make Tamil and Hindi text several times longer in tokens than English. A 256K vocabulary fixes that, but at d = 2048 the embedding is 256,000 × 2,048 ≈ 524M parameters — over a third of the model.
They choose a 256K vocabulary with tied weights, accepting that a third of the parameter budget is a lookup table. Their reasoning: the lookup costs almost no compute, and shorter token sequences save far more compute and KV cache at serving time than a smaller vocabulary would save in memory. They check that every language's common words got their own rows, so no big block of the table is left untrained.
Follow-up questions to expect
- "Why is it a lookup and not a matmul?" — Multiplying a one-hot vector by the matrix just selects a row, so indexing does the same thing without the arithmetic.
- "Why do small models tie embeddings more often?" — A second
V × dmatrix would be a large fraction of a small model. - "How does a bigger vocabulary affect serving?" — Fewer tokens per text (less compute and KV cache) but a bigger LM head matmul per generated token.