What is multi-head attention?

What you need to know

Splitting the width

Suppose the model width is 4,096 and there are 32 heads. Each head works with vectors of 4,096 / 32 = 128 dimensions.

head_i  = Attention(X·Wq_i, X·Wk_i, X·Wv_i)        # each 128-dimoutput  = Concat(head_1 ... head_32) · Wo          # back to 4,096

Because the heads are narrower, the total compute is about the same as one 4,096-wide head.

Why several heads help

A single softmax gives one set of weights per token. But a word needs several kinds of context at once: its grammatical subject, the noun a pronoun refers to, nearby words for phrases. Different heads can attend to different tokens for the same position. Studies of trained models find heads that track the previous token, heads that copy repeated patterns, and heads that follow syntax — though many heads are not neatly interpretable, and some can be removed with little loss.

KV cache arithmetic

For a model with 32 layers, key–value heads of dimension 128, and 16-bit (2-byte) numbers:

bytes per token = 2 (K and V) x layers x kv_heads x head_dim x 2 bytes32 KV heads (standard multi-head): 2 x 32 x 32 x 128 x 2 = 512 KB per token8 KV heads  (grouped-query):       2 x 32 x  8 x 128 x 2 = 128 KB per token1 KV head   (multi-query):         2 x 32 x  1 x 128 x 2 =  16 KB per token

An 8,000-token conversation needs 4 GB of cache with standard heads, but only 1 GB with 8 KV heads. For 50 users at once, that is 200 GB versus 50 GB of GPU memory.

The variants

• Multi-query attention (MQA) — all query heads share one key–value head. Smallest cache, some quality loss.
• Grouped-query attention (GQA) — query heads are grouped, each group shares one KV head (e.g. 32 query heads, 8 KV heads). The common middle ground in Llama, Mistral and Qwen models.
• Multi-head latent attention (MLA) — used in DeepSeek's models; compresses keys and values into a small latent vector to shrink the cache further.

A real-life example

A bank self-hosts an open-weight 8B model for its support bot on one 80 GB GPU. The weights take about 16 GB, leaving roughly 60 GB for the KV cache. With the model's grouped-query attention (8 KV heads, 128 KB per token), that fits about 460,000 cached tokens — roughly 55 conversations of 8,000 tokens each at the same time.

A developer suggests an older model of the same size with standard 32-head attention. The same 60 GB would hold only about 14 such conversations, so the bank would need four times as many GPUs for the evening peak. The attention design, not the parameter count, decides the serving bill here.

Follow-up questions to expect

• "Why not use one big head?" — One softmax per token can express only one weighting; several heads give several simultaneous views at the same cost.
• "Does GQA hurt quality?" — Slightly compared with full multi-head at the same size, but far less than MQA, and the memory saving is large, so almost all modern LLMs use it.
• "How many heads do large models have?" — Typically 32 to 128 query heads, with 8 or so KV heads when GQA is used.