How are attention scores computed in transformers?

What you need to know

A worked example over 4 tokens

A bank bot processes "my card was blocked". We follow the query of the last token, "blocked", as it attends to all four tokens. Use key size d_k = 4, so sqrt(d_k) = 2.

Step 1 — project. Learned matrices give (numbers chosen to be easy):

query("blocked") = [1, 2, 0, 1]key("my")      = [1, 0, 0, 0]      value("my")      = [1.0, 0.0]key("card")    = [1, 2, 1, 1]      value("card")    = [0.0, 1.0]key("was")     = [0, 0, 1, 0.4]    value("was")     = [0.5, 0.5]key("blocked") = [1, 0, 1, 1]      value("blocked") = [1.0, 1.0]

Step 2 — dot products.

q·k(my)      = 1q·k(card)    = 1 + 4 + 0 + 1 = 6q·k(was)     = 0.4q·k(blocked) = 1 + 0 + 0 + 1 = 2

Step 3 — scale by 2. Scores become 0.5, 3.0, 0.2, 1.0.

Step 4 — softmax. exp gives 1.65, 20.09, 1.22, 2.72; they sum to 25.67.

weights = 0.064 (my), 0.782 (card), 0.048 (was), 0.106 (blocked)   sum = 1.0

Step 5 — mix the values.

output = 0.064·[1,0] + 0.782·[0,1] + 0.048·[0.5,0.5] + 0.106·[1,1]       = [0.19, 0.91]

The new vector for "blocked" is mostly made from "card" — the model has attached what was blocked.

The causal mask

"blocked" is the last token, so it can see all four. For "card" (position 2), the scores for "was" and "blocked" are set to minus infinity before softmax. exp(minus infinity) = 0, so those tokens get exactly zero weight.

Why scaling matters

If each vector entry has variance about 1, a dot product over 128 dimensions has a standard deviation of about sqrt(128) ≈ 11. Scores like 30 and 5 make softmax almost one-hot, and its gradient becomes tiny, so learning stalls. Dividing by sqrt(d_k) brings the spread back to about 1.

Cost

For n tokens, QK^T is an n × n matrix, so compute is O(n squared × d). FlashAttention computes the same exact result in tiles without storing the full matrix, which saves memory and time but not the quadratic compute.

A real-life example

A legal-document summariser reads "The Lessee shall indemnify the Lessor ... Such indemnity survives termination." When computing the vector for "indemnity" in the second sentence, its query scores highest against the key for "indemnify" in the first sentence, the same way "blocked" scored 6 against "card" above. After softmax, most of the weight goes there, so the model knows whose indemnity survives. When the summariser once produced "the Lessor shall indemnify the Lessee", inspecting attention in a debugging tool showed the relevant head had split its weight between both parties — a hint that the clause needed to be quoted in the prompt, not paraphrased.

Follow-up questions to expect

• "Why subtract the maximum before softmax?" — exp of large numbers overflows; subtracting the row maximum gives the same result and keeps numbers safe.
• "Where do Wq, Wk and Wv come from?" — They are learned by backpropagation during pretraining, separately for each head and each layer.
• "What happens to padding tokens?" — They are masked out with minus infinity, just like future tokens, so they get zero weight.