What are sparse/local attention patterns, and how were they used in models like GPT-3?

What you need to know

The cost being attacked

Attention work per layer grows with T². For a 4096-wide model, the attention matmuls cost about 4·T²·d FLOPs, while the weight matmuls cost about 2·T·(params per layer):

T = 8,192:    attention ≈ 1.1 TFLOP,  weights ≈ 3.6 TFLOP  → attention ~30%T = 131,072:  attention ≈ 281 TFLOP,  weights ≈ 57 TFLOP   → attention ~5× the weights

At long context, attention dominates, and the KV cache grows with T in every layer.

The patterns

• Local / sliding window — each token sees the previous w tokens. Cost T·w instead of T².
• Strided / dilated — attend to every k-th token, so a few layers reach far with few links (Child et al., 2019, the Sparse Transformer).
• Global tokens — a few positions attend to everything and are seen by everything, giving a shortcut for long-range information (Longformer, Beltagy et al., 2020).
• Random links — BigBird (Zaheer et al., 2020) adds random edges on top of window plus global.

1import numpy as np2T, w = 12, 43i, j = np.indices((T, T))4causal = j <= i5local = causal & (i - j < w)             # each token sees itself + 3 before6print(causal.sum(), local.sum())         # 78 42 — score entries computed7print(local.astype(int)[-1])             # [0 0 0 0 0 0 0 0 1 1 1 1]

At 12 tokens the saving is small; at 32K tokens with a 1K window, a local layer computes about 1/16 of the dense scores.

How information travels past the window

A token in layer 2 sees tokens that, in layer 1, saw another window back. After L layers the reach is about L × w. The Mistral 7B paper uses exactly this argument: a 4,096-token window over 32 layers gives a theoretical span of about 131K tokens. In practice, information that must hop many times gets weaker.

GPT-3, then and now

GPT-3's paper says it uses the same architecture as GPT-2 except for "alternating dense and locally banded sparse attention patterns in the layers of the transformer, similar to the Sparse Transformer". That was the only structural change, at a 2,048-token context.

Then dense attention plus FlashAttention won for mainstream lengths, because irregular sparse patterns are hard to make fast on GPUs. Local windows came back in a simpler, GPU-friendly form:

• Mistral 7B — sliding window of 4,096 in every layer.
• Gemma 2 and Gemma 3 — local and global layers interleaved; Gemma 3 uses five local layers (1,024-token window) per global layer.
• gpt-oss (OpenAI, 2025) — alternating dense and locally banded layers, like GPT-3.

The other route is replacing attention in most layers with a state-space model such as Mamba (Gu and Dao, 2023), whose state does not grow with T. Hybrids such as Jamba (AI21, 2024) keep a few attention layers among many Mamba layers.

A real-life example

A code-completion assistant sends 32K tokens of repository context. Its model has 48 layers: 40 local layers with a 1,024-token window and 8 global layers.

all global:  48 layers × 32,768 tokens of cacheinterleaved:  8 × 32,768 + 40 × 1,024 → about 19% of the all-global cache

The cache is about 5× smaller, so each GPU holds about 5× more concurrent completions. The trade-off shows up when the model needs a function defined 20K tokens earlier: only the 8 global layers can look at it directly, so the team tests long-range recall on their own repositories before switching.

Follow-up questions to expect

• "Is a sliding window the same as truncating the prompt?" — No. Earlier tokens influenced the hidden states that are still in the window, so their information is carried forward, though weakened.
• "What are attention sinks?" — Models put a lot of weight on the first tokens; StreamingLLM (Xiao et al., 2023) showed a sliding window works far better if it always keeps those first few tokens.
• "Why did dense attention win?" — Dense attention runs as regular matrix tiles that GPUs are built for; sparse patterns waste bandwidth on gathers and often lose quality.