The Problem with Sequential Models

Here is a translation system from 2014. You feed it a French sentence, it reads the words one at a time with a recurrent network, and after the last word it hands the decoder a single vector of 512 numbers. Everything the sentence meant — every name, every clause, every negation — has to survive inside those 512 floats.

On short sentences it works beautifully. Then someone feeds it a 40-word legal clause and the output turns to mush: names get dropped, a ne...pas vanishes and the translation asserts the opposite of the original. The team plots BLEU score against source-sentence length and finds a cliff: the curve is flat and healthy up to about 20 words, then falls off a shelf. This was measured, published, and reproduced. It was not a bug in anyone's code.

That cliff is one of three structural problems baked into recurrent sequence models. They are not implementation defects you can engineer around with a better optimiser or more layers. They come from the shape of the architecture itself. Understanding them precisely is the whole reason attention exists, so it is worth doing the arithmetic rather than taking it on faith.

What a sequential model actually does

A recurrent neural network (RNN) processes a sequence by maintaining a hidden state — a fixed-size vector that acts as its running memory. At each position it combines the previous state with the current input to produce a new state:

Here $x_t$ is the embedding of the token at position $t$, $h_{t-1}$ is everything the model remembers so far, and $W_{hh}$, $W_{xh}$ are weight matrices shared across every position. That sharing is the point: one set of weights handles a sequence of any length.

The LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit) are refinements of the same idea. They add multiplicative gates — small learned sigmoids that decide how much of the old state to keep and how much new information to write. An LSTM cell carries a separate cell state $c_t$ that is updated additively:

where $f_t$ is the forget gate, $i_t$ the input gate, and $\odot$ is elementwise multiplication. The additive update is a genuine improvement — we will see exactly why in a moment — but it changes the constant factor, not the structure.

Every recurrent architecture, from vanilla RNN to the most elaborate gated cell, shares one commitment: position $t$ cannot be computed until position $t-1$ has finished. Everything that follows is a consequence of that single line.

Problem 1: the sequential bottleneck

Why the hardware sits idle

A modern GPU is a machine for doing tens of thousands of multiply-accumulates simultaneously. An A100 can sustain roughly $3 \times 10^{14}$ floating-point operations per second in half precision. To get anywhere near that, you must hand it enormous chunks of independent arithmetic at once.

An RNN cannot. Take a realistic setup: batch of 64 sequences, length 512, hidden size 512. One timestep is a matrix multiply of shape $(64 \times 512) \times (512 \times 512)$, which is

At $3 \times 10^{14}$ FLOP/s that is about 0.1 microseconds of actual work. But launching a GPU kernel costs roughly 5–10 microseconds of overhead. So the chip does 0.1 µs of maths and then waits 10 µs for the next instruction — a utilisation of about one percent. Multiply by 512 timesteps and you have spent roughly 5 milliseconds per layer per batch, essentially all of it in launch overhead and memory latency.

Now do the same total arithmetic as one big batched matmul, which is what a transformer layer does. All 512 positions go through their projections in a single kernel: $512 \times 64 \times 512 \times 512 \times 2 \approx 1.7 \times 10^{10}$ FLOPs, one launch, near-peak utilisation. Same FLOPs, roughly 100× less wall-clock time.

The complexity picture

It is tempting to conclude that RNNs are simply cheaper in raw operation count and transformers just buy speed with hardware. Partly true, and worth quantifying. Let $T$ be sequence length and $d$ the model dimension.

Put numbers in, with $d = 512$:

So for typical sentence lengths, attention is not even more expensive in FLOPs — and the FLOPs it does spend can all be issued at once. The sequential-steps row is the one that decides training time, and it is the row where recurrence loses by a factor of $T$ regardless of how the FLOPs compare.

Problem 2: gradients that vanish or explode

Where the gradient comes from

Training uses backpropagation through time: unroll the recurrence into $T$ copies of the same layer and backpropagate. If the loss at position $T$ depends on the input at position 1, the gradient must travel back through every intermediate state. By the chain rule:

That is a product of $T-1$ Jacobian matrices. For the vanilla RNN, each factor is

because the derivative of $\tanh(z)$ is $1 - \tanh^2(z)$, which is at most 1 and typically much less once the units saturate.

What goes wrong, in numbers

Suppose the effective per-step multiplier — the product of the weight matrix's dominant singular value and the average $\tanh$ derivative — settles at 0.9. Then over $T$ steps the gradient scales by $0.9^{T}$:

At distance 100 with a multiplier of 0.9, the gradient arriving at position 1 is 0.0027% of the gradient at position 100. In float32 that is not literally zero, but it is drowned out: the update to $W_{hh}$ is dominated entirely by short-range dependencies, and the long-range signal contributes noise. The model does not learn the long dependency slowly — it never learns it at all, because the short-range terms move the weights faster than the long-range term can accumulate.

The multiplier 1.1 is the opposite failure. A gradient scaled by 13,780 produces a parameter update that hurls the weights out of the region where the loss is finite. You see it as a loss curve that is decreasing normally and then reports nan on a single step.

Vanishing and exploding gradients are the same phenomenon — repeated multiplication of a number that is not exactly 1 — viewed from either side of the knife edge.

A concrete failure case

Consider language modelling on this sentence:

The keys to the cabinet that stood in the hallway of my grandmother'shouse, which she had bought shortly after the war, ___ on the table.

The blank is were, not was, because the subject is keys (plural), 26 tokens back. To learn this, the model needs gradient to flow from the verb all the way to keys. With a per-step multiplier of 0.9 that is $0.9^{26} \approx 0.065$ — 6.5% of the signal. Meanwhile the nearest noun, war (singular), is 4 tokens away and gets $0.9^{4} \approx 0.66$, ten times as much gradient. The model learns the wrong rule — agree with the closest noun — because that rule receives ten times the training signal.

The partial fixes, and where they stop

Problem 3: the fixed-size memory bottleneck

The encoder-decoder translation setup makes the third problem sharpest. The encoder RNN reads the source sentence and produces hidden states $h_1, \dots, h_T$. The decoder is then initialised from $h_T$ alone — one vector, usually 512 or 1024 dimensions. Every hidden state before it is discarded.

Count what has to fit. A 50-token source sentence drawn from a 50,000-word vocabulary carries at minimum $50 \times \log_2(50{,}000) \approx 50 \times 15.6 \approx 780$ bits of lexical identity, before you count word order, syntactic structure, or coreference. A 512-dimensional float32 vector has 16,384 raw bits, but its usable capacity is far smaller: the values are continuous, noisy, and passed through a decoder that must be smooth enough to generalise. Empirically, the effective capacity runs out somewhere around 20–30 tokens.

This is exactly the cliff in the opening scenario. The encoder is a lossy compressor with a fixed output size, and translation quality tracks how much got lost.

What makes attention different

Attention removes the compression step. Instead of forcing everything through $h_T$, keep all $T$ encoder states and let the decoder build a different summary at every output step:

The weights $\alpha_{ij}$ say how much output position $i$ should draw on input position $j$. They are computed fresh for each $i$, so the "summary" is task-specific and query-specific rather than one-size-fits-all.

Work through a tiny example. Source: le chat noir dort, four encoder states $h_1 \dots h_4$. The decoder is producing the English word black, and the alignment weights come out as:

They sum to $0.03 + 0.06 + 0.88 + 0.03 = 1.00$, as a softmax output must. The context vector is $c_3 = 0.03 h_1 + 0.06 h_2 + 0.88 h_3 + 0.03 h_4$ — overwhelmingly $h_3$, the state that read noir. Nothing told the model that noir means black; the alignment was learned end to end from translation pairs alone.

Two structural consequences follow immediately. First, capacity now grows with the input: a 50-token sentence gets 50 states to draw on, not one. Second, and more importantly for training, the gradient from output position $i$ reaches input position $j$ through the single multiplication by $\alpha_{ij}$ — one hop, not $|i - j|$ hops. The $0.9^{26}$ decay from the earlier example becomes a factor of $\alpha_{ij}$ itself, which is 0.88 in the example above rather than 0.065.

Attention converts a distance-dependent gradient path into a constant-length one. That, more than the extra capacity, is why it works.

How the three problems interlock

They are not three independent complaints. They all trace back to the same design commitment — information travels between positions only by passing through every position in between.

Notice that fixing any one in isolation does not help much. Gating (fix 2) does not restore parallelism. A bigger hidden state (fix 3) makes the per-step matmul more expensive and the sequential bottleneck worse. The three constraints trade against each other precisely because they share a root cause.

What this means when you build something

The practical takeaway is a diagnostic habit: when a sequence model underperforms, ask which of the three constraints is binding before you reach for more layers or more data.

And a caution in the other direction, because it is a genuine trade and not a free win. Attention's cost grows as $T^2$. At $T = 512$, $d = 512$ the two architectures cost the same; at $T = 4096$ attention costs eight times more arithmetic and, worse, its memory grows quadratically too — the $T \times T$ matrix of weights must exist. Recurrence's memory is $O(T)$. That is why long-context work spends so much effort on sparse, linear and chunked attention variants: the problem that recurrence solved for free is the one attention had to pay for.

The single sentence to hold on to: recurrence makes distance expensive, and attention makes width expensive. Everything in the architectures that followed is a negotiation between those two costs.