Recurrent Neural Networks (RNNs)

Here are two reviews. Classify them.

A: "the food was good, the service was not"B: "the food was not good, the service was"

Any representation that averages or sums word vectors gives these two sentences identical feature vectors. Same words, same counts. A model built on averaged embeddings or bag-of-words cannot distinguish them, not because it is undertrained but because the information was destroyed before it ever reached the model.

So try the obvious fix: keep the order. Concatenate the word vectors into one long vector and feed that to a fully connected network.

#  5 words x 100 dims -> a 500-dim inputx = np.concatenate([emb[w] for w in tokens])model = MLP(input_dim=500, hidden=128, output=2)

Two things break immediately.

Length. That network accepts exactly 500 numbers, so it accepts exactly five words. A four-word review will not fit; a 300-word review will not fit. You can pad and truncate to a fixed maximum, but if you pick 500 words to accommodate the longest review, then a typical ten-word review is 98% padding and the network spends nearly all its parameters processing zeros.

Position blindness. Suppose you train on reviews where not good appears at positions 4–5. The weights connecting input dimensions 300–500 learn to recognise it. Now a test review says not good at positions 1–2. Those weights are in a completely different part of the network and have learned nothing. The model must relearn the same pattern independently at every position — and it needs training examples at every position to do so.

What you actually need is one mechanism that reads the sequence a word at a time, applies the same logic at every position, and carries forward what it has read so far. That is a recurrent neural network.

The recurrence

An RNN keeps a single vector called the hidden state, written $h_t$. Think of it as the network's running summary of everything it has read up to position $t$. At each step it reads one input and updates that summary:

Read the three ingredients:

• $x_t$ — the embedding of the word at position $t$, size $d_x$.
• $h_{t-1}$ — the summary after the previous word, size $d_h$. At $t = 0$ it is usually zeros.
• $\tanh$ — squashes the result into $(-1, 1)$, which keeps the state from growing without bound as the sequence gets longer.

If you want a prediction at each step, add an output layer:

The part that makes it work: shared weights

$W_{xh}$, $W_{hh}$ and $W_{hy}$ do not change with $t$. The same three matrices process word 1, word 2 and word 400.

This solves both problems from the opening at once. A 5-word sequence and a 500-word sequence use the same parameters, so length is no longer fixed. And a pattern learned at position 4 is learned by the same weights that process position 40, so it transfers for free.

The parameter count is fixed by the dimensions, not the sequence:

With $d_x = 100$ and $d_h = 128$: $128 \times 100 + 128 \times 128 + 128 = 12{,}800 + 16{,}384 + 128 = 29{,}312$ parameters, whether the sequence is five tokens or five thousand.

Unrolling

Drawn as a loop the RNN looks circular, which is confusing. Drawn unrolled across time it is just a very deep feed-forward network where every layer shares weights:

          y1        y2        y3        y4          ^         ^         ^         ^          |W_hy     |W_hy     |W_hy     |W_hyh0 --W_hh--> h1 --W_hh--> h2 --W_hh--> h3 --W_hh--> h4          ^         ^         ^         ^          |W_xh     |W_xh     |W_xh     |W_xh          x1        x2        x3        x4        "the"     "food"    "was"     "good"

That picture is the key to everything that follows. A 100-token sequence is a 100-layer network. Depth is where the trouble lives.

A forward pass with actual numbers

Take $d_x = 2$, $d_h = 2$, and small weights so the arithmetic stays readable.

W_xh = [[0.5, -0.3],     W_hh = [[0.1,  0.4],     b_h = [0, 0]        [0.2,  0.8]]             [-0.2, 0.3]]x1 = [1.0, 0.0]    x2 = [0.0, 1.0]    h0 = [0.0, 0.0]t = 1:  W_xh x1  = [0.5*1.0 + (-0.3)*0.0,  0.2*1.0 + 0.8*0.0] = [0.50, 0.20]  W_hh h0  = [0.00, 0.00]  pre      = [0.50, 0.20]  h1       = tanh([0.50, 0.20]) = [0.462, 0.197]t = 2:  W_xh x2  = [-0.30, 0.80]  W_hh h1  = [0.1*0.462 + 0.4*0.197,  -0.2*0.462 + 0.3*0.197]           = [0.0462 + 0.0788,  -0.0924 + 0.0591]           = [0.125, -0.033]  pre      = [-0.175, 0.767]  h2       = tanh([-0.175, 0.767]) = [-0.173, 0.645]

Notice that $h_2$ depends on $x_2$ and on $h_1$, which depended on $x_1$. Information from the first word is still present at the second step, mixed in through $W_{hh}$. That is the memory.

The shapes an RNN can take

For classification the standard move is to take the last hidden state as a summary of the whole sequence and put a linear layer on top of it. That works, with a caveat covered below about padding.

Backpropagation through time

Training an RNN is ordinary backpropagation applied to the unrolled graph. The name backpropagation through time (BPTT) just emphasises that the "layers" are time steps and the weights are shared.

Because $W_{hh}$ is used at every step, its gradient is the sum of its contributions from every step:

And the gradient of a late loss with respect to an early hidden state has to travel back through every intervening step:

That product is the whole story. Each factor is

— a matrix, repeated $T - k$ times. Repeated matrix multiplication does one of two things, and neither is good.

Vanishing gradients

Suppose the effective per-step factor has magnitude around 0.8, which is entirely plausible: $\tanh'$ is at most 1 and is much smaller once the state saturates, and typical weight matrices have spectral norm below 1.

Read the 0.8 column at 100 steps. The gradient reaching word 1 from a loss at word 100 is $2 \times 10^{-10}$ times the gradient at the last step. In float32 that is not literally zero, but it is thoroughly drowned by the gradients from recent steps. The weights update almost entirely on the basis of the last handful of tokens.

The practical symptom is specific and recognisable: the model trains, the loss goes down, and it simply cannot learn dependencies more than about 10–20 steps apart. Give it "I bought this for my daughter, who had been asking for one for months, and after two weeks it broke" and it will latch onto the last clause and ignore everything before it. It never errors. It just has a memory horizon.

Exploding gradients

The 1.2 column is the other failure. Gradients grow to $10^7$, the weight update is enormous, the parameters land somewhere absurd, and the next forward pass produces NaN. Unlike vanishing, this one is loud — and it has a simple fix.

loss.backward()torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=5.0)optimizer.step()

Clipping rescales the whole gradient vector if its norm exceeds a threshold, preserving direction while capping magnitude. It costs almost nothing and there is no good reason to train a recurrent model without it.

Exploding gradients are a nuisance with a one-line fix. Vanishing gradients are the real problem, because they do not announce themselves — they quietly cap how far back your model can see, and no amount of extra training removes the cap.

Building one from scratch

Writing the loop by hand makes the mechanism concrete.

1import numpy as np23class SimpleRNN:4    def __init__(self, input_size, hidden_size, output_size, seed=0):5        rng = np.random.default_rng(seed)6        # Scale by 1/sqrt(fan_in) to keep activations in a sane range7        self.Wxh = rng.normal(0, 1 / np.sqrt(input_size),  (hidden_size, input_size))8        self.Whh = rng.normal(0, 1 / np.sqrt(hidden_size), (hidden_size, hidden_size))9        self.Why = rng.normal(0, 1 / np.sqrt(hidden_size), (output_size, hidden_size))10        self.bh = np.zeros((hidden_size, 1))11        self.by = np.zeros((output_size, 1))1213    def forward(self, inputs):14        """inputs: list of (input_size, 1) column vectors."""15        h = np.zeros((self.Whh.shape[0], 1))16        self.cache = {"h": {-1: h}, "x": {}}17        for t, x in enumerate(inputs):18            h = np.tanh(self.Wxh @ x + self.Whh @ h + self.bh)19            self.cache["h"][t] = h20            self.cache["x"][t] = x21        y = self.Why @ h + self.by22        return y, h2324    def backward(self, dy, clip=5.0):25        """dy: gradient of loss wrt final output y."""26        T = len(self.cache["x"])27        h_last = self.cache["h"][T - 1]2829        dWhy = dy @ h_last.T30        dby = dy31        dWxh = np.zeros_like(self.Wxh)32        dWhh = np.zeros_like(self.Whh)33        dbh = np.zeros_like(self.bh)3435        dh = self.Why.T @ dy          # gradient flowing into the last state36        for t in reversed(range(T)):37            h_t, h_prev = self.cache["h"][t], self.cache["h"][t - 1]38            draw = (1 - h_t ** 2) * dh          # through tanh39            dWxh += draw @ self.cache["x"][t].T40            dWhh += draw @ h_prev.T41            dbh += draw42            dh = self.Whh.T @ draw              # <-- the repeated multiplication4344        grads = [dWxh, dWhh, dWhy, dbh, dby]45        for g in grads:46            np.clip(g, -clip, clip, out=g)47        return grads

The line marked with the arrow is where vanishing and exploding gradients come from. Every iteration of that loop multiplies dh by Whh.T again. Nothing else in the class matters as much as that one line.

Using PyTorch's implementation

1import torch2import torch.nn as nn34class RNNClassifier(nn.Module):5    def __init__(self, vocab_size, embed_dim=100, hidden_dim=128, num_classes=2):6        super().__init__()7        self.embedding = nn.Embedding(vocab_size, embed_dim, padding_idx=0)8        self.rnn = nn.RNN(embed_dim, hidden_dim, batch_first=True)9        self.dropout = nn.Dropout(0.3)10        self.fc = nn.Linear(hidden_dim, num_classes)1112    def forward(self, x, lengths):13        # x: (batch, seq_len) of token ids14        emb = self.embedding(x)                       # (batch, seq, embed)15        packed = nn.utils.rnn.pack_padded_sequence(16            emb, lengths.cpu(), batch_first=True, enforce_sorted=False17        )18        _, h_n = self.rnn(packed)                     # h_n: (1, batch, hidden)19        return self.fc(self.dropout(h_n[-1]))

Two details in that code are doing real work.

padding_idx=0 tells the embedding layer that token 0 is padding, so its vector stays at zero and never receives a gradient. Without it, the model learns an embedding for "nothing", which is meaningless and occasionally harmful.

pack_padded_sequence is more important than it looks, and skipping it is the most common bug in RNN code. Consider a batch where one review is 8 tokens and another is 200. To batch them, the short one is padded to 200. Without packing, the RNN runs all 200 steps on the short sequence, updating its hidden state 192 times on padding tokens. By the time you read h_n, the actual content has been washed out by 192 steps of processing zeros.

Without packing, an 8-token review in a 200-length batch:  h_8   = a good summary of the review  h_200 = h_8 pushed through 192 more update steps  <- what you read          the review's signal has decayed awayWith packing:  the RNN stops at step 8 for that sequence  h_n is exactly h_8                               <- what you want

Packing tells PyTorch each sequence's true length so it stops at the right step and returns the correct final state for every element of the batch.

Sequence labelling instead of classification

When you need one output per token rather than one per sequence, take all the hidden states instead of just the last:

1class RNNTagger(nn.Module):2    def __init__(self, vocab_size, embed_dim, hidden_dim, num_tags):3        super().__init__()4        self.embedding = nn.Embedding(vocab_size, embed_dim, padding_idx=0)5        self.rnn = nn.RNN(embed_dim, hidden_dim, batch_first=True)6        self.fc = nn.Linear(hidden_dim, num_tags)78    def forward(self, x):9        outputs, _ = self.rnn(self.embedding(x))   # (batch, seq, hidden)10        return self.fc(outputs)                    # (batch, seq, num_tags)1112# Ignore padding positions when computing the loss13criterion = nn.CrossEntropyLoss(ignore_index=-100)14loss = criterion(logits.view(-1, num_tags), tags.view(-1))

Set padded positions in the target tensor to -100 and ignore_index excludes them from the loss. Forget this and the model spends most of its capacity learning to predict the pad tag, which it will do with 99% accuracy while learning nothing useful.

Truncated BPTT and long sequences

Backpropagating through a 10,000-token document means holding 10,000 hidden states in memory and running a 10,000-step backward pass. Memory grows linearly with sequence length and it becomes impractical fast.

Truncated BPTT splits the sequence into chunks of, say, 100 steps. The forward pass carries the hidden state from chunk to chunk, but the backward pass stops at the chunk boundary — you call .detach() on the state before starting the next chunk.

1h = None2for chunk in chunks_of(sequence, size=100):3    out, h = model(chunk, h)4    loss = criterion(out, targets_for(chunk))5    loss.backward()6    torch.nn.utils.clip_grad_norm_(model.parameters(), 5.0)7    optimizer.step()8    optimizer.zero_grad()9    h = h.detach()      # forward state continues, gradient does not

The state still carries information forward across the whole document. Only the gradient is cut. In practice this costs little, because the vanishing gradient had already made anything beyond ~50 steps unlearnable.

What to take to the keyboard

The vanilla RNN is not the architecture you will ship. Its memory horizon is roughly 10–20 tokens, which is shorter than most sentences you care about, and gated variants replaced it for exactly that reason. It is worth understanding anyway, because every recurrent architecture is this one plus machinery to protect the gradient, and because the failure modes are shared.

When you build your first recurrent model, do three things before tuning anything else. Clip the gradients. Pack your sequences. And check your metric on a mask that excludes padding. Those three account for the large majority of recurrent models that look broken for reasons that have nothing to do with the architecture.