Overfitting and Regularization

Train a network with two million parameters on 500 examples. Watch the training loss slide down to 0.001 and training accuracy hit 100%. Then evaluate on data it has not seen: 61%.

Now run the experiment that makes this impossible to explain away. Take the same 500 examples and randomly shuffle the labels — so a photograph of a dog is labelled "aeroplane", a cat is labelled "ship", and there is no pattern whatsoever connecting inputs to outputs. Train the same network.

It still reaches 100% training accuracy.

There was nothing to learn. The labels were noise. The network memorised 500 arbitrary input-output pairs, which it has more than enough capacity to do — two million parameters against 500 examples is 4,000 parameters per example. A model with that much room does not need to find a pattern. It can simply store the answers.

Low training loss proves the model has enough capacity. It proves nothing at all about whether the model learned anything general.

The gap between what the model does on data it has seen and data it has not is the only thing that matters, and regularisation is the collective name for every technique that closes it.

Bias and variance, with a concrete picture

Suppose the truth is a gentle curve and you have twenty noisy samples of it. Three models:

The degree-19 fit is worth picturing. With twenty parameters and twenty points, it can and does hit every point exactly. Between the points it swings wildly, because nothing constrains it there. Move one training point slightly and the entire curve reorganises. That instability is what "variance" means: the fitted function depends heavily on the particular sample you happened to draw.

Two terms, two very different remedies:

Diagnose before you treat. Adding dropout to an underfitting model is an actively harmful move, and it is a common one, because "add regularisation" has become a reflex.

Underfitting, and how to recognise it

Signs: training loss plateaus high and validation loss sits right next to it. The two curves are close together, which people sometimes read as a good sign. It is not — it means the model is equally bad everywhere.

The biggest lever is more data

Every regularisation technique below is a way of compensating for not having enough data. If you can get more, that beats all of them, and it is not close.

When you cannot collect more, you can often manufacture more. Data augmentation applies label-preserving transformations: a photograph of a cat rotated eight degrees is still a photograph of a cat, but it is a different array of pixels, so the model cannot memorise it as a fixed input.

1import torchvision.transforms as T23train_tf = T.Compose([4    T.RandomResizedCrop(224, scale=(0.7, 1.0)),5    T.RandomHorizontalFlip(),6    T.ColorJitter(brightness=0.2, contrast=0.2, saturation=0.2),7    T.RandomRotation(10),8    T.ToTensor(),9    T.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),10])1112# Validation and test get NO random augmentation -- only the deterministic parts13eval_tf = T.Compose([14    T.Resize(256), T.CenterCrop(224), T.ToTensor(),15    T.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),16])

Augmenting the validation set is a real and common mistake. It makes your validation metric noisy and pessimistic, so you cannot compare runs reliably.

The transformations must be label-preserving, and that is domain-specific. A horizontal flip is fine for a cat and wrong for a handwritten digit — a flipped 2 is not a 2, and a flipped 6 is not a 9 in any useful sense. For text, synonym replacement and back-translation work; for audio, time-shifting and adding background noise. For tabular data there is no general recipe, which is one reason tabular deep learning is harder than it looks.

L2 regularisation, or weight decay

Add a penalty proportional to the squared size of the weights:

The gradient of that penalty is simply $\lambda w$, so the update becomes:

Every step multiplies each weight by $(1 - \eta\lambda)$ before applying the data gradient. That is where the name "weight decay" comes from, and the arithmetic is instructive. With $\eta = 0.01$ and $\lambda = 0.01$, the factor is $0.9999$. Over 10,000 steps, a weight receiving no useful gradient shrinks to $0.9999^{10000} \approx 0.37$ of its starting value. A weight that is useful gets pushed back up by the data gradient every step and survives.

So L2 is a continuous negotiation: justify your magnitude or fade away. The effect is to prefer many small weights over a few large ones, which produces smoother functions that change less abruptly as the input changes.

1# PyTorch -- use AdamW, which decays weights correctly alongside Adam's scaling2opt = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-2)34# Do NOT decay biases or normalisation parameters -- they are not the problem5decay, no_decay = [], []6for name, p in model.named_parameters():7    (no_decay if p.ndim <= 1 else decay).append(p)8opt = torch.optim.AdamW([9    {"params": decay,    "weight_decay": 1e-2},10    {"params": no_decay, "weight_decay": 0.0},11], lr=1e-3)

1from tensorflow.keras import layers, regularizers23layers.Dense(128, activation="relu",4             kernel_regularizer=regularizers.l2(1e-4))   # weights only, not bias

Typical values for $\lambda$ span $10^{-5}$ to $10^{-2}$. Search it in log space; the difference between $10^{-4}$ and $10^{-3}$ is meaningful, the difference between $10^{-4}$ and $2 \times 10^{-4}$ is not.

L1 regularisation and sparsity

Penalise absolute size instead of squared size:

The gradient of $|w|$ is $\text{sign}(w)$ — a constant pull of the same size regardless of how small $w$ already is. L2's pull, $\lambda w$, shrinks as the weight shrinks, so it asymptotes towards zero without arriving. L1's constant pull drives weights exactly to zero and holds them there.

In deep learning L2 is the default by a wide margin. L1 earns its place when you specifically want sparsity — to identify which of 5,000 input features actually matter, or to prune a model for deployment.

Dropout

During each training step, randomly set a fraction $p$ of a layer's activations to zero. Different units are dropped every step.

Why this helps: if a unit can rely on a specific partner unit always being present, the two co-adapt into a fragile pair that only works together. Dropout makes any given partner unavailable half the time, so every unit must contribute something individually useful. The result is redundant, distributed representations rather than brittle specialised ones. A second reading: with $n$ units there are $2^n$ possible dropout masks, so training is approximately training an ensemble of exponentially many networks that share weights — and ensembles generalise better than their members.

The scaling detail that must be right

If you drop half the units, the sum arriving at the next layer is halved, so the next layer sees a different input scale at training time than at inference time. Modern implementations use inverted dropout: scale the surviving activations up by $1/(1-p)$ during training, so the expected value is unchanged and inference needs no adjustment at all.

With $p = 0.5$ and a layer producing $[2, 4, 6, 8]$, suppose the mask keeps units 2 and 4:

Both frameworks do this for you, which is precisely why model.eval() matters: it switches dropout off entirely, and the inverted scaling means no other correction is needed.

1model = nn.Sequential(2    nn.Linear(784, 512), nn.ReLU(), nn.Dropout(0.5),3    nn.Linear(512, 256), nn.ReLU(), nn.Dropout(0.3),4    nn.Linear(256, 10),                     # NEVER dropout before the output5)

Early stopping: free, and everyone should use it

Validation loss typically falls, reaches a minimum, and then rises while training loss keeps falling. Everything after that minimum is the model memorising. Early stopping watches validation loss and stops when it stops improving — then restores the weights from the best epoch, not the last one.

1best_val, patience, wait = float("inf"), 10, 023for epoch in range(500):4    train_one_epoch(model, opt, train_loader)5    val = evaluate(model, val_loader)67    if val < best_val - 1e-4:              # a real improvement, not noise8        best_val, wait = val, 09        torch.save(model.state_dict(), "best.pt")10    else:11        wait += 112        if wait >= patience:13            print(f"stopping at epoch {epoch}; best val {best_val:.4f}")14            break1516model.load_state_dict(torch.load("best.pt"))   # restore the BEST, not the last

1from tensorflow.keras.callbacks import EarlyStopping23EarlyStopping(monitor="val_loss", patience=10,4              min_delta=1e-4, restore_best_weights=True)

Two settings carry the weight. patience too small (say 2) stops on a normal fluctuation and leaves performance on the table; 10–20 is a sensible range. And restore_best_weights must be on — without it you keep the weights from the last epoch, which by construction are worse than the best ones you already had.

A default recipe

These techniques stack, and a reasonable starting configuration for a moderate-sized problem is:

1model = nn.Sequential(2    nn.Linear(n_features, 256),3    nn.BatchNorm1d(256), nn.ReLU(), nn.Dropout(0.3),4    nn.Linear(256, 128),5    nn.BatchNorm1d(128), nn.ReLU(), nn.Dropout(0.3),6    nn.Linear(128, n_classes),7)8opt = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-2)9sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=epochs)10# plus data augmentation, plus early stopping on validation loss

One interaction to be aware of: batch normalisation and dropout together can hurt. Batch norm normalises using batch statistics, dropout changes the variance of what it sees, and at inference the two disagree about what the activation scale should be. The common resolution is to put dropout after the normalisation and activation, as above, and to use a lower rate than you would without batch norm — or to skip dropout entirely in convolutional networks, where batch norm alone regularises well.

Turning a symptom into an action

When you sit down in front of a disappointing model, run this procedure rather than reaching for a favourite technique.

First, compare training and validation loss. That single comparison splits the world in two, and the two halves have opposite treatments. If both are high, you are underfitting and every regularisation technique on this page will make things worse. If training is low and validation is much higher, you are overfitting and they will help.

If overfitting, work in order of expected value. More data first, if it exists. Then augmentation, which is nearly as good and costs only code. Then early stopping, which is free and never hurts. Then weight decay, tuned in log space. Then dropout. Only then consider shrinking the model — capacity plus regularisation almost always beats less capacity, because a large regularised model can represent the right function and simply chooses not to overreach, whereas a small model cannot represent it at all.

Change one thing at a time and record the result. Adding dropout, weight decay and augmentation together and seeing an improvement tells you nothing about which one worked, and one of them may be hurting while the others carry it.

Watch for the gap that never closes. If you have applied everything and validation loss is still far above training loss, the problem may not be regularisation at all — it may be that your validation set is drawn from a different distribution than your training set. Photographs taken with a different camera, transactions from a different quarter, users from a different country. No amount of dropout fixes a distribution shift, and diagnosing it as overfitting will send you in circles for a week.