Transfer Learning with Pretrained Models

A hospital wants a model that spots pneumonia in chest X-rays. They have 900 labelled images — every one reviewed by a radiologist, which is why there are only 900.

You train a ResNet-50 from scratch. Twenty-five million parameters, 900 examples. That is roughly 28,000 parameters per training image. Predictably, training accuracy hits 100% within a few epochs and validation accuracy sits around 65%. The model has memorised 900 pictures.

Collecting 100,000 more X-rays is not an option. Neither is a smaller model, really — the tiny networks that would suit 900 images are not good enough at the task.

So consider what the model is actually missing. To read a chest X-ray it needs to detect edges, textures, opacities, gradients, shapes, and how those combine into anatomical structures. Almost none of that is specific to medicine. Edges are edges. Texture is texture. A network trained on a million photographs of dogs, cars and mushrooms has already learned all of it — and that training was paid for by somebody else.

Why borrowed features work

Inspect a trained convolutional network layer by layer and a consistent picture emerges, regardless of what it was trained on:

The first-layer filters of a network trained on ImageNet, on satellite imagery, and on X-rays look remarkably alike — a set of oriented edge detectors and colour-opponent blobs, essentially the same set the mammalian visual cortex uses. This is not a coincidence: those are the efficient primitives for describing natural images, and any system that learns from images converges on something close to them.

Early layers learn what images are like. Late layers learn what your labels are. Transfer learning keeps the first and replaces the second.

That is the whole method. Take a network trained on a large dataset, discard the final classification layer, and reuse everything before it as a feature extractor for your task. With 900 X-rays you are no longer learning what an edge is from 900 examples. You are learning only how known visual features map to "pneumonia" or "healthy" — a far smaller problem.

The practical payoff is large and consistent: typically 10–30 percentage points of accuracy over training from scratch on a small dataset, and convergence in a fraction of the epochs.

The two strategies, and how to choose

Feature extraction: freeze the backbone

Freeze every pretrained layer so its weights never update, replace the final layer with one sized for your classes, and train only that.

The backbone becomes a fixed function that turns an image into a vector — 2048 numbers for ResNet-50 — and you are training a simple classifier on those vectors. With only a couple of thousand trainable parameters, overfitting is nearly impossible.

1import torch.nn as nn2from torchvision import models34model = models.resnet50(weights=models.ResNet50_Weights.IMAGENET1K_V2)56for p in model.parameters():7    p.requires_grad = False              # freeze everything89model.fc = nn.Linear(2048, 2)            # new layers default to requires_grad=True1011trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)12print(trainable)   # 4098 -- of roughly 25 million

It is fast, memory-light, and safe. Its ceiling is that the features are fixed: whatever ImageNet did not teach the network to see, it will never see.

Fine-tuning: let the backbone adapt

Unfreeze some or all of the pretrained layers and train them alongside the new head, at a much lower learning rate.

Higher ceiling, higher risk. The risk has a name — catastrophic forgetting — and a very specific cause. The new final layer starts with random weights, so its first few batches produce large, essentially meaningless gradients. Those gradients propagate back into the carefully learned pretrained weights and scramble them. You destroy the thing you came for, in the first thirty seconds of training.

The randomly initialised head is the danger, not the backbone. Warm it up before you unfreeze anything, or its first meaningless gradients will erase the very features you came for.

The defence is a two-phase schedule:

1. Warm up the head. Freeze the backbone, train only the new layer for 2–5 epochs at a normal learning rate such as $10^{-3}$. The head becomes sensible, so its gradients become sensible.
2. Unfreeze and fine-tune. Now unfreeze the backbone and continue at a much smaller learning rate, typically $10^{-4}$ or $10^{-5}$ — ten to a hundred times lower than you would use from scratch. You are nudging good weights, not searching from random ones.

Which to use

Two questions decide it: how much data you have, and how far your images are from the pretraining distribution.

The logic behind the top-right cell is the interesting one. X-rays are visually unlike photographs — greyscale, different textures, different statistics — so the mid-level and high-level ImageNet features are a poor match and need to adapt. But with only a few hundred images you cannot afford to train 25 million parameters. So you compromise: keep the universally useful early layers frozen, and let the later, more task-specific ones move.

Which layers to unfreeze

Always unfreeze from the end backwards. Unfreezing an early block while later ones stay frozen is close to nonsense — you would be changing the inputs that the frozen layers were calibrated for, while forbidding them to adjust.

1def unfreeze_from(model, layer_names):2    for name, param in model.named_parameters():3        param.requires_grad = any(name.startswith(n) for n in layer_names)45# ResNet blocks run layer1 -> layer2 -> layer3 -> layer4 -> fc6unfreeze_from(model, ["layer4", "fc"])       # last block plus head7# unfreeze_from(model, ["layer3", "layer4", "fc"])   # go deeper if data allows

Discriminative learning rates

A refinement that follows directly from the layer hierarchy: if early layers need almost no change and late layers need a lot, why give them the same learning rate?

Assign a lower learning rate to earlier layers and higher rates further in, usually by a factor of 2–10 per group.

1import torch23optimiser = torch.optim.AdamW([4    {"params": model.layer1.parameters(), "lr": 1e-6},5    {"params": model.layer2.parameters(), "lr": 5e-6},6    {"params": model.layer3.parameters(), "lr": 1e-5},7    {"params": model.layer4.parameters(), "lr": 5e-5},8    {"params": model.fc.parameters(),     "lr": 1e-3},9], weight_decay=1e-4)

The generic edge detectors barely move; the head, which knows nothing, learns fast. In practice this buys a point or two of accuracy over a single learning rate and makes training noticeably more stable. It is worth setting up once and keeping as a template.

Details that decide whether it works

Match the preprocessing exactly

This is the most common silent failure in transfer learning. A pretrained network's filters were tuned for inputs with a particular distribution. Feed a different one and you get a quiet accuracy loss with no error message anywhere.

For ImageNet weights that means: RGB channel order, values scaled to $[0, 1]$, then standardised with mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225]. If you load images with OpenCV, remember that it returns BGR — a model fed reversed channels typically loses several points, and nothing in the logs will tell you why.

Handling greyscale and other channel counts

Pretrained models expect three channels. An X-ray has one. Two reasonable options:

• Replicate the channel three times. Crude but effective, and it keeps the pretrained first layer completely intact. This is the usual choice.
• Sum the first-layer weights across the input dimension into a single-channel kernel. Cleaner and cheaper at inference. Mathematically this gives the same response as replication for a greyscale input, so it is strictly better if you are willing to modify the layer.

Input resolution

Most ImageNet models were trained at 224×224. A modern convolutional network with global average pooling will accept other sizes without error, but the features degrade if you stray far, because object scale relative to the receptive field changes. Going larger — 384 or 512 — often helps for fine detail such as small lesions, at quadratic cost in compute. Going much smaller than 224 usually hurts noticeably.

Batch normalisation in frozen layers

An easily missed trap. Batch normalisation layers hold running mean and variance statistics that update during training even when their weights are frozen, because that update is not a gradient operation. If your data differs from ImageNet, these statistics drift towards your data while the frozen weights still expect the original ones, and accuracy degrades in a way that looks inexplicable.

Setting the module to evaluation mode freezes the statistics as well:

1def freeze_bn(module):2    for m in module.modules():3        if isinstance(m, nn.BatchNorm2d):4            m.eval()                       # stop running-stat updates5            for p in m.parameters():6                p.requires_grad = False    # stop gamma/beta updates78model.train()9freeze_bn(model.layer1)   # call AFTER model.train(), which resets modes

Note the ordering comment. Calling model.train() puts every submodule back into training mode, so the freeze must come afterwards, every epoch.

Learning rate, batch size, regularisation

A cosine learning rate schedule with a short linear warm-up is a reliable default. Warm-up matters especially here: the first few hundred steps with a randomly initialised head are exactly when the backbone is most at risk.

When transfer learning makes things worse

Negative transfer is real. It happens when the source features actively mislead the target task, and it shows up as a fine-tuned model performing below a from-scratch model.

The usual causes:

• The domain gap is too wide. ImageNet is photographs of objects under natural light. Ultrasound, radar returns, spectrograms and electron micrographs share very little with that beyond the first layer or two.
• The task is structurally different. ImageNet training teaches the network to be invariant to texture-level colour differences and to focus on object identity. If your task is fine-grained colour or texture discrimination, that invariance is a learned handicap.
• Fine-tuning too aggressively. A learning rate meant for from-scratch training will erase the pretrained knowledge in the first epoch, leaving you with a randomly restarted model plus a bad initialisation.

The diagnostic is simple and cheap: always train a from-scratch baseline. It costs one training run and tells you whether transfer is helping at all. If it is not, the mitigation ladder runs: warm up the head properly, lower the learning rate, freeze more layers, or find a pretraining source closer to your domain — a model pretrained on medical images rather than on photographs, for instance.

Where to get weights

That last row is worth knowing about. Self-supervised pretraining does not optimise for the 1000 ImageNet classes, so its features are less specialised to that particular label set and frequently generalise better when your domain is unusual.

A working recipe

For a new task with limited data, this sequence will get you a strong result quickly and tells you where the ceiling is at each step.

Start by establishing what "good" means. Train a from-scratch baseline and freeze-only transfer with a linear head. Those two numbers bracket the problem: the gap between them tells you how much the pretrained features are worth, and if there is no gap, stop and reconsider your pretraining source before investing in anything more elaborate.

Then improve the transfer run in order of risk. Warm up the head for a few epochs with everything frozen. Unfreeze the last block and continue at $10^{-4}$. If validation is still improving and you have the data to support it, unfreeze another block and drop to $5 \times 10^{-5}$. Add discriminative learning rates once more than one block is training. Stop as soon as validation stops improving — with a few hundred images, the difference between the best epoch and three epochs later can be several points.

The judgement to internalise is that freezing is a regularisation dial. Every layer you freeze removes parameters that could overfit and removes capacity that could adapt. Small dataset, similar domain: freeze almost everything. Large dataset, distant domain: freeze almost nothing. Everything in between is a matter of watching the gap between your training and validation curves and moving the dial in the direction that closes it.