Domain Adaptation — Closing the Train/Deploy Gap

A manufacturer deploys a surface-defect detector on production Line A. It was fine-tuned on 12,000 labelled images from that line's cameras and it reaches an F1 score of 94.2% on a held-out test set from the same line. Everyone is pleased. Six weeks later the same model is installed on Line B — identical product, identical defect types, identical camera model.

F1 on Line B: 68.5%.

Nothing is broken. The model has not degraded, the code is the same, the weights are byte-for-byte identical. The only difference is that Line B's overhead LEDs are a slightly cooler colour temperature and mounted 40 cm higher, so specular highlights fall in different places and the average image is about 18% brighter. To a human inspector the two lines look the same. To a network whose features were tuned on Line A's exact illumination, they are different worlds.

There are no labels for Line B. Getting 12,000 more images annotated by a metallurgist would take four months and cost more than the project. The question is what you can do with unlabelled target data, and the answer is a family of techniques called domain adaptation.

What domain shift actually is

Standard supervised learning rests on one assumption that is almost never stated out loud: training and deployment data are drawn from the same distribution. When that fails, you have domain shift, and every guarantee your validation set gave you evaporates.

Write the source distribution as $P_s(X, Y)$ and the target as $P_t(X, Y)$. Domain shift means $P_s(X, Y) \neq P_t(X, Y)$. But the joint distribution factors two ways, $P(X,Y) = P(Y|X)P(X) = P(X|Y)P(Y)$, and which factor moved determines which fix works. This is the distinction people skip, and skipping it is why the wrong technique gets applied.

Unsupervised domain adaptation only works under covariate shift. If the labelling rule itself has changed, no amount of clever feature alignment will recover it — you are trying to learn a function from data that contains no evidence of it.

Before spending a week on adversarial training, work out which row you are in. The Line B problem is covariate shift: a scratch is still a scratch, it just photographs differently. That is the fixable case.

Measuring how big the shift is

You cannot compute $P(X)$ directly in 150,000 dimensions, but you can measure the shift with a trick: train a classifier to tell source from target. Label every source image 0 and every target image 1, train a small model on the backbone's features, and look at how well it does on held-out data.

1import numpy as np2from sklearn.linear_model import LogisticRegression3from sklearn.model_selection import cross_val_score45# feats_s, feats_t: (N, 2048) arrays of backbone features6X = np.vstack([feats_s, feats_t])7d = np.hstack([np.zeros(len(feats_s)), np.ones(len(feats_t))])89err = 1 - cross_val_score(LogisticRegression(max_iter=1000),10                          X, d, cv=5, scoring="accuracy").mean()11proxy_a_distance = 2 * (1 - 2 * err)12print(f"domain classifier error {err:.3f}  ->  dA = {proxy_a_distance:.2f}")

The proxy A-distance is $d_A = 2(1 - 2\epsilon)$, where $\epsilon$ is that classifier's error rate. Work through the two extremes. If the domains are indistinguishable the classifier is at chance, $\epsilon = 0.5$, and $d_A = 2(1 - 1) = 0$. If they are trivially separable, $\epsilon = 0.02$, and $d_A = 2(1 - 0.04) = 1.92$ — close to the maximum of 2.

On the Line A / Line B features, the domain classifier scored 97% accuracy, so $\epsilon = 0.03$ and $d_A = 2(1 - 0.06) = 1.88$. A linear model can separate the two lines almost perfectly from backbone features alone. That number is both the diagnosis and the target: successful adaptation should drive it down.

Four scenarios, distinguished by what labels you have

The row worth pausing on is semi-supervised. Going from zero target labels to just five per class often buys more than the most sophisticated unsupervised method. For the Line B problem, 31 defect classes × 5 labelled examples = 155 annotations, roughly a day of one metallurgist's time. Before committing to adversarial training, always price out that day.

Batch normalisation adaptation: the cheapest fix that works

Start here, always, because it costs one forward pass and no training at all.

Why frozen BatchNorm breaks under domain shift

A BatchNorm layer stores running estimates of the mean and variance of its inputs, accumulated during training, and at inference it normalises using those stored numbers:

Those running statistics are a compressed description of the source distribution. Feed target data through and the normalisation is simply wrong.

Take a real measurement from one early-layer channel on the defect detector. On Line A images that channel's activations had $\mu_s = 0.42$, $\sigma_s = 0.19$. On Line B images the same channel produced $\mu_t = 0.61$, $\sigma_t = 0.31$. Normalising a typical Line B activation of 0.61 with the stored Line A statistics gives:

The layer's own statistics say this activation should normalise to 0 — it is exactly the target-domain mean. Instead it comes out at 1.0, a full standard deviation off centre. Every downstream layer now receives systematically shifted inputs, and in a 50-layer network those shifts compound.

The fix: recompute the statistics on target data

Adaptive BatchNorm (AdaBN) simply replaces $\mu_{running}, \sigma_{running}$ with statistics computed on unlabelled target data. No labels, no gradients, no optimiser.

1import torch, torch.nn as nn23def adapt_batchnorm(model, target_loader, num_batches=50, device="cuda"):4    """Recompute BN running statistics on unlabelled target data."""5    # Reset the stored statistics and put BN into training mode so6    # forward passes update them. Everything else stays in eval mode.7    model.eval()                   # dropout and everything else: inference mode8    for m in model.modules():9        if isinstance(m, (nn.BatchNorm1d, nn.BatchNorm2d)):10            m.reset_running_stats()11            m.momentum = None      # None -> cumulative moving average12            m.train()1314    with torch.no_grad():15        for i, (images, _) in enumerate(target_loader):   # labels unused16            if i >= num_batches:17                break18            model(images.to(device))1920    model.eval()21    return model

Setting momentum = None makes PyTorch accumulate an exact cumulative average rather than an exponential one, which is what you want for a one-off recalibration. Fifty batches of 64 images is 3,200 unlabelled target images — for stable estimates you want at least a few hundred, and returns flatten past a few thousand.

Recomputing BatchNorm statistics on unlabelled target data recovered 41% of a 25-point gap in twenty seconds. Try it before anything else, every time.

Domain adversarial training: make the features unable to tell the domains apart

AdaBN fixes the normalisation but leaves the features themselves source-specific. Domain adversarial training goes further: it changes what the network computes, so that the representation carries class information but no domain information.

The core idea

Three components share one backbone:

• A feature extractor $G_f$ that maps images to a representation.
• A label predictor $G_y$ that classifies the representation. Trained on source data only, because only source data has labels.
• A domain discriminator $G_d$ that tries to say whether a representation came from source or target. Trained on both, using the free domain label.

The discriminator minimises its own domain-classification loss. The feature extractor maximises it — it is rewarded for producing features the discriminator cannot classify. At equilibrium the features are domain-invariant, and since the label predictor is simultaneously being trained to work on those features, they are also class-discriminative.

The objective is one saddle point:

The gradient reversal layer

The elegant implementation trick is a layer that is the identity going forwards and multiplies the gradient by $-\lambda$ going backwards. That single sign flip turns a minimisation into the required maximisation, and lets you train the whole thing with one ordinary optimiser and one backward pass.

1import torch, torch.nn as nn2from torch.autograd import Function34class GradientReversal(Function):5    @staticmethod6    def forward(ctx, x, lambd):7        ctx.lambd = lambd8        return x.view_as(x)              # identity forwards910    @staticmethod11    def backward(ctx, grad_output):12        return -ctx.lambd * grad_output, None   # negate backwards1314def grad_reverse(x, lambd=1.0):15    return GradientReversal.apply(x, lambd)1617class DANN(nn.Module):18    def __init__(self, backbone, feat_dim, num_classes):19        super().__init__()20        self.features = backbone                 # e.g. ResNet-50 body21        self.classifier = nn.Linear(feat_dim, num_classes)22        self.discriminator = nn.Sequential(23            nn.Linear(feat_dim, 1024), nn.ReLU(),24            nn.Dropout(0.5),25            nn.Linear(1024, 2),                  # source vs target26        )2728    def forward(self, x, lambd=1.0):29        f = self.features(x).flatten(1)30        return self.classifier(f), self.discriminator(grad_reverse(f, lambd))

The training step feeds source and target batches together:

1for step, ((xs, ys), (xt, _)) in enumerate(zip(src_loader, tgt_loader)):2    p = step / total_steps3    lambd = 2.0 / (1.0 + np.exp(-10.0 * p)) - 1.0    # ramp 0 -> 145    cls_s, dom_s = model(xs.cuda(), lambd)6    _,     dom_t = model(xt.cuda(), lambd)78    loss_cls = F.cross_entropy(cls_s, ys.cuda())9    loss_dom = (F.cross_entropy(dom_s, torch.zeros(len(xs), dtype=torch.long).cuda())10              + F.cross_entropy(dom_t, torch.ones(len(xt),  dtype=torch.long).cuda()))1112    (loss_cls + loss_dom).backward()13    optimiser.step(); optimiser.zero_grad()

The $\lambda$ schedule matters more than people expect. At $p = 0$ it gives $\lambda = 2/(1+e^{0}) - 1 = 0$; at $p = 0.5$, $\lambda = 2/(1 + e^{-5}) - 1 = 0.987$; at $p = 1$, $\lambda = 0.9999$. So the adversarial pressure starts at exactly zero and ramps smoothly to full strength. Turn it on at full strength from step 1 and the features are pushed to be domain-invariant before they encode anything useful about the classes — the network happily collapses to a constant representation, which is perfectly domain-invariant and perfectly useless.

That third limitation deserves emphasis because it is the named failure mode of adversarial adaptation. The discriminator only sees the overall distribution of features; it has no idea which class anything is. A solution that maps target images of class 3 onto the source cluster for class 7 satisfies the discriminator completely. Class-conditional variants exist that condition the discriminator on the predicted label, precisely to close this hole.

Self-training with pseudo-labels

A different angle, and often a stronger one: use the model's own confident predictions on target data as if they were ground truth.

1@torch.no_grad()2def make_pseudo_labels(model, target_loader, threshold=0.95):3    model.eval()4    keep_x, keep_y, total = [], [], 05    for images, _ in target_loader:6        probs = model(images.cuda()).softmax(dim=1)7        conf, pred = probs.max(dim=1)8        mask = conf >= threshold9        total += len(images)10        keep_x.append(images[mask.cpu()]); keep_y.append(pred[mask].cpu())11    x, y = torch.cat(keep_x), torch.cat(keep_y)12    print(f"kept {len(y)}/{total} = {100*len(y)/total:.1f}% above {threshold}")13    return x, y

Run it on the 5,000 unlabelled Line B images with a threshold of 0.95 and 1,240 pass — 24.8%. Spot-checking those 1,240 against a small audited sample shows 91% are correct. So you have just added 1,128 correct labels and 112 wrong ones to your training set.

Whether that is a good trade depends entirely on the errors being random rather than systematic, and they never are. The model is confidently wrong on exactly the cases it misunderstands — and now it trains on those mistakes, becomes more confident in them, and passes them through the threshold more easily next round. This is confirmation bias, and it is the reason naive self-training sometimes gets worse with every iteration.

Pseudo-labelling amplifies whatever the model already believes. Used with a class-balanced, regenerated, high-threshold selection it is one of the strongest adaptation methods available; used naively it is a machine for manufacturing confident errors.

Maximum Mean Discrepancy: align the distributions explicitly

Adversarial training aligns distributions implicitly, through a game. MMD does it directly, with a differentiable distance you can add to the loss — and with no adversary, so no instability.

MMD maps both sets of features into a reproducing kernel Hilbert space and measures the distance between their means there. The empirical squared form, for $n$ source and $m$ target samples, is:

Read it as: source-to-source similarity, plus target-to-target similarity, minus twice the cross similarity. If the two sets are drawn from the same distribution, the cross term is as large as the within terms and the whole expression goes to zero. With a characteristic kernel such as the Gaussian, $\text{MMD} = 0$ if and only if the distributions are identical.
Python
Copy
1def gaussian_mmd(source, target, sigmas=(1, 2, 4, 8, 16)):2    """Multi-kernel MMD^2 between two (B, D) feature batches."""3    x = torch.cat([source, target], dim=0)4    d2 = torch.cdist(x, x) ** 2                      # squared distances56    # Median heuristic: scale bandwidths by the median pairwise distance.7    base = d2.detach().median().clamp(min=1e-8)8    k = sum(torch.exp(-d2 / (base * s)) for s in sigmas) / len(sigmas)910    n = source.size(0)11    k_ss = k[:n, :n].mean()12    k_tt = k[n:, n:].mean()13    k_st = k[:n, n:].mean()14    return k_ss + k_tt - 2 * k_st1516# Used as an auxiliary loss on the penultimate features:17loss = F.cross_entropy(logits_s, y_s) + 0.5 * gaussian_mmd(feat_s, feat_t)
Using several bandwidths at once matters. A single Gaussian kernel is only sensitive to differences at roughly its own length scale; a mixture over five bandwidths spanning a 16× range detects both fine and coarse mismatches. The median heuristic sets the centre of that range from the data rather than from a guess.
Adversarial (DANN)
MMD-based
Stability
Can oscillate or collapse; seed-sensitive
Stable — it is just an extra loss term
Hyperparameters
$\lambda$ schedule, discriminator architecture, LR balance
Loss weight and kernel bandwidths
Compute
Extra network, extra forward pass
$O(B^2)$ kernel matrix per batch — cheap for $B \le 128$
Alignment strength
Matches full distributions in principle
Matches kernel mean embeddings; weaker but more predictable
Typical use
When you can afford to tune it
When you want a reliable improvement with little tuning
Putting it together on a real benchmark
Office-31 is the standard test bed: 31 object categories photographed in three domains — Amazon product shots on white backgrounds (2,817 images), a DSLR in an office (498 images), and a low-resolution webcam in the same office (795 images). The hard transfer is Amazon → Webcam, because studio product photography and a grainy webcam are genuinely different worlds.
Python
Copy
1for step in range(total_steps):2    xs, ys = next(src_iter)3    xt, _  = next(tgt_iter)          # target labels never touched45    p = step / total_steps6    lambd = 2.0 / (1.0 + np.exp(-10.0 * p)) - 1.078    fs = model.features(xs.cuda()).flatten(1)9    ft = model.features(xt.cuda()).flatten(1)1011    loss = F.cross_entropy(model.classifier(fs), ys.cuda())      # supervised12    loss = loss + 0.5 * gaussian_mmd(fs, ft)                     # alignment13    loss = loss + lambd * domain_adversarial_loss(fs, ft)        # adversarial1415    loss.backward(); optimiser.step(); optimiser.zero_grad()
Published results for single methods on this pair, each with a ResNet-50 backbone and no target labels at all (as reported by Long et al., Conditional Adversarial Domain Adaptation, 2018):
Method (Amazon → Webcam)
Target accuracy
Target labels used
Source-only ResNet-50
68.4%
0
DAN (multi-kernel MMD alignment)
80.5%
0
DANN (domain adversarial)
82.0%
0
JAN (joint MMD over features and predictions)
85.4%
0
CDAN+E (class-conditional adversarial)
94.1%
0
Two lessons sit in that table. Plain distribution alignment, MMD or adversarial, recovers 12 to 14 of the points lost to the domain gap without a single target label. And the biggest jump comes from conditioning the alignment on the predicted class — exactly the fix for the "target cat onto source dog" failure described above. Treat these as benchmark numbers from careful research code; on your own data, expect smaller and noisier gains.
How to run this on a real deployment
The practical consequence is a sequence, ordered by cost, with a stopping rule at each step.
Diagnose the shift type before touching anything. Compare class frequencies between source and target if you have any target labels at all — a big difference means label shift, and threshold recalibration will beat every method in this topic. Sample fifty target images and check by eye whether the labelling rule still holds; if it has changed, that is concept shift and unsupervised adaptation is provably hopeless.
Measure $d_A$. A proxy A-distance below about 0.5 means the shift is mild and your problem is probably something else. Above 1.5 means the domains are trivially separable and adaptation has real work to do. Track this number as you adapt; it should fall.
Run AdaBN. Twenty seconds, no labels, no risk. On the defect detector it recovered 41% of the gap. There is no reason not to do it.
Add an MMD term before you try anything adversarial. It is stable, has two hyperparameters instead of five, and on Office-31 Amazon → Webcam it came within 1.5 points of DANN. Reach for adversarial training when MMD has plateaued and you have budget to tune it.
Then pseudo-label, carefully. Class-balanced top-k selection, labels regenerated each round, source data still in every batch. Like class-conditional alignment, it brings back the class-level information that plain distribution matching ignores.
Always price the annotation alternative. Three to five labels per class is often one afternoon of work, and it is frequently competitive with weeks of tuning an unsupervised method. If someone can label, let them.
One warning that applies to every step: you cannot validate any of this on a labelled target validation set, because if you had one you would be doing supervised adaptation. Practitioners routinely tune unsupervised methods on target test labels and report the result as unsupervised, which is meaningless. Use whatever you legitimately have — source validation accuracy, the domain classifier's $d_A$, the fraction of target samples above the confidence threshold, prediction entropy on target data — and accept the honest cost of that constraint: unsupervised adaptation is much harder to tune reliably than the benchmark tables make it look.