How would you use backpropagation?

What you need to know

The standard loop

1model.train()2for xb, yb in train_loader:3    xb, yb = xb.to(device), yb.to(device)4    optimizer.zero_grad()                    # 1. clear gradients from the last step5    logits = model(xb)                       # 2. forward pass, builds the graph6    loss = criterion(logits, yb)             # 3. one number to minimise7    loss.backward()                          # 4. backprop: fills p.grad for every p8    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)  # 5. optional9    optimizer.step()                         # 6. update weights using p.grad1011model.eval()12with torch.no_grad():                        # no graph, less memory13    for xb, yb in val_loader:14        val_logits = model(xb.to(device))

What each line does to the numbers:

• zero_grad(): PyTorch adds new gradients to whatever is already in p.grad. Without clearing, step 2's gradient is step 1's plus step 2's, and updates grow wrong fast.
• Forward pass: as each operation runs, autograd records it in a graph, together with the tensors it will need later.
• loss.backward(): walks that graph from the loss back to every parameter with requires_grad=True, applying the chain rule. Then it frees the graph.
• Clipping: must sit after backward(), when gradients exist, and before step(), when they are used.
• optimizer.step(): reads each p.grad and updates p according to its rule (SGD, AdamW...).

Useful things you can do with backprop

• Freeze layers: p.requires_grad_(False) stops gradients for those parameters. This is how transfer learning keeps a pretrained backbone fixed.
• Gradient accumulation: skip zero_grad() and step() for a few batches, dividing each loss by the number of batches, to simulate a larger batch on a small GPU.
• Inspect gradients: after backward(), p.grad.norm() tells you whether gradients are vanishing or exploding.

Pitfalls interviewers ask about

• Forgetting model.eval() at validation: dropout stays on and batch norm uses the current batch's statistics, so metrics are noisy and too low.
• Logging loss instead of loss.item(): keeping the tensor keeps its whole graph alive, and memory creeps up every step.
• In-place changes to a tensor autograd still needs, such as x += 1 on an activation, which can raise an error during backward().

A real-life example

A hospital's imaging team fine-tunes a CNN to triage chest X-rays into "urgent" and "routine". Their GPU fits only 16 images of 512×512 at a time, but training is noticeably more stable with an effective batch size of 64. They use gradient accumulation:

1accum = 42optimizer.zero_grad()3for i, (xb, yb) in enumerate(train_loader):          # batches of 164    loss = criterion(model(xb), yb) / accum5    loss.backward()                                   # gradients add up6    if (i + 1) % accum == 0:7        optimizer.step()                              # one update per 64 images8        optimizer.zero_grad()

Dividing by 4 makes the accumulated gradient equal the average over 64 images. Early on, their validation accuracy swung between 71% and 84% from one run to the next with identical weights. The cause was a missing model.eval(): batch norm was using the statistics of each validation batch. Adding it made evaluation deterministic.

Follow-up questions to expect

• "Why does PyTorch accumulate gradients instead of overwriting them?" — Because accumulation is useful: gradient accumulation and models that sum losses from several forward passes both rely on it. The cost is that you must clear gradients yourself.
• "What is the difference between torch.no_grad() and model.eval()?" — no_grad() stops autograd recording, saving memory and time. eval() changes layer behaviour, turning dropout off and making batch norm use running statistics. At validation you need both.
• "Where does the learning-rate scheduler step go?" — After optimizer.step(), either every iteration or every epoch depending on the scheduler.