What is a computational graph? How is it useful for Deep Learning?

What you need to know

A tiny graph, by hand

Take one weight w, one bias b, an input x = 2, and a target of 10:

z    = x * w + b         with w = 3, b = 1  ->  z = 7loss = (z - 10)^2                            ->  loss = 9

The graph is: x and w feed a multiply node, its output and b feed an add node giving z, then subtract and square give the loss.

To go backwards, apply the chain rule node by node:

dloss/dz = 2 * (z - 10)   = -6dz/dw    = x              =  2   ->  dloss/dw = -6 * 2 = -12dz/db    = 1                     ->  dloss/db = -6 * 1 = -6

PyTorch does exactly this for you:

1import torch23x = torch.tensor(2.0)4w = torch.tensor(3.0, requires_grad=True)5b = torch.tensor(1.0, requires_grad=True)67loss = (x * w + b - 10) ** 28print(loss.grad_fn)          # <PowBackward0 ...>  the last node in the graph9loss.backward()              # walk the graph in reverse10print(w.grad, b.grad)        # tensor(-12.) tensor(-6.)

Every tensor produced by an operation carries a grad_fn, a link to the node that made it. backward() follows those links from the loss back to the leaves. The negative gradients say: increase w and b to reduce the loss, which makes sense because z = 7 is below the target 10.

Why the graph is useful

• Automatic gradients — for a model with millions of weights, the same mechanism gives every gradient in roughly the cost of one extra forward pass.
• Memory planning — the framework knows which intermediate values the backward pass needs, and can free the rest. torch.no_grad() and torch.inference_mode() skip recording the graph entirely, saving memory at inference.
• Optimisation and export — a captured graph can be compiled (operations fused, memory reused), exported to ONNX, or split across devices.

Dynamic vs static graphs

Modern frameworks offer both: write eagerly, then compile for speed.

A real-life example

A team fine-tunes a speech-command model and notices GPU memory growing every batch until it crashes. The cause: they logged the loss with total += loss instead of total += loss.item(). Because loss is attached to its graph, adding it to total kept every batch's graph alive. .item() returns a plain Python number with no graph, and memory stays flat. Understanding the graph turned a mystery crash into a one-word fix.

Follow-up questions to expect

• "Why do we call optimizer.zero_grad()?" — PyTorch adds new gradients to .grad instead of replacing them, so you clear them before each backward pass. Accumulation is useful on purpose for simulating larger batches.
• "What does .detach() do?" — Returns a tensor with the same data but cut off from the graph, so no gradients flow through it.
• "Is backpropagation the same as autograd?" — Backpropagation is reverse-mode automatic differentiation applied to a neural network's loss.