What is Adaptive Moment Estimation (Adam)?

What you need to know

The two moments

For each parameter, at every step t with gradient g, Adam updates two numbers:

m = β1 × m + (1 − β1) × g          # first moment: average gradient (momentum)v = β2 × v + (1 − β2) × g²         # second moment: average squared gradientm_hat = m / (1 − β1^t)             # bias correction (m and v start at 0)v_hat = v / (1 − β2^t)w = w − lr × m_hat / (sqrt(v_hat) + ε)

• m (momentum) smooths noisy gradients. If the gradient keeps pointing the same way, m grows and the parameter moves faster.
• v (scale) tracks how big this parameter's gradients usually are. Dividing by sqrt(v) shrinks steps for parameters with large gradients and enlarges steps for parameters with small ones.
• Bias correction fixes a start-up problem. m and v begin at zero, so for the first few steps they are too small. Dividing by 1 − β^t scales them back up.
• ε (about 1e-8) avoids division by zero.

What happens to the numbers on step 1

Say a parameter's first gradient is g = 0.2. Then m = 0.1 × 0.2 = 0.02 and v = 0.001 × 0.04 = 0.00004. After bias correction, m_hat = 0.02 / 0.1 = 0.2 and v_hat = 0.00004 / 0.001 = 0.04, so sqrt(v_hat) = 0.2. The step is lr × 0.2 / 0.2 = lr.

Now try a gradient 100 times smaller, g = 0.002. The ratio is still about 1, so the step is still about lr. That is the key idea: Adam's step size depends on the direction and consistency of the gradient, not its raw size. Parameters with tiny gradients are not left behind.

Adam versus AdamW

L2 regularisation adds λ × w to the gradient. In Adam, that extra term is then divided by sqrt(v), so parameters with large gradients get almost no decay, which is not what you wanted. AdamW (decoupled weight decay) applies the decay directly to the weight, outside the adaptive scaling. It usually generalises better and is the default for transformers.

1import torch23model = torch.nn.Linear(20, 1)4opt = torch.optim.AdamW(model.parameters(), lr=1e-3,5                        betas=(0.9, 0.999), weight_decay=0.01)

Costs and caveats

Adam stores m and v for every parameter, so optimiser state is twice the model's size. For a 7-billion-parameter model in 32-bit floats that is about 56 GB just for m and v. On some image-classification benchmarks, well-tuned SGD with momentum has matched or beaten Adam's final accuracy, so many vision recipes still use SGD. For transformers and most new projects, AdamW is the safe starting point.

A real-life example

An e-commerce site trains a recommendation model with an embedding vector for each of 2 million products. A popular phone case appears in thousands of batches a day; a niche hand-made lamp appears in a handful. With plain SGD, the lamp's embedding gets a few tiny updates and stays close to random, so the model never learns what the lamp is like.

With Adam, the lamp's v stays small because its gradients are rare and small, so each of its few updates is relatively large. The phone case has large, frequent gradients, so its steps are scaled down and it does not overshoot. Long-tail products start getting sensible recommendations, which is where much of a marketplace's catalogue lives.

Follow-up questions to expect

• "Why is bias correction needed?" — Because m and v start at zero, early averages are biased towards zero. Without correction, the first steps would be the wrong size, and v is especially slow to warm up with β2 = 0.999.
• "When would you choose SGD with momentum over Adam?" — For well-studied vision tasks with a known recipe, or when you need to squeeze out the last bit of test accuracy and can afford tuning. Adam converges faster with less tuning.
• "What is RMSProp?" — Adam without the momentum term: it divides the gradient by a running average of its square. Adam is roughly RMSProp plus momentum plus bias correction.