Course Content

What are the challenges with gradient descent, and how do you treat them?

What you need to know

Gradient descent repeats one rule: w = w − learning_rate × gradient. It only sees the slope at the current point. Every challenge below comes from that short-sighted view.

The learning rate

The step size decides everything. Too big and each step jumps over the valley floor, so the loss bounces or grows to NaN. Too small and training takes days to go anywhere. There is a separate lesson on this; the practical fix is warmup, then a decaying schedule.

Saddle points and plateaus

A saddle point is a spot where the surface curves up in some directions and down in others, like a horse's saddle. The gradient there is close to zero, so plain gradient descent almost stops. In networks with millions of weights, saddle points and flat plateaus are far more common than bad local minima. Momentum helps: it keeps a running average of past gradients, so the weights keep rolling through a flat region instead of stopping.

Ravines (poor conditioning)

If one weight affects the loss 100 times more strongly than another, the loss surface is a long, narrow valley. The gradient points mostly across the valley, not along it, so the weights zig-zag from wall to wall. The usual cause is unscaled inputs: one feature in rupees (0 to 1,00,000) and another as a ratio (0 to 1). Standardising inputs and using batch or layer normalisation make the valley rounder. Adam also helps, because it gives each weight its own step size.

Vanishing and exploding gradients

In deep networks the gradient is a product of many layer derivatives. If they are mostly below 1 the product shrinks to nothing; above 1 and it explodes. Fixes are ReLU-family activations, He initialisation, normalisation, residual connections and gradient clipping. Two later lessons cover this in depth.

Noise and cost

Full-batch gradient descent over millions of rows is too slow per step. Mini-batches make each step cheap, at the cost of some noise, which momentum smooths out.

Symptom you see	Likely cause	First thing to try
Loss jumps to `NaN` in the first few hundred steps	Learning rate too high, or exploding gradients	Lower the LR 10x, add warmup and gradient clipping
Loss falls very slowly from the start	Learning rate too low, or unscaled inputs	LR range test, standardise features
Loss stalls on a long flat stretch	Plateau or saddle point	Momentum or Adam, a schedule such as cosine decay
Early layers' gradients near zero	Vanishing gradients	ReLU, He init, residual connections, normalisation
Loss curve very jagged	Batch too small or LR too high	Bigger batch, lower LR, momentum

A real-life example

A grocery-delivery company trains a 6-layer network to forecast how many units of each item a dark store will sell tomorrow. Inputs include last week's sales (0 to 5,000 units), price in rupees (10 to 2,000) and a festival flag (0 or 1).

The first run uses plain SGD with learning rate 0.1. The loss hits NaN at step 180. The engineer lowers the rate to 0.01 and the loss stops exploding but barely moves for two hours. Checking the inputs, she finds price and sales on wildly different scales. She standardises every feature to mean 0 and standard deviation 1, switches to AdamW with a 500-step warmup to 0.001 followed by cosine decay, and adds clip_grad_norm_ at 1.0. The loss now falls smoothly and converges in 25 minutes.

No single fix did it. Scaling rounded the valley, warmup and a sensible rate stopped the jumps, and clipping caught the rare huge gradient from a festival-week outlier.

Follow-up questions to expect

"Are local minima a big problem in deep learning?" — Less than people expect. In very high dimensions most points with zero gradient are saddle points, and the local minima that exist tend to have similar loss. Plateaus and saddles are the practical problem.
"What does momentum do?" — It adds a fraction (often 0.9) of the previous update to the current one. Consistent directions build up speed, and directions that flip sign every step cancel out, which damps zig-zagging.
"How do you debug a loss that goes to NaN?" — Lower the learning rate, add gradient clipping, check the inputs for NaN or huge values, and check for things like log(0) in a custom loss.