Definition
Gradient descent repeatedly adjusts parameters in the direction that most reduces the loss. Variants like SGD, Adam, and AdaGrad change how steps are sized and smoothed.
Almost all modern neural network training relies on some form of gradient descent.
In simple terms
Imagine hiking downhill in fog: at each step you feel which way the ground slopes steepest and take a small step that way until you reach a valley (low loss).
Where you see it
- Training classifiers, transformers, and diffusion models.
- Fine-tuning LLMs with AdamW optimizers.
How it works
1.Compute loss
Measure how wrong current predictions are.
2.Compute gradients
Find how each weight affects the loss.
3.Update weights
Subtract learning-rate × gradient.
4.Repeat
Many steps (or epochs) until convergence.
Why it matters
- Understanding gradient descent demystifies how neural nets actually "learn."
Often confused
Gradient descent always finds the global best solution.
Non-convex losses have many valleys; practice settles for good enough local minima.