Definition
A loss function compares predictions to targets and returns a number. Cross-entropy is common for classification; MSE for regression; specialized losses exist for ranking, generation, and contrastive learning.
The choice of loss shapes what the model prioritizes during training.
In simple terms
The loss is a referee's score for how badly you missed. Training is practicing until that score drops.
Where you see it
- Cross-entropy for spam classifiers.
- Contrastive losses for embedding models.
- RLHF uses reward models instead of classic supervised loss.
How it works
1.Predict
Model outputs scores or values.
2.Compare to target
Loss formula produces an error signal.
3.Backpropagate
Gradients flow from loss to update weights.
Why it matters
- Wrong loss → wrong behavior. Matching loss to the task is a core ML skill.
Often confused
Lower training loss always means a better model.
You can overfit; validation loss and real-world metrics matter more.