Definition
Batch normalization re-centers and re-scales activations using batch statistics, then learns scale and shift parameters. It can reduce internal covariate shift and allow higher learning rates.
LayerNorm is preferred in many transformers; BatchNorm remains common in CNNs.
In simple terms
Standardizing exam scores so each class has a similar average — making later grading (layers) more stable.
Where you see it
- ResNet-style computer vision models.
- Some speech and tabular deep models.
How it works
1.Compute batch stats
Mean and variance over the mini-batch.
2.Normalize
Standardize activations.
3.Scale and shift
Learnable γ and β restore representational power.
Why it matters
- Normalization layers made training very deep networks more practical.
Often confused
BatchNorm is required in every architecture.
Transformers typically use LayerNorm; BatchNorm can behave oddly with tiny batches.