Definition
During training, dropout sets a fraction of activations to zero at random each step, forcing the network not to rely on any single pathway. At inference, all units are active (with scaling).
Related ideas appear as DropPath / stochastic depth in modern architectures.
In simple terms
If teammates randomly sit out of practice, everyone must learn the play — the team cannot depend on one star player.
Where you see it
- Classic MLPs and CNNs with dropout layers.
- Transformers often use attention/residual dropout.
How it works
1.Sample a mask
Randomly zero activations with probability p.
2.Forward and backprop
Train on the thinned network.
3.Disable at inference
Use the full network for predictions.
Why it matters
- Dropout is a simple, effective defense against overfitting in deep nets.
Often confused
More dropout is always better.
Too much dropout underfits; tune it like any hyperparameter.