Definition
Without nonlinear activations, stacked layers collapse into one linear transform — limited power. Activations like ReLU, GELU, and tanh introduce the nonlinearity deep learning needs.
Choice of activation affects gradient flow, speed, and final accuracy.
In simple terms
Linear layers are straight roads. Activations add turns and curves so the network can draw complex decision boundaries.
Where you see it
- ReLU in classic CNNs and MLPs.
- GELU in many transformer architectures.
- Softmax at classification outputs to produce probabilities.
How it works
1.Weighted sum
Neuron computes linear combination of inputs.
2.Apply activation
Pass through ReLU/GELU/etc.
3.Pass forward
Activated values become the next layer's inputs.
Why it matters
- Activations are a small detail with outsized impact on whether deep nets can learn.
Often confused
Softmax is always the hidden-layer activation.
Softmax is typically for output probabilities; hidden layers use ReLU-family functions.