Definition
The Transformer architecture, introduced in the paper "Attention Is All You Need" (2017), replaced recurrent layers with self-attention for sequence modeling.
Self-attention lets each token in a sequence weigh the importance of every other token when building its representation — enabling long-range dependencies without sequential processing bottlenecks.
In simple terms
Reading a paragraph, you don't process words strictly left-to-right in isolation. You constantly look back and ahead to resolve pronouns and context. Transformers formalize that 'looking around' with attention scores between all word pairs.
Where you see it
- GPT, Claude, and Llama families are Transformer-based language models.
- Google Translate and modern MT systems use Transformers.
- Vision Transformers (ViT) apply the same idea to images.
- Whisper and other speech models use Transformer encoders/decoders.
How it works
1.Tokenize input
Text is split into tokens and converted to embedding vectors.
2.Self-attention layers
Each layer computes attention weights — how much each token should focus on others — and updates representations.
3.Feed-forward blocks
Position-wise neural networks transform each token representation further.
4.Stack and predict
Many layers are stacked; the final layer outputs logits for the next token, class label, or task head.
Why it matters
- Transformers are the foundation of virtually all state-of-the-art language and multimodal models today.
- Understanding attention clarifies why context windows, scaling, and compute costs behave the way they do.
Often confused
Transformers memorize training text verbatim.
They learn distributed representations; verbatim recall is possible but not the primary mechanism for generalization.
Attention is the same as search.
Attention is a differentiable weighting mechanism inside a neural network — related in spirit to retrieval but mathematically distinct.