Definition
Quantization maps high-precision floats (FP16/FP32) to lower-bit integers or floats. Techniques include post-training quantization and quantization-aware training.
It enables running larger LLMs on smaller GPUs or CPUs with some quality tradeoff.
In simple terms
Storing music as a smaller compressed file — you save space and bandwidth, sometimes with a slight fidelity loss.
Where you see it
- llama.cpp and GGUF quantized models on laptops.
- INT8/FP8 inference servers in production.
- QLoRA fine-tuning on a single consumer GPU.
How it works
1.Choose bit width
8-bit, 4-bit, or mixed precision schemes.
2.Calibrate scales
Map float ranges to integer grids.
3.Run inference
Kernels compute with quantized weights.
Why it matters
- Quantization is how open models become practical on real hardware budgets.
Often confused
Any quantization is lossless.
Lower bits usually cost some quality — evaluate on your task.