Definition
LoRA freezes the base model and injects trainable low-rank matrices into layers (often attention projections). You store and swap tiny adapters instead of full checkpoints.
QLoRA combines LoRA with quantized base weights for even leaner fine-tunes on consumer GPUs.
In simple terms
Instead of rewriting an entire textbook, you clip in a thin booklet of updates for your course — lighter to print and share.
Where you see it
- Domain adapters for legal, medical, or Somali language tasks.
- Multiple LoRAs on one shared base model in production.
How it works
1.Freeze base weights
Keep the pretrained model fixed.
2.Train low-rank updates
Optimize small A and B matrices.
3.Merge or swap
Merge into weights or load adapters at runtime.
Why it matters
- LoRA made custom LLM fine-tuning accessible without huge GPU budgets.
Often confused
LoRA always matches full fine-tuning quality.
It often comes close, but full updates can still win on some demanding adaptations.