Definition
RLHF trains a reward model from human comparisons of outputs, then optimizes the LLM (often with PPO or similar) to score highly on that reward while staying close to a reference model.
It is a major reason chat models feel helpful and refuse some harmful requests — with known limitations and biases.
In simple terms
Humans rank two essays; a coach learns what "good" looks like; the student practices to please that coach without forgetting how to write.
Where you see it
- InstructGPT / ChatGPT-style alignment pipelines.
- Preference tuning for tone, safety, and helpfulness.
How it works
1.Collect preferences
Humans rank or rate model outputs.
2.Train reward model
Predict which output humans prefer.
3.RL optimize policy
Update the LLM to maximize reward with KL constraints.
Why it matters
- RLHF (and cousins like DPO) turn raw base models into usable assistants.
Often confused
RLHF makes models truthful about everything.
It optimizes preference scores — which may reward confident tone over factual rigor.