Definition
Parameters (weights) are learned during training. Hyperparameters are knobs you set: learning rate, number of layers, dropout rate, temperature at inference, and more.
Tuning them well often matters as much as model architecture.
In simple terms
Weights are the recipe the oven invents; hyperparameters are oven temperature and bake time — you set those before cooking.
Where you see it
- Grid search or Bayesian optimization over learning rates.
- Choosing context length and temperature for LLM APIs.
How it works
1.Pick candidates
Define a search space of settings.
2.Train and evaluate
Measure validation performance for each config.
3.Select best
Keep settings that generalize; retest on a holdout set.
Why it matters
- Poor hyperparameters can make a strong architecture look weak.
Often confused
More layers always help.
Depth is a hyperparameter — too much capacity without data causes overfitting.