difficulty 3/5
Hyperparameter tuning
Searching the settings a model cannot learn for itself. Which search you use matters less than noticing that most settings do not matter, and that a grid spends its budget as if they all did.
Intuition
Some numbers a model works out for itself; others you have to hand it. The second kind are hyperparameters — "inputs to a machine learning algorithm that govern how the algorithm's performance generalizes to new, unseen data" [1, §1] — and the model has no opinion about them, because changing them changes what "the model" even means.
The obvious way to choose them is to try combinations and keep the best. The obvious way to try combinations is a grid, and the grid is where the intuition goes wrong.
A grid assumes every hyperparameter deserves equal attention. Almost none do. The documentation says so plainly: "it is common that a small subset of those parameters can have a large impact on the predictive or computation performance of the model while others can be left to their default values" [3, §3.2]. Once you accept that, the grid's geometry becomes an active liability — it spends its budget trying the same few values of the parameter that matters, over and over, in combination with values of parameters that do not.
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
On the cheat sheet: Validation and generalisation →
Sources
- Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization (2018)paperhttps://ar5iv.labs.arxiv.org/html/1603.06560
- Random Search for Hyper-Parameter Optimization (2012)paperhttps://www.jmlr.org/papers/v13/bergstra12a.html
- scikit-learn User Guide — Tuning the hyper-parameters of an estimatordocs · retrieved 2026-09-06https://scikit-learn.org/stable/modules/grid_search.html