difficulty 3/5
Ridge regression
Linear regression with a penalty on the sum of squared coefficients — it shrinks every coefficient toward zero without ever removing one, which stabilises models with correlated features.
Intuition
Give least squares two features that measure nearly the same thing and it becomes unstable: it can add 500 to one coefficient and subtract 500 from the other with almost no change in fit. The coefficients become enormous, opposite, and meaningless, and they swing wildly if you refit on slightly different data.
Ridge makes that arrangement expensive. Because the penalty is on squared coefficients, a pair of ±500 coefficients costs enormously more than a pair of ±5. The model is pushed toward spreading influence evenly across correlated features rather than playing them against each other [2, §7.1.1].
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
On the cheat sheet: Linear models →
Sources
- scikit-learn — sklearn.linear_model.Ridgedocs · retrieved 2026-08-31https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Ridge.html
- Deep Learning (2016)book · retrieved 2026-10-07https://www.deeplearningbook.org/contents/regularization.html