difficulty 3/5
Regularization
Adding a penalty on coefficient size to a model's objective, so it trades a little fit on the training data for far better behaviour on data it has not seen.
Intuition
Least squares has one instruction: make the training error as small as possible. Given enough features it will follow that instruction off a cliff — fitting the noise in the training set, producing enormous coefficients that cancel each other out, and collapsing the moment it meets new data.
Regularization changes the instruction. It adds a second term to the objective that grows with the size of the coefficients, so the model must now justify every coefficient it uses. Fit still matters; it is just no longer free.
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
On the cheat sheet: Validation and generalisation →On the cheat sheet: Linear models →
Sources
- scikit-learn — sklearn.linear_model.Ridgedocs · retrieved 2026-08-31https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Ridge.html
- Deep Learning (2016)book · retrieved 2026-10-07https://www.deeplearningbook.org/contents/regularization.html