Which penalty, or none?

Seven linear models side by side, the pairs people confuse, and one run in which the best predictor is not the best selector of features.

prints on 2 A4 pagesfrom 8 pages and 1 cataloguefree to read · PDF with a free accountUpdated 23 Oct 2026

Download PDFTest yourself

The PDF and the Print button ask you to sign in; a free account is enough. Reading and Test yourself are open to everyone.

Side by side

The first columns are the Linear model family catalogue’s; the rest come from each page.

Say what each one answers, what it cannot see and when to reach for it, then reveal its row.

Model	Penalty	Zeroes coefficients?	Hyperparameters	Reach for it when	Trap
Linear regression Reveal Linear regression	none	No	none	interpretability matters as much as accuracy	A coefficient is not a cause.
Ridge Reveal Ridge	L2 — ‖w‖²	No	alpha	most features carry some signal and the model must be stable	Ridge never gives you a sparse model.
Lasso Reveal Lasso	L1 — ‖w‖₁	Yes	alpha	most features are irrelevant and the model must say which	Correlated features make selection unstable.
Elastic Net Reveal Elastic Net	L1 + L2	Yes	alpha, l1_ratio	you want sparsity and the features are correlated	Two hyperparameters means a 2-D search.
Bayesian ridge Reveal Bayesian ridge	L2, learned	No	none tuned	you need calibrated uncertainty, or data is scarce	A wide interval is information, not a defect.
Polynomial Reveal Polynomial	none by itself	No	degree	the curve is smooth and low-order	Extrapolation is catastrophic, not merely unreliable.
Logistic Reveal Logistic	L2 by default	with l1_ratio	C, l1_ratio	you need probabilities you can trust and coefficients to defend	C runs backwards from alpha.

A formula is printed only when it is shorter than the words; the others are said in words, and each page has its formula.

Easy to confuse

The “what is the difference between…” questions, in two lines each.

Ridge regression vs Lasso regression
Ridge shrinks every coefficient and keeps them all; lasso sets some exactly to zero.
Ridge when most features matter; lasso when most do not and the model must prove which.
Lasso regression vs Elastic Net
Of a group of correlated features lasso tends to keep one, almost at random; elastic net tends to keep the group.
Elastic net when features come in correlated groups, or when you do not know which penalty fits.
Linear regression vs Logistic regression
Both score a weighted sum of the features; logistic regression turns it into a probability.
Logistic for a yes-or-no outcome, read as log-odds; linear for a number.
α vs C
Ridge and lasso's α is the penalty's strength; logistic regression's C is its inverse.
Larger α means more penalty; larger C means less. Search both on a log scale.

One real run

Fifty features, five with signal and each with a near-twin; each penalty tuned by cross-validation.

Model	Test MSE	Nonzero	True kept	Twins	Stable
ols	6.95	50	5	5	1.00
ridge	6.14	50	5	5	1.00
lasso	4.46	20	4	2	0.45
elastic-net	5.10	35	5	5	0.63

“Lasso predicts best and still chooses badly.”

“Elastic net keeps the pairs together.”

“Ridge shrinks and keeps everything.”

The run and these lines are the worked example on Regularization, as it printed: 50 features, 5 with signal, each with a near-twin; 100 training rows

Which one?

Is the outcome a class?
- Yes, yes or no Logistic regression
Do most features matter?
- Yes, most carry some signal Ridge regression
- No, most are noise Lasso regression
- Unsure, and they come in groups Elastic Net
Is the relation curved?
- Smoothly, at low order Polynomial regression
Do you need uncertainty on each prediction?
- Yes, or there is little data Bayesian linear regression