Metrics & losses · 7 compared
Regression metrics
Squared, absolute, relative or logarithmic — each regression metric encodes a different belief about what a bad prediction costs, and choosing one is making that claim.
Comparison
| Name What it measures | Units | Outliers | Requires |
|---|---|---|---|
| MSE Average squared error. One miss of 10 counts as much as a hundred misses of 1. Full page → | target² | Highly sensitive | Any real target |
| RMSE Square root of MSE, so the number reads in the target's own units. Same ranking, readable answer. Full page → | target | Highly sensitive | Any real target |
| MAE Average absolute error. Every unit of error counts the same, whatever its size. Full page → | target | Resistant | Any real target |
| R² Share of the mean-predictor's error that the model removed. Unitless, but not comparable across datasets — each one's variance sets the scale. Full page → | none | Sensitive (built on MSE) | Non-constant target |
| MAPE Error as a fraction of the true value. Scale-free and intuitive, and it explodes near zero. Full page → | none (ratio) | Sensitive to small y | y far from zero |
| MSLE Squared error on logs, so being wrong by a factor costs the same at any magnitude. Full page → | none (log ratio) | Compresses large y | y ≥ 0 |
| Huber Quadratic near zero, linear beyond a threshold. MSE's smoothness with MAE's resistance. Full page → | target (δ-dependent) | Resistant | Any real target |
On the cheat sheet: Regression metrics and losses →
One dataset, five verdicts
Run the same four predictions through several of these and they disagree — not about the numbers, but about which mistake mattered:
Those are all y_true = [3, -0.5, 2, 7] against y_pred = [2.5, 0, 2, 8]. The
errors are 0.5, 0.5, 0 and 1. Under MSE the single error of 1 contributes two
thirds of the total; under MAE it contributes half; under MAPE the error of 0.5
against a true value of −0.5 dominates everything, because relative to that value
it is a 100% miss.
Same predictions. Different answer to "what went wrong".
The choice is about cost, not about accuracy
There is no most accurate metric here. Each encodes a claim:
- Squared (MSE, RMSE, R²) — large errors are disproportionately bad. One 10-unit miss is worse than ten 1-unit misses.
- Absolute (MAE) — errors cost in proportion to size. Ten 1-unit misses and one 10-unit miss are equally bad.
- Relative (MAPE, MSLE) — what matters is the factor, not the difference. Being 10% off is equally bad at any scale.
- Hybrid (Huber) — proportional in the tail, squared near zero, because you want smooth optimisation without outliers steering the fit.
Pick the row that matches how your problem actually loses money, then report the number in units someone can act on.
What each one fits
An under-appreciated consequence: the metric you train on decides what the model predicts.
Minimising squared error fits the conditional mean; minimising absolute error fits the conditional median; minimising MSLE fits something closer to the geometric mean of 1 + y [2, §6.2.1.2]. On right-skewed data these are noticeably different numbers, so an MAE-trained model sitting lower than an MSE-trained one is behaving correctly, not underfitting.
Practical advice
Report two: one absolute number in the target's units so people can act on it (RMSE or MAE), and R² for the share of variance explained — on this dataset only, since R² is not comparable across datasets either. Never report MSE alone — its units are the target squared, which no one can interpret [1, return value].
Check min(abs(y_true)) before choosing MAPE and min(y_true) before choosing
MSLE. Both have domain restrictions that turn them from useful into meaningless
without warning.
Sources
- scikit-learn — sklearn.metrics.mean_squared_errordocs · retrieved 2026-08-30https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_error.html
- Deep Learning (2016)bookhttps://www.deeplearningbook.org/contents/mlp.html