MSE

Intuition

Take every prediction, measure how far off it was, square that gap, and average the results. That average is the MSE.

Squaring does two things at once. It removes the sign, so overshooting by 3 and undershooting by 3 count equally. And it punishes large misses much harder than small ones — being wrong by 10 costs a hundred times as much as being wrong by 1, not ten times. A model tuned on this metric will work hard to avoid the occasional disaster, even at the cost of being slightly worse on average [2, §3.1.1.2].

Formal definition

Sum the squared residuals across all observations and divide by the number of observations. The result is always non-negative, and zero only when every prediction is exactly right [1, return value].

The units are the square of the target's units. If you are predicting prices in dollars, the MSE is in dollars squared — which is why its square root, the RMSE, is often reported instead when the number has to be read by a human.

How it works

MSE is used in two distinct roles, and confusing them causes real problems:

• As a loss function, minimised during fitting. This is what linear regression is doing when it picks its coefficients [2, §3.1.1.2].
• As an evaluation metric, computed on held-out data to estimate how the model will behave on data it has never seen [3, §3.6.1].

The same formula, but a low training value and a low test value mean very different things. A model can drive the first to nearly zero by memorising the training set while the second gets worse.

Where it's used

• Fitting regression models — the default objective for least squares.
• Model selection — comparing candidate models on the same held-out split.
• Early stopping — watching validation MSE rise to decide when to stop training.
• Hyperparameter search — as neg_mean_squared_error in scikit-learn, negated so that higher is better, matching the convention that "higher return values are better than lower return values" [4, §3.4.3.1].

Why & when

Reach for MSE when large errors are genuinely worse than small ones — forecasting demand where a big miss empties a warehouse, or estimating a dose where overshoot is dangerous. The squaring encodes that judgement directly.

Reach for something else when your data has outliers you do not want the model chasing. A single wild point can dominate the entire MSE and drag the fit toward it. Mean absolute error treats all errors proportionally and is the usual alternative; Huber loss sits between the two.

Worked example

from sklearn.metrics import mean_squared_error

y_true = [3, -0.5, 2, 7]
y_pred = [2.5, 0.0, 2, 8]

mean_squared_error(y_true, y_pred)
# 0.375

Worth doing by hand once. The errors are 0.5, −0.5, 0 and −1. Squared they are 0.25, 0.25, 0 and 1, summing to 1.5. Divided by the four observations, that is 0.375 [1, examples].

The same errors, four ways. One set of 200 predictions, scored by six metrics while one thing changes at a time: one prediction goes badly wrong, the target is shifted to sit near zero, the units are made ten times smaller.

import numpy as np
from sklearn.metrics import (
    mean_absolute_error,
    mean_absolute_percentage_error,
    mean_squared_error,
    r2_score,
)

rng = np.random.default_rng(20261021)
N, DELTA = 200, 5.0

y = rng.normal(50, 10, N)
pred = y + rng.normal(0, 3, N)


def huber(t, p, delta=DELTA):
    r = np.abs(t - p)
    return float(np.mean(np.where(r <= delta, 0.5 * r**2, delta * (r - 0.5 * delta))))


outlier = pred.copy()
outlier[0] += 60
shift = y.min() - 0.05  # the smallest target lands at 0.05

cases = {
    "clean": (y, pred),
    "one-outlier": (y, outlier),
    "near-zero": (y - shift, pred - shift),
    "x10-units": (10 * y, 10 * pred),
}

print(f"  {N} targets, the same noise in every row; Huber delta = {DELTA:g}")
print(f"{'case':>13}{'MSE':>10}{'RMSE':>9}{'MAE':>9}{'R²':>8}{'MAPE':>10}{'Huber':>9}")
for name, (t, p) in cases.items():
    mse = mean_squared_error(t, p)
    print(
        f"{name:>13}{mse:>10.2f}{np.sqrt(mse):>9.2f}{mean_absolute_error(t, p):>9.2f}"
        f"{r2_score(t, p):>8.3f}{mean_absolute_percentage_error(t, p):>10.3f}{huber(t, p):>9.2f}"
    )

  200 targets, the same noise in every row; Huber delta = 5
         case       MSE     RMSE      MAE      R²      MAPE    Huber
        clean      8.99     3.00     2.42   0.926     0.050     4.40
  one-outlier     27.07     5.20     2.72   0.777     0.059     5.84
    near-zero      8.99     3.00     2.42   0.926     0.215     4.40
    x10-units    898.78    29.98    24.22   0.926     0.050   109.03

One bad prediction triples MSE and barely moves MAE. A single miss of 60 among 200 takes MSE from 8.99 to 27.07 and R² from 0.926 to 0.777, while MAE goes from 2.42 to 2.72. The squaring is what gives one error that weight.

Move the target near zero and only MAPE moves. The errors are the same to the last digit, so MSE, RMSE, MAE, R² and Huber do not change; MAPE goes from 0.050 to 0.215, almost all of it from the one target that now sits at 0.05.

Change the units and only R² and MAPE stay put. Ten times every value makes MSE a hundred times larger and RMSE and MAE ten times; Huber grows by neither, 24.8 times, because δ stays at 5 while the errors grow past it.

Variants & extensions

Pitfalls

MSE is not comparable across datasets. It carries the scale of the target [2, §3.1.1.2]. An MSE of 4 is excellent when predicting house prices in thousands and impossible when predicting a probability, where no squared error can exceed 1.

Training MSE is not a performance estimate. It only tells you how well the model fits data it has already seen. The number that matters is computed on data held back from fitting [3, §3.6.1].

The squaring is a decision, not a technicality. By choosing MSE you have declared that one error of 10 is worse than ten errors of 1. If that is not true for your problem, the metric is quietly optimising for the wrong thing [2, §3.1.1.2].