Linear regression

Intuition

You have a cloud of points and you want one straight line through it that is wrong by as little as possible. Linear regression is that line, generalised to any number of inputs.

Each input gets a weight. The prediction is the inputs multiplied by their weights and added up. Fitting the model means searching for the set of weights that makes the total squared error smallest [2, §3.1.1.2].

The word "linear" describes the weights, not the picture. The model is linear because the prediction is a weighted sum — you can still feed it a squared or logged feature and get a curve.

Formal definition

For an observation with features $x_1$ through $x_p$, the model predicts a weighted sum plus an intercept. The coefficients are chosen to minimise the residual sum of squares — the total of the squared gaps between what was observed and what the line predicts.

In matrix form, scikit-learn states the objective that LinearRegression minimises as the squared Euclidean norm of the residual vector [1, §1.1.1].

How it works

The fitting procedure is direct, not iterative. Because squared error is a smooth bowl-shaped function of the coefficients, it has exactly one lowest point as long as no input is a linear combination of the others, and that point can be solved for in closed form [2, §3.1.1.3]. There is no learning rate and no random initialisation — the same data always produces the same coefficients.

That closed-form solution is what makes linear regression cheap to fit. It is by far the most widely used modelling method [3, §4.1.4.1], and the baseline that everything else usually has to beat.

Where it's used

Linear regression turns up wherever a numeric prediction has to be explained to someone, not just produced [3, Advantages]:

• Pricing and valuation — estimating a house price from floor area, number of rooms and location, where a buyer expects to see which factor moved the number.
• Credit and risk scoring — settings where a model's influence on a decision has to be explained coefficient by coefficient.
• Dose–response studies — estimating how an outcome changes per unit of exposure, with a confidence interval on that estimate.
• Forecasting baselines — the first model fitted on a new problem, to establish the error a more complex model has to improve on.
• Feature screening — reading coefficient magnitudes and significance to decide which inputs are worth keeping.

Why & when

Choose it when interpretability matters as much as accuracy. Each coefficient has a plain-language reading: holding everything else constant, one more unit of this input moves the prediction by this much [5, §5.1.4]. Very few model families offer that.

It also assumes a lot, and those assumptions are the decision criteria:

• the relationship between inputs and target is genuinely additive and linear [3, §4.1.4.1]
• the errors are roughly independent, with roughly constant spread [4, §4.2.1]
• the inputs are not near-duplicates of one another [1, §1.1.1]

Reach for something else when the relationship is visibly curved or has interactions the features don't encode (gradient boosting, splines) [3, Disadvantages], when the inputs vastly outnumber the observations (ridge or lasso) [5, §7.3], or when accuracy matters far more than being able to explain the model.

Worked example

from sklearn.datasets import fetch_california_housing
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split

X, y = fetch_california_housing(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=0
)

model = LinearRegression().fit(X_train, y_train)
pred = model.predict(X_test)

print(f"MSE {mean_squared_error(y_test, pred):.3f}")
names = fetch_california_housing().feature_names
for name, coef in zip(names, model.coef_):
    print(f"{name:>10} {coef:+.3g}")

MSE 0.529
    MedInc +0.433
  HouseAge +0.00929
  AveRooms -0.0986
 AveBedrms +0.593
Population -7.56e-06
  AveOccup -0.00475
  Latitude -0.421
 Longitude -0.434

The first line is the MSE on held-out data — the quantity the fit was minimising, now measured on rows the model never saw [1, §1.1.1].

The coefficients are the whole point of the output. The positive weight on median income (+0.433) says that, holding the other features fixed, higher-income districts carry higher house values — a claim you can take to someone and defend.

Not every sign reads that cleanly. Average rooms gets a negative weight and average bedrooms a large positive one, each the opposite of its sign when fitted alone. The two columns correlate at 0.85, the near-duplicate inputs warned about above, and with such features the least-squares estimate "becomes highly sensitive to random errors in the observed target" [1, §1.1.1].

Variants & extensions

All four stay linear in their coefficients, so each coefficient keeps its reading. Ridge keeps a closed-form solution [5, §7.1.1], and so does polynomial regression, which is least squares on expanded features [1, §1.1.16]. The lasso gives it up and is fitted iteratively, by coordinate descent [1, §1.1.3], and so is the elastic net, which in exchange keeps correlated features together rather than picking one at random [1, §1.1.5].

Pitfalls

A coefficient is not a cause. The model reports association given the features it was shown. Omit a confounder and the coefficient absorbs its effect.

A high score is not a good fit. Adding any feature — including pure noise — never decreases the training score. A model can score well and still be wrong in every way that matters.

Extrapolation is unsupported. The fit describes the range of the training data. Predicting far outside it is arithmetic, not evidence.