Correlation

Intuition

Correlation compresses the relationship between two variables into a single number between −1 and 1. That compression is the whole point and the whole problem: one number cannot carry the shape of a relationship, and the one it does carry answers a narrower question than most readers assume.

The question it answers is how tightly do these points sit on a straight line. Not how strongly are these related, and not how much does y change when x changes. Those are three different questions, and correlation answers only the first [1, §7.6].

Formal definition

Take each variable, subtract its mean and divide by its standard deviation — what [1, §7.3] calls converting to standard scores. Multiply the two standardised series pointwise and take the mean, as the formula above the page states.

The standardising step is where the interpretation comes from. Covariance alone, which [1, §7.4] defines as the mean product of the two deviations, is in the units of x times the units of y, so it cannot be compared across pairs of variables. Dividing by both standard deviations removes the units and pins the result into [−1, 1].

It also removes something else, which is the first surprise.

How it works

Because both variables are standardised, any scaling of either one cancels. The consequence is that correlation is unrelated to slope [1, §7.6]. A relationship where y rises by a ten thousandth per unit of x and one where it rises by ten thousand have the same correlation, provided both are equally tight.

This is worth holding onto, because "strongly correlated" is often heard as "large effect". A correlation of 0.99 between two variables says the points lie close to a line. It says nothing at all about whether moving x by one unit moves y by a lot or by nothing worth measuring.

Where it's used

As a first screen over many variable pairs, where a correlation matrix is cheap and shows which relationships are worth plotting. As an input to feature selection, where highly correlated features signal redundancy. And as the basis of R², which in simple linear regression is exactly the square of the correlation — so an r of 0.7 corresponds to 49% of the variance, not 70% of anything.

[2, §1.3.3.26] lists the questions a scatter plot answers, and the list itself makes the point: are X and Y related and are X and Y linearly related are two of them, asked separately. Correlation answers only the second.

Why & when

Use it when you have reason to expect a linear relationship, when both variables are roughly symmetric, and when you will look at the scatter plot anyway. Reach for a rank correlation instead when the relationship is monotone but curved, or when outliers are likely.

Do not use it alone. The number is a summary of a picture, and the summary is faithful only for one shape of picture.

Worked example

Four demonstrations, all in scripts/correlation.py. The first checks the claim about slope.

import numpy as np


def pearson(x, y):
    """Standard scores, then the mean of their products (Think Stats 7.4-7.5)."""
    zx = (x - x.mean()) / x.std()
    zy = (y - y.mean()) / y.std()
    return float((zx * zy).mean())


x = np.linspace(-3, 3, 400)
for m in (0.0001, 0.01, 1.0, 10000.0):
    print(f"{'m * x':>14}{m:>12g}{pearson(x, m * x):>12.6f}")

  the same relationship at four steepnesses
           y =       slope           r
         m * x      0.0001    1.000000
         m * x        0.01    1.000000
         m * x           1    1.000000
         m * x       10000    1.000000

Eight orders of magnitude of slope, one value of r.

Zero correlation, perfect dependence

[1, §7.6] warns that a correlation near zero does not license the conclusion that there is no relationship — Pearson's r measures linear relationships only, and where the relationship is nonlinear it understates the strength. Taken to the limit:

  y is an exact function of x in every row below
          relationship           r    R^2 of the right fit
        y = x (linear)    1.000000                1.000000
   y = x^2 (symmetric)   -0.000000                1.000000
               y = |x|   -0.000000                0.937506
         y = cos(pi x)    0.000000                0.893510

In every row, y is determined by x exactly — there is no noise anywhere in this table. Three of the four report a correlation of zero. A variable can be a perfect function of another and correlate with it not at all.

The same summary, four different pictures

  Anscombe's quartet: four datasets, one summary
   set    mean x     var x    mean y     var y         r           fitted line
     I      9.00     11.00     7.501     4.127     0.816      y = 3.00 + 0.50x
    II      9.00     11.00     7.501     4.128     0.816      y = 3.00 + 0.50x
   III      9.00     11.00     7.500     4.123     0.816      y = 3.00 + 0.50x
    IV      9.00     11.00     7.501     4.123     0.817      y = 3.00 + 0.50x

Anscombe constructed these in 1973, and they remain the most economical argument for plotting the data. Every number a correlation analysis reports agrees across all four; only the first is a linear relationship with scatter. The second is a clean parabola, the third is an exact line with one contaminated point, and the fourth contains no variation in x at all except in a single observation — so one point is setting both the slope and the correlation.

What ranks recover

  Pearson against Spearman
              relationship     Pearson    Spearman
      y = exp(x), monotone      0.9012      1.0000
      y = log(x), monotone      0.9248      1.0000
    y = x with one outlier     -0.1792      0.8891
    y = x^2 on symmetric x     -0.0000     -0.0022

Replacing values with their ranks, which is the other transform [1, §7.3] describes, fixes the first two rows exactly: a monotone relationship is perfectly monotone in the ranks whatever its shape.

The third row is the one to remember. Fifty points lying exactly on a line, plus one outlier, and Pearson reports −0.18 — not merely weakened but reversed in sign. Spearman reports 0.89, because a rank cannot be extreme: the outlier is just the largest value, not a value forty times larger than it should be.

The last row is the limit of the trick. Ranks do not rescue a relationship that is not monotone.

Pitfalls

Reading r as an effect size. Correlation is scale-free by construction, so it cannot tell you how much y moves. If that is the question, fit a line and report the slope in its units [1, §7.6].

Reading r near zero as independence. It licenses only one conclusion: no linear relationship. The table above has three counterexamples where the dependence is exact [1, §7.6].

Squaring it in your head. r = 0.7 is not "70 percent related". Its square is the share of variance accounted for, and 0.49 is a different-feeling number from 0.7 [3, §10.5].

Letting a single point decide. One outlier moved a correlation from +1 to −0.18 above [1, §7.7]. Always plot before believing [1, §7.6].

Treating it as evidence of causation. Correlation is symmetric and causation is not, and a third variable can produce the whole association [1, §7.8] — see confounding, where the same regression, on the same kind of data, is right in one world and badly wrong in another.