Correlation: Pearson, Spearman and Kendall

Three measures of association between two variables, each with its bounds proved. Pearson's coefficient as the cosine of an angle, with the equality cases; Spearman's as Pearson's coefficient of the ranks, with its rank-difference formula and what ties do to it; Kendall's as a count of concordant pairs and as a cosine of sign vectors; their population values, with the arcsine formulas for the bivariate normal proved; exact and permutation tests of zero correlation; and Fisher's z interval, with the limiting variance of r derived for any population with four moments. Worked problems compute the three coefficients by hand, the null distribution of Kendall's statistic for four pairs, the z interval, and the permutation variance of Spearman's coefficient.

Twelve adults drawn from the 2017–2018 National Health and Nutrition Examination Survey (NHANES) range in age from 19 to 77, and their systolic blood pressures from 97 to 148 mm Hg. Older adults in the twelve tend to have higher pressures. Pearson's correlation coefficient puts the strength of that tendency at 0.584, Spearman's at 0.602 and Kendall's at 0.439. Why do three standard measures of one association give three different numbers, what does each of them measure, and how far can a value from twelve adults be trusted?

A correlation coefficient is a number between $-1$ and $1$ that measures how closely two variables move together: near $1$ when large values of one go with large values of the other, near $-1$ when they go with small values, near 0 when neither tendency dominates. The three coefficients of this article differ in what "move together" means. Pearson's measures closeness to a straight line, Spearman's closeness to some increasing or decreasing relation, and Kendall's the balance of pairs of people ordered the same way by both variables against pairs ordered oppositely.

First we read Pearson's coefficient as the cosine of an angle and prove its bounds with their cases of equality. Then we define the two rank coefficients and prove their formulas, and we find the population quantities the three estimate, including exact formulas for the bivariate normal. Next we test whether a correlation is zero, exactly and by permutation. Finally we derive the large-sample variance of Pearson's coefficient and Fisher's $z$ transformation, and measure how well the $z$ interval covers on the survey's adults. The article assumes Expectation, Variance and Covariance, which proves the bounds for the population correlation, and uses the projection theorem of Sampling Distributions from the Normal for the exact test.

This is the first article of Linear Models, and the unit's other articles use its notation: $r$ for Pearson's sample coefficient and $\rho$ for the population value, $r_S$ and $\rho_S$ for Spearman's, $\hat\tau$ and $\tau$ for Kendall's.

The data

The unit's running population is the 5,176 adults (age 18 and over) examined in NHANES 2017–2018 who have at least one systolic reading and a recorded body mass index. Each adult's pressure is the mean of the available readings, as in the course articles. The adults are treated as a finite population to sample from, without the survey's weights, so the numbers describe these 5,176 people and not the United States. The twelve adults were drawn from them at random with a fixed seed; the script tools/experiments/statistics/correlation-coefficients.py computes every number on this page.

Adult 1 2 3 4 5 6 7 8 9 10 11 12
Age (years) 54 39 43 61 63 69 19 21 28 27 77 67
Pressure (mm Hg) 129.33 116.00 119.33 148.00 144.00 125.33 106.67 111.33 119.33 97.33 112.67 123.33

Adults 3 and 9 have the same pressure, 119.33 (the mean of three readings that sum to 358). That tie matters for the rank coefficients below.

Pearson's coefficient is a cosine

For $n$ pairs $(x_i, y_i)$ with means $\bar{x}$ and $\bar{y}$, Pearson's correlation coefficient is

$$r = \frac{\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum_{i=1}^n (x_i - \bar{x})^2\,\sum_{i=1}^n (y_i - \bar{y})^2}} = \frac{S_{xy}}{\sqrt{S_{xx}S_{yy}}},$$

defined when neither variable is constant. Simple Linear Regression introduces it as the average product of $z$-scores and shows that the least squares slope is $r\,s_y/s_x$ and that $R^2 = r^2$ for a straight-line fit. Here we look at the same formula as geometry.

Here is the idea. Put the $n$ deviations of $x$ in a vector $\mathbf{a} = (x_1 - \bar{x}, \ldots, x_n - \bar{x})$ and those of $y$ in $\mathbf{b}$. The numerator of $r$ is the dot product $\mathbf{a}\cdot\mathbf{b}$, and the two sums of squares are the squared lengths $\lVert\mathbf{a}\rVert^2$ and $\lVert\mathbf{b}\rVert^2$. So $r = \mathbf{a}\cdot\mathbf{b}/(\lVert\mathbf{a}\rVert\,\lVert\mathbf{b}\rVert)$, which in two or three dimensions is the cosine of the angle between the two vectors. Fisher (1915, §3, p. 510) used exactly this picture: he observed that $r$ "measures the cosine of the angle" between the two sample points once the means are removed, and derived the sampling distribution of $r$ from it.

A cosine cannot exceed 1 in absolute value. In $n$ dimensions that is the Cauchy–Schwarz inequality for vectors, and we prove it with an identity that also gives the cases of equality. For any two vectors $\mathbf{a}$ and $\mathbf{b}$ in $n$ dimensions, define their cosine as $\mathbf{a}\cdot\mathbf{b}/(\lVert\mathbf{a}\rVert\,\lVert\mathbf{b}\rVert)$ when neither is zero.

Lemma 1 (Lagrange's identity). For any numbers $a_1, \ldots, a_n$ and $b_1, \ldots, b_n$,

$$\Big(\sum_i a_i^2\Big)\Big(\sum_j b_j^2\Big) - \Big(\sum_i a_ib_i\Big)^2 = \frac12\sum_{i=1}^n\sum_{j=1}^n (a_ib_j - a_jb_i)^2.$$

Proof. Expand the square on the right: $(a_ib_j - a_jb_i)^2 = a_i^2b_j^2 + a_j^2b_i^2 - 2a_ib_ia_jb_j$. Summed over all $i$ and $j$, the first two terms each give $\sum_i a_i^2\sum_j b_j^2$, and the third gives $-2(\sum_i a_ib_i)^2$. Half of the total is the left side. $\square$

Theorem 1 (bounds on Pearson's coefficient). If neither $x$ nor $y$ is constant, then $-1 \le r \le 1$. Further, $r = 1$ exactly when the points lie on a line with positive slope, $y_i = \bar{y} + c\,(x_i - \bar{x})$ for some $c > 0$ and every $i$, and $r = -1$ exactly when they lie on a line with negative slope.

Proof. The right side of Lemma 1 is a sum of squares, so $(\mathbf{a}\cdot\mathbf{b})^2 \le \lVert\mathbf{a}\rVert^2\lVert\mathbf{b}\rVert^2$, which is $r^2 \le 1$. Equality holds exactly when $a_ib_j = a_jb_i$ for every $i$ and $j$. Since $\mathbf{a} \ne \mathbf{0}$, pick $j$ with $a_j \ne 0$ and set $c = b_j/a_j$; then $b_i = c\,a_i$ for every $i$, and conversely $\mathbf{b} = c\,\mathbf{a}$ makes every term zero. With $\mathbf{b} = c\,\mathbf{a}$ and $c \ne 0$ (because $\mathbf{b} \ne \mathbf{0}$), $r = c\lVert\mathbf{a}\rVert^2/(\lvert c\rvert\,\lVert\mathbf{a}\rVert^2)$, which is $1$ when $c > 0$ and $-1$ when $c < 0$. The condition $b_i = c\,a_i$ is the line $y_i - \bar{y} = c\,(x_i - \bar{x})$. $\square$

The population version, $-1 \le \rho \le 1$ with equality only for an exact linear relation, is Theorem 7 of Expectation, Variance and Covariance, proved there from the variances of the sum and the difference of standardized variables. (That page works with the 5,247 adults who have a pressure reading, for whom the correlation of age and pressure is 0.4866; this unit's 5,176 also have a body mass index, and their value is 0.486.) Theorem 1 is the sample version, and it is in fact a special case: the sample correlation is the population correlation of the distribution that puts probability $1/n$ on each of the $n$ points.

Two further properties follow from the formula at once. Changing units or origin, $x \mapsto \alpha x + \beta$ and $y \mapsto \gamma y + \delta$ with $\alpha\gamma > 0$, multiplies $\mathbf{a}$ by $\alpha$ and $\mathbf{b}$ by $\gamma$ and leaves the cosine unchanged; with $\alpha\gamma < 0$ it changes the sign. And $r$ is symmetric in $x$ and $y$, unlike the regression slope.

For the twelve adults, $r = 0.584$. For all 5,176 adults the population value is $\rho = 0.486$.

What Pearson's coefficient does not measure

Theorem 1 says what $r = \pm 1$ means: a perfect straight line. It does not say that a smaller $\lvert r\rvert$ means a weaker relation. Three things lower $r$ without weakening the dependence, and the survey shows each.

  • A curved relation. Pearson's coefficient measures the straight-line part of the association only; Visualizing Data with Multiple Features gives the survey's example of diastolic pressure and age.
  • Skew and extreme values. Body mass index is skewed to the right; its fourth standardized moment in the population is 5.81, against 3 for a normal variable. Its correlation with pressure is 0.153 by Pearson's coefficient and 0.187 by Spearman's (defined below). Taking its logarithm raises Pearson's value to 0.159.
  • A restricted range. Among the 5,176 adults the correlation of age and pressure is 0.486. Within age bands it is much smaller: 0.173 among the 1,731 adults aged 18 to 39, 0.233 among the 1,563 aged 40 to 59, and 0.182 among the 1,882 aged 60 to 80. Nothing about the dependence changed. Narrowing the range of $x$ shrinks the spread that a common trend can explain, while the scatter around the trend stays. Theorem 8 of Expectation, Variance and Covariance makes this exact for a straight-line trend: $\rho^2$ is the share of $\operatorname{Var}(Y)$ removed by the best line, and the removed part, $b^2\operatorname{Var}(X)$, shrinks with $\operatorname{Var}(X)$.

Example (a correlation is not a slope). "The correlation among adults over 60 is lower, so pressure rises more slowly with age after 60." The correlation does not say that. A lower correlation in a band can come entirely from the band's narrower range of ages. Comparing how fast pressure rises needs the slopes, which The Geometry of Least Squares computes from the same data.

Spearman's coefficient: Pearson's of the ranks

Replace each $x_i$ by its rank $R_i$ among $x_1, \ldots, x_n$ (1 for the smallest, $n$ for the largest) and each $y_i$ by its rank $S_i$. Tied values share the average of the ranks they would occupy, their midrank. Spearman's rank correlation coefficient $r_S$ is Pearson's coefficient computed on the two vectors of ranks. Spearman (1904) set out to measure association by comparing each person's rank on one faculty with the rank on another; the formula proved below is the standard form of that idea.

By Theorem 1, $-1 \le r_S \le 1$. With no ties, $r_S = 1$ exactly when the ranks agree, that is, when $y$ is a strictly increasing function of $x$ on the sample. To see this, note that both rank vectors are then arrangements of $1, \ldots, n$, so their deviations from the mean $(n+1)/2$ have the same length. Theorem 1 gives equality only for $\mathbf{S} - \bar{S}\mathbf{1} = c\,(\mathbf{R} - \bar{R}\mathbf{1})$ with $c > 0$, and equal lengths force $c = 1$. So $r_S$ measures closeness to a monotone relation, of any shape.

When there are no ties a shorter formula holds, in terms of the rank differences $d_i = R_i - S_i$.

Theorem 2 (Spearman's formula). If neither variable has ties,

$$r_S = 1 - \frac{6\sum_{i=1}^n d_i^2}{n(n^2 - 1)}.$$

Proof. Without ties each rank vector is an arrangement of $1, 2, \ldots, n$. Both have mean $m = (n+1)/2$, and both have the same sum of squared deviations,

$$\sum_{k=1}^n (k - m)^2 = \sum_{k=1}^n k^2 - nm^2 = \frac{n(n+1)(2n+1)}{6} - \frac{n(n+1)^2}{4} = \frac{n(n+1)\big[(4n + 2) - (3n + 3)\big]}{12} = \frac{n(n^2 - 1)}{12}.$$

Call this $T$. Expanding $d_i = (R_i - m) - (S_i - m)$,

$$\sum_i d_i^2 = T + T - 2\sum_i (R_i - m)(S_i - m),$$

so the numerator of Pearson's formula is $\sum_i (R_i - m)(S_i - m) = T - \tfrac12\sum_i d_i^2$, and the denominator is $\sqrt{T \cdot T} = T$. Dividing, $r_S = 1 - \sum_i d_i^2/(2T) = 1 - 6\sum_i d_i^2/[n(n^2 - 1)]$. $\square$

With ties the proof breaks at the first step. Midranks keep the mean at $(n+1)/2$ but shrink the sum of squared deviations below $T$, so the formula no longer equals Pearson's coefficient of the ranks. For the twelve adults, the ages are all different and the two pressures of 119.33 share the midrank 5.5. The rank differences give $\sum d_i^2 = 113.5$, and the formula gives $1 - 6 \times 113.5/(12 \times 143) = 0.6031$, while Pearson's coefficient of the midranks is 0.6025. One tie moves the fourth decimal; many ties, as in data recorded to whole units, move more. This article takes $r_S$ to be Pearson's coefficient of the midranks, which keeps Theorem 1 and its bounds.

Example (a monotone relation that is not a line). For $x = 1, 2, 3, 4, 5$ and $y = x^3$, the ranks agree and $r_S = 1$, while $r$ is below 1 because the points are not on a line. Spearman's coefficient is unchanged by any strictly increasing transformation of either variable, such as a logarithm, because such a transformation does not change the ranks. That is why the logarithm of body mass index changed Pearson's coefficient above and leaves Spearman's at 0.187.

Kendall's coefficient: pairs that agree

Spearman's coefficient still averages products of rank deviations, so an adult ranked far apart on the two variables counts heavily. Kendall's coefficient counts only orderings. Take two adults $i$ and $j$. The pair is concordant if the older of the two also has the higher pressure, $(x_j - x_i)(y_j - y_i) > 0$, and discordant if the older has the lower pressure, $(x_j - x_i)(y_j - y_i) < 0$. If either difference is zero the pair is tied. With $C$ concordant and $D$ discordant pairs out of $\binom{n}{2}$, Kendall's coefficient (Kendall, 1938) is

$$\hat\tau_a = \frac{C - D}{\binom{n}{2}}.$$

It is 1 when every pair is concordant and $-1$ when every pair is discordant. Without ties, $D$ is the number of inversions in the sequence of $y$-ranks listed in order of $x$: the number of swaps of neighbors that sorting the sequence needs. Relevance Judgments, Pooling, and Agreement uses $\hat\tau$ in exactly that way, to compare two orderings of retrieval systems.

For the twelve adults there are $\binom{12}{2} = 66$ pairs: $C = 47$ concordant, $D = 18$ discordant, and 1 tied (adults 3 and 9). So $\hat\tau_a = (47 - 18)/66 = 0.439$.

Ties, and Kendall's coefficient as a cosine. A tied pair counts in the denominator of $\hat\tau_a$ but can be neither concordant nor discordant, so with ties $\hat\tau_a$ cannot reach $\pm1$. The usual correction has a geometric reading that proves its bounds. List the $N = \binom{n}{2}$ pairs in a fixed order and form the sign vectors $\mathbf{u}$, with entries $\operatorname{sign}(x_j - x_i)$, and $\mathbf{v}$, with entries $\operatorname{sign}(y_j - y_i)$; each entry is $1$, $-1$ or $0$. Their dot product is $C - D$, since a concordant pair contributes $+1$, a discordant pair $-1$ and a tied pair 0. The squared length of $\mathbf{u}$ is $N - t_x$, where $t_x$ is the number of pairs tied in $x$, and likewise $\lVert\mathbf{v}\rVert^2 = N - t_y$. Their cosine is

$$\hat\tau_b = \frac{C - D}{\sqrt{(N - t_x)(N - t_y)}},$$

the coefficient Kendall (1945) studied for tied rankings, and by Lemma 1 it lies between $-1$ and 1. For the twelve adults $t_x = 0$ and $t_y = 1$, so $\hat\tau_b = 29/\sqrt{66 \times 65} = 0.443$. The pair vectors are not centered, so this is a cosine, not a Pearson coefficient; Lemma 1 needs no centering.

Why is Kendall's value, 0.439, so much smaller than the other two, 0.584 and 0.602? Not because it sees a weaker association: the three coefficients are on different scales. The next section finds the population quantities they estimate and shows that, even for a bivariate normal population, $\lvert\tau\rvert$ is smaller than $\lvert\rho\rvert$ except at 0 and $\pm1$.

What the coefficients estimate

Each sample coefficient estimates a property of the population the pairs come from. Let $(X, Y)$ have a joint distribution with continuous distribution functions $F$ for $X$ and $G$ for $Y$, so that ties occur with probability 0.

  • Pearson. $r$ estimates $\rho = \operatorname{Cov}(X, Y)/(\sigma_X\sigma_Y)$, the correlation of Expectation, Variance and Covariance.
  • Kendall. Let $(X_1, Y_1)$ and $(X_2, Y_2)$ be two independent draws. Then $\tau = P(\text{concordant}) - P(\text{discordant}) = E\big[\operatorname{sign}(X_2 - X_1)\operatorname{sign}(Y_2 - Y_1)\big]$. Each pair $i < j$ of a sample of independent draws contributes $\operatorname{sign}(x_j - x_i)\operatorname{sign}(y_j - y_i)$, whose expectation is $\tau$, and $\hat\tau_a$ is the average of the $\binom{n}{2}$ contributions. So $E[\hat\tau_a] = \tau$ exactly, by linearity of expectation.
  • Spearman. The ranks of a sample estimate the positions $F(X)$ and $G(Y)$, and $\rho_S = \operatorname{Corr}\big(F(X), G(Y)\big)$, the correlation of the two grades. Each grade is uniform on $(0, 1)$ by the probability integral transform proved in Random Variables and Distribution Functions, so it has mean $\tfrac12$ and variance $\tfrac1{12}$, and $\rho_S = 12\,E[F(X)G(Y)] - 3$.

Kruskal (1958) surveys these population measures, the probabilistic reading of each and the relations among them. For the 5,176 adults, $\rho = 0.486$ and $\rho_S = 0.509$. Kendall's value needs care, because the population has ties (ages are recorded in whole years). Over all pairs of the population, $(C - D)/N = 0.348$, the quantity defined above, and the tie-corrected $\tau_b = 0.353$; the two definitions differ when there are ties, unless both are 0. The comparisons below use $\tau_b = 0.353$. For body mass index and pressure the three are 0.153, 0.187 and 0.126.

The bivariate normal: exact formulas

For one family of distributions the three population values are functions of one another. The family is the bivariate normal: $X = \mu_X + \sigma_X Z_1$ and $Y = \mu_Y + \sigma_Y\big(\rho Z_1 + \sqrt{1 - \rho^2}\,Z_2\big)$, with $Z_1$ and $Z_2$ independent standard normal variables and $-1 < \rho < 1$. Its correlation is $\rho$, by part 4 of Theorem 6 in Expectation, Variance and Covariance. The key is an orthant probability. Fisher (1915, §1, p. 508) cites it as "Sheppard's theorem", by which the distribution of $r$ in samples of two had been determined.

Lemma 2 (orthant probability). Let $\mathbf{W} = (W_1, \ldots, W_k)$ be independent standard normal variables, and let $\mathbf{a}$ and $\mathbf{b}$ be unit vectors in $\mathbb{R}^k$ with $\mathbf{a}\cdot\mathbf{b} = \rho$, where $-1 < \rho < 1$. Then $U = \mathbf{a}\cdot\mathbf{W}$ and $V = \mathbf{b}\cdot\mathbf{W}$ are standard normal with correlation $\rho$, and

$$P(U > 0,\ V > 0) = \frac14 + \frac{\arcsin\rho}{2\pi}.$$

The idea: write $U$ and $V$ as projections of one rotationally symmetric random point in the plane on two directions, and measure the angle of the region where both projections are positive.

Proof. First reduce to the plane. Let $\mathbf{q}_1 = \mathbf{a}$ and $\mathbf{q}_2 = (\mathbf{b} - \rho\mathbf{a})/\sqrt{1 - \rho^2}$, two orthonormal vectors, and complete them to an orthonormal basis of $\mathbb{R}^k$ by the Gram–Schmidt process (The Geometry of Least Squares, Lemma 1). With these as the rows of an orthogonal matrix $Q$, the coordinates of $\mathbf{Z} = Q\mathbf{W}$ are independent standard normal variables, by the rotation theorem (Theorem 1 of Sampling Distributions from the Normal). Since $\mathbf{b} = \rho\mathbf{q}_1 + \sqrt{1 - \rho^2}\,\mathbf{q}_2$, we get $U = Z_1$ and $V = \rho Z_1 + \sqrt{1 - \rho^2}\,Z_2$. So $U$ and $V$ are standard normal with covariance $\rho$, and only $(Z_1, Z_2)$ matters. Let $\theta \in (0, \pi)$ be the angle with $\cos\theta = \rho$, and let $\mathbf{p} = (1, 0)$ and $\mathbf{q} = (\cos\theta, \sin\theta)$, two unit vectors at angle $\theta$ in the plane. Then $U = \mathbf{p}\cdot(Z_1, Z_2)$ and $V = \mathbf{q}\cdot(Z_1, Z_2)$, because $\sin\theta = \sqrt{1 - \rho^2}$. Write $\mathbf{Z}$ for $(Z_1, Z_2)$ from here on.

First, every sector of the plane with vertex at the origin and angle $\varphi$ has probability $\varphi/(2\pi)$. By the rotation theorem again, a rotation of the plane leaves the distribution of $\mathbf{Z}$ unchanged, so all sectors of the same angle have the same probability. The $m$ sectors of angle $2\pi/m$ that start at a fixed ray cover the plane and overlap only on rays, which are lines through the origin and have probability 0 under a density. So each has probability $1/m$, and a sector of angle $2\pi j/m$ has probability $j/m$. A sector of any angle $\varphi$ lies between sectors of angles $2\pi j/m$ and $2\pi(j+1)/m$ with $j/m \le \varphi/(2\pi) < (j+1)/m$; letting $m$ grow squeezes its probability to $\varphi/(2\pi)$.

The event $\{\mathbf{p}\cdot\mathbf{Z} > 0\}$ is the half-plane of directions within $\pi/2$ of $\mathbf{p}$, and $\{\mathbf{q}\cdot\mathbf{Z} > 0\}$ the half-plane within $\pi/2$ of $\mathbf{q}$. Their intersection is a sector of angle $\pi - \theta$. So $P(U > 0, V > 0) = (\pi - \theta)/(2\pi)$. Since $\theta = \arccos\rho = \pi/2 - \arcsin\rho$, this is $\tfrac14 + \arcsin\rho/(2\pi)$. $\square$

For $\rho = 0$ the lemma gives $\tfrac14$, the product of two probabilities of $\tfrac12$, as independence requires.

Theorem 3 (rank correlations of the bivariate normal). For a bivariate normal population with correlation $\rho$,

$$\tau = \frac{2}{\pi}\arcsin\rho, \qquad \rho_S = \frac{6}{\pi}\arcsin\frac{\rho}{2}.$$

Proof. Both coefficients are unchanged by increasing transformations of $X$ and $Y$, so take $\mu = 0$ and $\sigma = 1$.

Write $s = \sqrt{1 - \rho^2}$.

Kendall. Let $(X_1, Y_1)$ and $(X_2, Y_2)$ be independent copies, built from independent standard normal variables $Z_1, Z_2$ and $Z_1', Z_2'$ respectively, and let $\mathbf{W} = (Z_1, Z_2, Z_1', Z_2')$. Then $U = (X_2 - X_1)/\sqrt2 = \mathbf{a}\cdot\mathbf{W}$ with $\mathbf{a} = (-1, 0, 1, 0)/\sqrt2$, and $V = (Y_2 - Y_1)/\sqrt2 = \mathbf{b}\cdot\mathbf{W}$ with $\mathbf{b} = (-\rho, -s, \rho, s)/\sqrt2$. Both are unit vectors and $\mathbf{a}\cdot\mathbf{b} = (\rho + \rho)/2 = \rho$, so Lemma 2 applies. The pair of draws is concordant when $U$ and $V$ have the same sign. By Lemma 2 and the symmetry $(U, V) \mapsto (-U, -V)$, which preserves the distribution, $P(\text{concordant}) = 2\big(\tfrac14 + \arcsin\rho/(2\pi)\big) = \tfrac12 + \arcsin\rho/\pi$. Ties have probability 0, so $\tau = 2P(\text{concordant}) - 1 = (2/\pi)\arcsin\rho$.

Spearman. Here $F = G = \Phi$, the standard normal distribution function. Let $X'$ and $Y''$ be standard normal, independent of each other and of $(X, Y)$. Then $\Phi(x) = P(X' < x)$, so by conditioning on $(X, Y)$, $E[\Phi(X)\Phi(Y)] = P(X' < X,\ Y'' < Y)$. With $\mathbf{W} = (Z_1, Z_2, X', Y'')$, the variables $U = (X - X')/\sqrt2$ and $V = (Y - Y'')/\sqrt2$ are $\mathbf{a}\cdot\mathbf{W}$ and $\mathbf{b}\cdot\mathbf{W}$ with unit vectors $\mathbf{a} = (1, 0, -1, 0)/\sqrt2$ and $\mathbf{b} = (\rho, s, 0, -1)/\sqrt2$, and $\mathbf{a}\cdot\mathbf{b} = \rho/2$. The event $\{X' < X,\ Y'' < Y\}$ is $\{U > 0,\ V > 0\}$. By Lemma 2, $E[\Phi(X)\Phi(Y)] = \tfrac14 + \arcsin(\rho/2)/(2\pi)$, and $\rho_S = 12\,E[\Phi(X)\Phi(Y)] - 3 = (6/\pi)\arcsin(\rho/2)$. $\square$

For $\rho = 0.486$ the formulas give $\tau = 0.323$ and $\rho_S = 0.469$. The survey's adults have $\tau = 0.353$ and $\rho_S = 0.509$. The gap is evidence that age and pressure are not bivariate normal: age, for one, is spread nearly evenly over 18 to 80, with a fourth standardized moment of 1.85 against the normal's 3. Theorem 3 also explains the scales. The function $g(\rho) = (2/\pi)\arcsin\rho - \rho$ is 0 at $\rho = 0$ and at $\rho = 1$, and it is convex on $[0, 1)$, since $g''(\rho) = (2/\pi)\rho(1 - \rho^2)^{-3/2} \ge 0$. A convex function lies below its chord, so $g < 0$ strictly between, and Kendall's $\tau$ runs below $\rho$ for every normal population with $0 < \rho < 1$. Both $\tau$ and $\rho$ are odd functions of $\rho$ here, so for $-1 < \rho < 0$ the inequality reverses and $\lvert\tau\rvert < \lvert\rho\rvert$ again. Comparing a $\hat\tau$ of 0.44 with an $r$ of 0.58 compares two scales, not two strengths.

Is the correlation zero?

Twelve adults gave $r = 0.584$. Could a population with no association produce a value that large by chance? Two answers are exact: one assumes normal errors, the other assumes nothing but independence.

The exact t test

Theorem 4 (the null distribution of r). Let $x_1, \ldots, x_n$ be fixed numbers, not all equal, with $n \ge 3$, and let $Y_1, \ldots, Y_n$ be independent normal variables with a common mean $\alpha$ and variance $\sigma^2$. Then

$$T = \frac{r\sqrt{n - 2}}{\sqrt{1 - r^2}}$$

has the $t$ distribution with $n - 2$ degrees of freedom, and $r$ has density

$$f(r) = \frac{\Gamma\big(\tfrac{n-1}{2}\big)}{\sqrt{\pi}\,\Gamma\big(\tfrac{n-2}{2}\big)}\,(1 - r^2)^{(n-4)/2}, \qquad -1 < r < 1.$$

The same holds when the $x_i$ are random and independent of the $Y_i$, whatever their distribution, provided that, with probability 1, they are not all equal.

The idea: split the data vector into the part along the centered $x$ direction and the part perpendicular to both $x$ and the constant. The projection theorem makes the two independent, one normal and one chi-squared, and $r/\sqrt{1 - r^2}$ is their ratio.

Proof. Let $\mathbf{u} = (\mathbf{x} - \bar{x}\mathbf{1})/\lVert\mathbf{x} - \bar{x}\mathbf{1}\rVert$, a unit vector perpendicular to the vector of ones $\mathbf{1}$, because the deviations sum to zero. Let $A = \mathbf{u}\mathbf{u}^{\mathsf T}$ and $B = I - \tfrac1n J - \mathbf{u}\mathbf{u}^{\mathsf T}$, where $J$ is the matrix of ones, so that $\tfrac1nJ$ replaces every coordinate by the mean. Both are symmetric and idempotent, $AB = 0$ (because $\mathbf{u}^{\mathsf T}\mathbf{1} = 0$ and $\mathbf{u}^{\mathsf T}\mathbf{u} = 1$), and their ranks are 1 and $n - 2$, their traces. The mean vector $\alpha\mathbf{1}$ satisfies $A\alpha\mathbf{1} = \mathbf{0}$ and $B\alpha\mathbf{1} = \mathbf{0}$. By the projection theorem (Theorem 2 of Sampling Distributions from the Normal), $A\mathbf{Y}$ and $B\mathbf{Y}$ are independent, and $\lVert B\mathbf{Y}\rVert^2/\sigma^2$ is chi-squared with $n - 2$ degrees of freedom. The single coordinate $\mathbf{u}^{\mathsf T}\mathbf{Y}$ is normal with mean $\alpha\,\mathbf{u}^{\mathsf T}\mathbf{1} = 0$ and variance $\sigma^2\lVert\mathbf{u}\rVert^2 = \sigma^2$ (it is the first coordinate after a rotation whose first row is $\mathbf{u}^{\mathsf T}$, Theorem 1 there), and it is a function of $A\mathbf{Y} = (\mathbf{u}^{\mathsf T}\mathbf{Y})\mathbf{u}$.

Now express $r$. Because $\mathbf{u} \perp \mathbf{1}$, the numerator of $r$ divided by $\lVert\mathbf{x} - \bar{x}\mathbf{1}\rVert$ is $\mathbf{u}\cdot(\mathbf{Y} - \bar{Y}\mathbf{1}) = \mathbf{u}^{\mathsf T}\mathbf{Y}$. The centered vector splits as $\mathbf{Y} - \bar{Y}\mathbf{1} = A\mathbf{Y} + B\mathbf{Y}$ into perpendicular parts, so $\lVert\mathbf{Y} - \bar{Y}\mathbf{1}\rVert^2 = (\mathbf{u}^{\mathsf T}\mathbf{Y})^2 + \lVert B\mathbf{Y}\rVert^2$. Hence

$$r = \frac{\mathbf{u}^{\mathsf T}\mathbf{Y}}{\sqrt{(\mathbf{u}^{\mathsf T}\mathbf{Y})^2 + \lVert B\mathbf{Y}\rVert^2}}, \qquad \frac{r}{\sqrt{1 - r^2}} = \frac{\mathbf{u}^{\mathsf T}\mathbf{Y}}{\lVert B\mathbf{Y}\rVert},$$

and $T = \dfrac{\mathbf{u}^{\mathsf T}\mathbf{Y}/\sigma}{\sqrt{\lVert B\mathbf{Y}\rVert^2/[\sigma^2(n-2)]}}$ is a standard normal variable over the square root of an independent chi-squared variable divided by its degrees of freedom. That is the definition of the $t$ distribution with $n - 2$ degrees of freedom (The Chi-Squared and t Distributions).

For the density, $T = g(r) = \sqrt{\nu}\,r/\sqrt{1 - r^2}$ with $\nu = n - 2$ is increasing on $(-1, 1)$, with $g'(r) = \sqrt{\nu}\,(1 - r^2)^{-3/2}$ and $1 + T^2/\nu = 1/(1 - r^2)$. The $t$ density is $c_\nu(1 + t^2/\nu)^{-(\nu+1)/2}$ with $c_\nu = \Gamma\big(\tfrac{\nu+1}{2}\big)/\big(\sqrt{\nu\pi}\,\Gamma\big(\tfrac{\nu}{2}\big)\big)$. By the change of variables for a monotone function, the density of $r$ is $c_\nu(1 - r^2)^{(\nu+1)/2}\sqrt{\nu}\,(1 - r^2)^{-3/2} = c_\nu\sqrt{\nu}\,(1 - r^2)^{(n-4)/2}$, and $c_\nu\sqrt\nu$ is the stated constant.

Finally, if the $x_i$ are random, independent of the $Y_i$, and with probability 1 not all equal, the conditional distribution of $T$ given the $x_i$ is the $t$ distribution whatever their values, so the unconditional distribution is the same. $\square$

Only $Y$ needs to be normal; the $x_i$ may have any distribution, provided they are independent of the $Y_i$. The exponent $(n-4)/2$ is the one in Fisher's (1915, §3, p. 510) derivation of the same density. The statistic $T$ is also the $t$ statistic for the slope of the least squares line, which The Normal Linear Model derives as a case of the general $t$ test.

For the twelve adults, $T = 0.584\sqrt{10}/\sqrt{1 - 0.584^2}$, which is $t = 2.27$ on 10 degrees of freedom, and the two-sided p-value is 0.046.

Permutation: no normality needed

Under independence, with continuous distributions, the $y$-values carry no information about which $x$-values they are paired with: given the two sets of values, every pairing is equally likely. That gives each coefficient an exact null distribution, its permutation distribution, from which a test needs no model at all. Its mean and variance can be found in general.

Theorem 5 (the permutation variance). Let $\mathbf{a}$ and $\mathbf{b}$ be fixed vectors with $\sum a_i = \sum b_i = 0$ and $\lVert\mathbf{a}\rVert = \lVert\mathbf{b}\rVert = 1$, and let $\pi$ be a uniformly random permutation of $1, \ldots, n$, with $n \ge 2$. Then $W = \sum_i a_ib_{\pi(i)}$ has mean 0 and variance $1/(n - 1)$.

Proof. Each $b_{\pi(i)}$ is equally likely to be any $b_k$, so $E[b_{\pi(i)}] = \tfrac1n\sum_k b_k = 0$ and $E[b_{\pi(i)}^2] = \tfrac1n$. For $i \ne j$, the pair $(\pi(i), \pi(j))$ is equally likely to be any of the $n(n - 1)$ ordered pairs of different indices, so

$$E[b_{\pi(i)}b_{\pi(j)}] = \frac{1}{n(n-1)}\sum_{k \ne l} b_kb_l = \frac{(\sum_k b_k)^2 - \sum_k b_k^2}{n(n-1)} = -\frac{1}{n(n-1)}.$$

So $E[W] = 0$ and

$$E[W^2] = \sum_i a_i^2\cdot\frac1n + \sum_{i \ne j} a_ia_j\cdot\Big(-\frac{1}{n(n-1)}\Big) = \frac1n - \frac{(\sum_i a_i)^2 - \sum_i a_i^2}{n(n-1)} = \frac1n + \frac{1}{n(n-1)} = \frac{1}{n-1}. \qquad \square$$

Pearson's $r$ and Spearman's $r_S$ are both of the form $W$, with $\mathbf{a}$ and $\mathbf{b}$ the normalized centered vectors of values or of ranks. So under random pairing both have mean 0 and variance exactly $1/(n - 1)$, whatever the data. For the twelve adults, $r_S\sqrt{n - 1} = 2.00$ standard deviations from 0. Referring it to the normal distribution is an approximation that this article does not justify for $n = 12$; the permutation distribution itself can be computed exactly by listing permutations, or estimated by sampling them, as the course's nonparametric methods do for rank tests.

Kendall's null distribution

Kendall's statistic $S = C - D$ has a simple permutation variance as well. Order the pairs by $x$, so that, with no ties, $S = \sum_{i < j}\operatorname{sign}(\pi_j - \pi_i)$, where $\pi_1, \ldots, \pi_n$ are the $y$-ranks in order of $x$, a uniformly random permutation under independence.

Theorem 6 (Kendall's null variance). Under random pairing with no ties, $E[S] = 0$ and

$$\operatorname{Var}(S) = \frac{n(n-1)(2n+5)}{18}, \qquad \operatorname{Var}(\hat\tau_a) = \frac{2(2n+5)}{9n(n-1)}.$$

Proof. Write $a_{ij} = \operatorname{sign}(\pi_j - \pi_i)$ for $i < j$. By symmetry each is $\pm1$ with probability $\tfrac12$, so $E[S] = 0$ and each term has variance 1. Two terms with no index in common involve disjoint positions, and their product has mean 0 by the symmetry that reverses the order of the values in one pair's positions; so they are uncorrelated. The remaining covariances come from triples $i < j < k$. Among $\pi_i, \pi_j, \pi_k$ each of the six orderings has probability $\tfrac16$, and the three pairs of terms sharing an index are:

  • $a_{ij}$ and $a_{ik}$ (sharing the smallest index): equal exactly when $\pi_i$ is the smallest or the largest of the three, probability $\tfrac23$, so $E[a_{ij}a_{ik}] = \tfrac23 - \tfrac13 = \tfrac13$;
  • $a_{ik}$ and $a_{jk}$ (sharing the largest): equal when $\pi_k$ is the smallest or largest, so again $\tfrac13$;
  • $a_{ij}$ and $a_{jk}$ (sharing the middle index): equal exactly when $\pi_j$ is the middle value, probability $\tfrac13$, so $E[a_{ij}a_{jk}] = \tfrac13 - \tfrac23 = -\tfrac13$.

Each triple contributes $2(\tfrac13 + \tfrac13 - \tfrac13) = \tfrac23$ to the sum of covariances over ordered pairs of terms. So

$$\operatorname{Var}(S) = \binom{n}{2} + \frac23\binom{n}{3} = \frac{n(n-1)}{2} + \frac{n(n-1)(n-2)}{9} = \frac{n(n-1)\big[9 + 2(n-2)\big]}{18} = \frac{n(n-1)(2n+5)}{18}.$$

Dividing by $\binom{n}{2}^2 = n^2(n-1)^2/4$ gives the variance of $\hat\tau_a$. $\square$

The script checks the formula by listing every permutation for $n = 4$, 5 and 6. For $n = 12$ the null standard deviation of $\hat\tau_a$ is $\sqrt{2 \times 29/(9 \times 12 \times 11)} = 0.221$, and the twelve adults' $\hat\tau_a = 0.439$ lies 1.99 standard deviations from 0 (ignoring the single tie). The three statistics agree here: Pearson's gives $p = 0.046$, and the two rank statistics lie about two null standard deviations from 0.

Fisher's z: an interval for rho

A test of $\rho = 0$ answers only one question. An interval for $\rho$ needs the distribution of $r$ when $\rho \ne 0$, and that distribution is awkward. Fisher (1921, Introduction) recalled that his formulas of 1915 showed that "in the neighbourhood of ±1, the curves become extremely skew, even for large samples", so that the ordinary statement of its probable error "is practically valueless" there, and that a transformation gives a variable whose sampling curves are "practically normal and of constant standard deviation". The transformation is $z = \operatorname{artanh} r = \tfrac12\ln\frac{1+r}{1-r}$. Here we derive it, by the delta method and a variance-stabilizing transformation, as The Delta Method does for a proportion.

The limiting variance of r

Theorem 7 (asymptotic normality of r). Let $(X_1, Y_1), (X_2, Y_2), \ldots$ be independent draws of a pair $(X, Y)$ with finite fourth moments, positive variances and correlation $\rho$, and let $X^* = (X - \mu_X)/\sigma_X$ and $Y^* = (Y - \mu_Y)/\sigma_Y$. Then $\sqrt{n}\,(r_n - \rho)$ converges in distribution to the normal distribution with mean 0 and variance

$$\sigma_r^2 = \operatorname{Var}\Big(X^*Y^* - \frac{\rho}{2}\big(X^{*2} + Y^{*2}\big)\Big).$$

For a bivariate normal pair, $\sigma_r^2 = (1 - \rho^2)^2$.

The idea: $r$ is a smooth function of five sample means, so the delta method gives its limiting variance from the gradient of that function.

Proof. Since $r$ is unchanged by the standardization (Theorem 1 and the remark after it), compute it from $X_i^*$ and $Y_i^*$. With bars denoting sample means,

$$r_n = g\big(\overline{X^{*2}},\ \overline{Y^{*2}},\ \overline{X^*Y^*},\ \bar{X}^*,\ \bar{Y}^*\big), \qquad g(a, b, c, d, e) = \frac{c - de}{\sqrt{(a - d^2)(b - e^2)}},$$

because $\tfrac1n\sum(X_i^* - \bar{X}^*)^2 = \overline{X^{*2}} - (\bar{X}^*)^2$ and likewise for the other sums. The vector of the five means is the mean of i.i.d. vectors $(X^{*2}, Y^{*2}, X^*Y^*, X^*, Y^*)$, whose components have finite variances because the fourth moments are finite. So $\sqrt{n}$ times its deviation from the expectation $\boldsymbol\theta = (1, 1, \rho, 0, 0)$ satisfies the hypothesis of the delta method for several variables (Theorem 3 of The Delta Method), by Corollary 3 of Convergence in Distribution and the Proof of the Central Limit Theorem. The function $g$ has continuous partial derivatives near $\boldsymbol\theta$, where its denominator is 1, and a function with continuous partial derivatives near a point is differentiable there (a theorem of multivariable calculus, cited). At $\boldsymbol\theta$ the partial derivatives are

$$\frac{\partial g}{\partial a} = -\frac{\rho}{2}, \quad \frac{\partial g}{\partial b} = -\frac{\rho}{2}, \quad \frac{\partial g}{\partial c} = 1, \quad \frac{\partial g}{\partial d} = \frac{\partial g}{\partial e} = 0,$$

the last two because every term of their derivatives carries a factor $d$ or $e$. The limiting variance $\nabla g^{\mathsf T}\Sigma\,\nabla g$ is the variance of the linear combination $\nabla g\cdot(X^{*2}, Y^{*2}, X^*Y^*, X^*, Y^*)$, which is the stated $\sigma_r^2$.

For the bivariate normal write $X^* = Z_1$ and $Y^* = \rho Z_1 + \sqrt{1 - \rho^2}\,Z_2$. The standard normal moments $E[Z^2] = 1$, $E[Z^4] = 3$ and the vanishing odd moments give $E[X^{*4}] = E[Y^{*4}] = 3$, $E[X^{*3}Y^*] = E[X^*Y^{*3}] = 3\rho$ and $E[X^{*2}Y^{*2}] = 3\rho^2 + (1 - \rho^2) = 1 + 2\rho^2$. Hence $\operatorname{Var}(X^*Y^*) = 1 + 2\rho^2 - \rho^2 = 1 + \rho^2$, $\operatorname{Var}(X^{*2}) = \operatorname{Var}(Y^{*2}) = 2$, $\operatorname{Cov}(X^{*2}, Y^{*2}) = 2\rho^2$ and $\operatorname{Cov}(X^*Y^*, X^{*2}) = \operatorname{Cov}(X^*Y^*, Y^{*2}) = 3\rho - \rho = 2\rho$. So

$$\sigma_r^2 = (1 + \rho^2) + \frac{\rho^2}{4}\big(2 + 2 + 4\rho^2\big) - \rho\,(2\rho + 2\rho) = 1 - 2\rho^2 + \rho^4 = (1 - \rho^2)^2. \qquad \square$$

The transformation

Under normality the standard deviation of $r$ is about $(1 - \rho^2)/\sqrt{n}$: it depends on the unknown $\rho$ and shrinks to nothing near $\pm1$. A variance-stabilizing transformation $h$ needs $h'(\rho)(1 - \rho^2) = 1$, by the recipe of The Delta Method. Integrating $1/(1 - \rho^2) = \tfrac12\big[1/(1 + \rho) + 1/(1 - \rho)\big]$ gives

$$h(\rho) = \tfrac12\ln\frac{1 + \rho}{1 - \rho} = \operatorname{artanh}\rho,$$

and by Theorem 7 and the one-variable delta method, $\sqrt{n}\,(\operatorname{artanh} r_n - \operatorname{artanh}\rho)$ tends to the standard normal distribution for every $\rho$ strictly between $-1$ and 1, under normality. The usual finite-sample form replaces $1/n$ by $1/(n - 3)$, and the 95% Fisher interval is

$$\tanh\Big(\operatorname{artanh} r \pm \frac{1.96}{\sqrt{n - 3}}\Big).$$

The $1/(n - 3)$ is not part of Theorem 7. It matches, to two terms, Fisher's (1921, §3) expansion of the exact distribution: he gives the second moment of $z$ about $\zeta = \operatorname{artanh}\rho$ as the series

$$\frac{1}{n - 1}\Big\{1 + \frac{8 - \rho^2}{4(n - 1)} + \frac{88 - 9\rho^2 - 9\rho^4}{24(n - 1)^2} + \cdots\Big\},$$

whose first two terms at $\rho = 0$ agree with those of $1/(n - 3) = \frac{1}{n-1}\big(1 + \frac{2}{n-1} + \cdots\big)$, and he gives the mean of $z$ as exceeding $\zeta$ by about $\rho/[2(n - 1)]$.

The check. Fisher (1915) derived the exact density of $r$ for a bivariate normal sample, and Fisher (1921, §3) prints it as

$$f(r) = \frac{n - 2}{\pi}(1 - \rho^2)^{(n-1)/2}(1 - r^2)^{(n-4)/2}\int_0^\infty\frac{dt}{(\cosh t - \rho r)^{n-1}}.$$

At $\rho = 0$ it reduces to the density of Theorem 4, which the script confirms numerically. Integrating it numerically gives the exact moments of $z$ and the exact coverage of the Fisher interval for $n = 12$:

$\rho$ $\operatorname{Var}(z)$ $E[z] - \zeta$ Coverage of the 95% interval
0 0.1107 0.0000 0.9490
0.5 0.1094 0.0257 0.9496
0.9 0.1063 0.0470 0.9511

Against $1/(n - 3) = 1/9 = 0.1111$, the variance of $z$ barely moves as $\rho$ runs from 0 to 0.9, which is the point of the transformation. At $\rho = 0.5$ Fisher's three-term series gives 0.1096 for the second moment about $\zeta$, against the exact 0.1101. On the original scale the distribution is far from symmetric: at $\rho = 0.9$ the mean of $r$ is 0.891, and its distribution has a long tail toward 0. The coverage is within 0.0011 of 95% in every row, at a sample size of only twelve.

For the twelve adults, $z = \operatorname{artanh}0.584 = 0.668$, and $0.668 \pm 1.96/3$ transforms back to the interval $[0.015, 0.867]$ for $\rho$. It contains the population value 0.486, and its width says how little twelve pairs reveal about a correlation.

When the population is not normal

Theorem 7 holds for any population with four moments; only the value $(1 - \rho^2)^2$ needs normality. If the true limiting variance is $\sigma_r^2$, the limiting variance of $\sqrt{n}\,(z - \zeta)$ is $\sigma_r^2/(1 - \rho^2)^2$ by the delta method, and the Fisher interval, built for the value 1, covers with probability tending to $2\Phi\big(1.96/\sqrt{\kappa_r}\big) - 1$, where $\kappa_r = \sigma_r^2/(1 - \rho^2)^2$ is their ratio. For independent $X$ and $Y$, $\rho = 0$ and $\sigma_r^2 = E[X^{*2}]E[Y^{*2}] = 1$ whatever the margins, so the interval is right in the limit, in agreement with Theorem 5. Away from $\rho = 0$ it depends on the joint fourth moments.

For age and pressure in the 5,176 adults, the limiting variance of $r$ computed from the population is 0.506, against $(1 - \rho^2)^2 = 0.583$ for a normal population with the same $\rho$: a ratio of 0.866. Theorem 7's formula combines several fourth moments, so no single one decides the direction. For these adults the mixed moment $E[X^{*2}Y^{*2}]$ is 1.14, against $1 + 2\rho^2 = 1.47$ for the normal; the fourth standardized moment of age is 1.85, below the normal's 3, and that of pressure is 4.55, above it. The limit of the coverage is $2\Phi(1.96/\sqrt{0.866}) - 1 = 0.965$: the interval is too wide here, not too narrow. For body mass index and pressure the ratio is 1.016 and the limit 0.948.

A simulation checks the limit at finite sizes. Each run draws samples with replacement from the 5,176 adults (so the pairs are independent draws from a population with $\rho = 0.486$), computes the Fisher interval, and records whether it contains 0.486. Thirty runs of 10,000 samples each, with seeds $21{,}100{,}000 + 1000k + s$ for $s = 0, \ldots, 29$ and size index $k$:

$n$ Coverage of the 95% interval, median over 30 runs Interquartile range
12 0.960 0.958 to 0.961
50 0.963 0.962 to 0.964
200 0.965 0.964 to 0.966

In this experiment the coverage moves from 0.960 toward the limit 0.965 as $n$ grows, and the gap from 95% does not close with more data: it is a property of the population, not of the sample size. Whether $\kappa_r$ is above or below 1 for another population has to be computed from Theorem 7's formula, as it was here; for independent variables it is exactly 1, however heavy their tails. The bootstrap, treated in Resampling Methods, estimates the variance of $r$ from the sample itself and does not need the normal value.

What is proved here, what is cited, what the experiment shows

Proved here. Lagrange's identity and the bounds on $r$, $r_S$ and $\hat\tau_b$ with the cases of equality for $r$ (Theorem 1); Spearman's formula without ties (Theorem 2); the unbiasedness of $\hat\tau_a$ for $\tau$; the orthant probability (Lemma 2) and the arcsine formulas for $\tau$ and $\rho_S$ under the bivariate normal (Theorem 3); the exact $t$ distribution and density of $r$ under independence with normal $Y$ (Theorem 4); the permutation mean and variance of $r$ and $r_S$ (Theorem 5); Kendall's null variance (Theorem 6); the limiting normal distribution of $r$ with its variance in general and under normality (Theorem 7), and the arctanh transformation as the variance-stabilizing function.

Cited. The projection and rotation theorems of Sampling Distributions from the Normal; the probability integral transform; the multivariate delta method and the central limit theorem for vector means, from the unit on probability theory; that a function with continuous partial derivatives is differentiable; Fisher's (1915) exact density of $r$ under normality, which the script integrates; Fisher's (1921) series for the moments of $z$; the asymptotic normality of the rank statistics, used only as an approximation for the twelve adults.

Computed. The coefficients, tests and intervals for the twelve adults and the population, the exact moments and coverages of the $z$ interval from Fisher's density, the population's limiting variance of $r$, and the enumerations of the permutation distributions for small $n$ are computed by the script. The coverage figures from the simulation are observations of that seeded experiment for one population.

Worked problems

The problems use the unit's twelve adults and small exact cases, chosen to exercise the three definitions, Kendall's null distribution, the $z$ interval and Theorem 5. Every number is recomputed by the article's script.

Problem 1 (the three coefficients by hand). For the first six of the twelve adults (ages 54, 39, 43, 61, 63, 69; pressures 129.33, 116.00, 119.33, 148.00, 144.00, 125.33), compute $r$, $r_S$ and $\hat\tau_a$.

Solution. $r = 0.659$, $r_S = 0.600$ and $\hat\tau_a = 0.467$. The pressures are means of readings with exact values $388/3$, $116$, $358/3$, $148$, $144$ and $376/3$. The means are $\bar{x} = 329/6 = 54.833$ and $\bar{y} = 391/3 = 130.333$. In exact arithmetic the sums are $S_{xx} = 696.83$, $S_{yy} = 851.33$ and $S_{xy} = 507.67$, so $r = 507.67/\sqrt{696.83 \times 851.33} = 0.659$.

The ranks of the ages are $(3, 1, 2, 4, 5, 6)$ and those of the pressures $(4, 1, 2, 6, 5, 3)$, with no ties. The differences are $(-1, 0, 0, -2, 0, 3)$, so $\sum d_i^2 = 1 + 4 + 9 = 14$, and Theorem 2 gives $r_S = 1 - 6 \times 14/(6 \times 35) = 1 - 0.4 = 0.600$.

Listed in order of age (39, 43, 54, 61, 63, 69), the pressure ranks are $1, 2, 4, 6, 5, 3$. A pair is discordant when a later entry is smaller than an earlier one: $(6, 5)$, $(6, 3)$, $(5, 3)$ and $(4, 3)$, so $D = 4$ discordant pairs, and the other $C = 11$ of the 15 pairs are concordant. So $\hat\tau_a = (11 - 4)/15 = 0.467$. The order of the three values repeats that of the twelve adults: Spearman's and Pearson's are close, Kendall's is lower.

Problem 2 (Kendall's test with four pairs). Find the exact null distribution of $S = C - D$ for $n = 4$, verify Theorem 6, and find the smallest two-sided p-value a Kendall test on four pairs can give.

Solution. Of the 24 orderings of four ranks, the numbers of inversions $D$ are $0, 1, 2, 3, 4, 5, 6$ for $1, 3, 5, 6, 5, 3, 1$ orderings, and $S = 6 - 2D$. So $S$ takes the values $6, 4, 2, 0, -2, -4, -6$ with probabilities $1, 3, 5, 6, 5, 3, 1$ out of 24. The counts of inversions can be built up one element at a time: inserting the fourth rank into an ordering of three adds 0, 1, 2 or 3 inversions, and the counts for three ranks are $1, 2, 2, 1$ for $0, 1, 2, 3$ inversions. The variance is

$$\operatorname{Var}(S) = \frac{2\,(1 \times 36 + 3 \times 16 + 5 \times 4)}{24} = \frac{208}{24} = 8.667,$$

which is $4 \times 3 \times 13/18 = 26/3$, as Theorem 6 requires. The most extreme outcome, $\lvert S\rvert = 6$, has probability $2/24$, so the smallest two-sided p-value is $1/12 = 0.083$: no two-sided Kendall test on four pairs can reject independence at the 5% level, however strong the association. A one-sided test can, but only at the single extreme outcome: $P(S = 6) = 1/24 = 0.042$.

Problem 3 (an interval and a test for the twelve). For the twelve adults, $r = 0.584$. Test $\rho = 0$ with Theorem 4 and give the 95% Fisher interval for $\rho$. What assumptions does each need?

Solution. $t = 2.27$ on 10 degrees of freedom, two-sided $p = 0.046$; the interval is $[0.015, 0.867]$. By Theorem 4, $t = 0.584\sqrt{10}/\sqrt{1 - 0.584^2} = 2.27$, and the probability that a $t$ variable with 10 degrees of freedom exceeds 2.27 in absolute value is 0.046. The test needs the pressures to be independent normal variables with a common mean and variance under the null hypothesis, given the ages; the ages may have any distribution. For the interval, $z = \operatorname{artanh}0.584 = 0.668$ and $1.96/\sqrt{9} = 0.653$, so the interval for $\zeta$ is $[0.015, 1.322]$ and, applying $\tanh$, $[0.015, 0.867]$ for $\rho$ (the lower end hardly changes because $\tanh u \approx u$ near 0). The interval rests on normality of the pairs. For these adults the population is not normal, and the experiment above found coverage near 0.960 at $n = 12$, slightly above the nominal level.

Problem 4 (the permutation variance by listing). For $n = 3$ with no ties, list the values of $r_S$ over the six possible pairings and confirm Theorem 5.

Solution. With $x$-ranks $(1, 2, 3)$ and $y$-ranks in the six orders, $\sum d_i^2$ is 0 for $(1, 2, 3)$; 2 for $(2, 1, 3)$ and $(1, 3, 2)$; 6 for $(2, 3, 1)$ and $(3, 1, 2)$; and 8 for $(3, 2, 1)$. With $n(n^2 - 1) = 24$, Theorem 2 gives $r_S = 1$, $\tfrac12$, $\tfrac12$, $-\tfrac12$, $-\tfrac12$ and $-1$. Their mean is 0 and their mean square is $(1 + \tfrac14 \times 4 + 1)/6 = \tfrac12$, which is $1/(n - 1)$ for $n = 3$. Under independence $r_S$ never takes any other value for three pairs, so even $r_S = 1$ has probability $1/6$.

Summary

Coefficient Definition Bounds proved Population value Bivariate normal
Pearson $r$ cosine of the centered vectors Theorem 1, equality for a line $\rho = \operatorname{Corr}(X, Y)$ $\rho$
Spearman $r_S$ Pearson's of the (mid)ranks Theorem 1, equality for a monotone relation $\operatorname{Corr}(F(X), G(Y))$ $\frac{6}{\pi}\arcsin\frac{\rho}{2}$
Kendall $\hat\tau_b$ cosine of the pair-sign vectors Lemma 1 $P(\text{conc.}) - P(\text{disc.})$ $\frac{2}{\pi}\arcsin\rho$

Under independence: $T = r\sqrt{n-2}/\sqrt{1-r^2}$ is $t_{n-2}$ if $Y$ is normal; the permutation variance of $r$ and $r_S$ is $1/(n - 1)$; $\operatorname{Var}(\hat\tau_a) = 2(2n+5)/[9n(n-1)]$. In large samples $\sqrt{n}(r - \rho)$ is normal with variance $\operatorname{Var}\big(X^*Y^* - \tfrac{\rho}{2}(X^{*2} + Y^{*2})\big)$, which is $(1 - \rho^2)^2$ for the normal, and $\operatorname{artanh} r$ stabilizes it.

What we have learned

Pearson's coefficient is the cosine of the angle between the centered data vectors, which bounds it by 1 and makes $\pm1$ mean a straight line. Spearman's is the same cosine for the ranks, so it measures closeness to any monotone relation, and Kendall's is the cosine of the vectors of pair signs. Their population values differ even when the population is normal, where the arcsine formulas link them, so the three numbers for the twelve adults are on different scales, not in conflict. Independence gives each coefficient an exact null distribution, through the $t$ distribution for normal data or through permutations for any data, and Fisher's transformation makes the sampling distribution of $r$ nearly normal with a variance free of $\rho$. That last step leans on normality, and on the survey's adults it gave intervals slightly too wide.

The correlation is the simplest case of a projection: $r^2$ is the share of the centered $y$ vector's squared length captured by its projection on the centered $x$ vector. The Geometry of Least Squares develops that projection for any number of predictors and turns $r^2$ into $R^2$.

References

  • Fisher, R. A. (1915). Frequency distribution of the values of the correlation coefficient in samples from an indefinitely large population. Biometrika, 10(4), 507–521. record
  • Fisher, R. A. (1921). On the "probable error" of a coefficient of correlation deduced from a small sample. Metron, 1, 3–32. record
  • Kendall, M. G. (1938). A new measure of rank correlation. Biometrika, 30(1/2), 81–93. record · 2 pages
  • Kendall, M. G. (1945). The treatment of ties in ranking problems. Biometrika, 33(3), 239–251. record
  • Kruskal, W. H. (1958). Ordinal measures of association. Journal of the American Statistical Association, 53(284), 814–861. record
  • Spearman, C. (1904). The proof and measurement of association between two things. The American Journal of Psychology, 15(1), 72–101. record