Asymptotic Relative Efficiency of Rank Tests

How much power a rank test gives up or gains against the t and F tests, measured by the limiting ratio of the sample sizes the two tests need. The definition, the cited theorem that reduces it to a ratio of slopes, the slopes of the t, sign, signed-rank and rank-sum tests derived, the efficiencies 3/pi and 2/pi under normality, tables for other shapes, the lower bound 0.864, and the cited result for the Kruskal-Wallis test; with two worked problems that compute efficiencies in closed form.

The tests of this unit, the sign and signed-rank tests, the rank-sum test and the Kruskal–Wallis test, replace the measurements by their signs or ranks, and so use less of the information in the data than the $t$ and $F$ tests do. How much power does that cost when the data are normal, and what does it gain when they are not? Under normality the cost is small for the rank tests: the signed-rank test needs about 5% more pairs than the $t$-test for the same power, and the sign test about 57% more. For heavy-tailed and skewed shapes the comparison reverses, and the rank tests need fewer observations than the $t$-test.

The measure behind those statements is the asymptotic relative efficiency of one test to another, defined in the first section. This article is its home in the unit. The three test articles quote its values and keep their own simulations of power in small samples.

First we define the measure, state the cited theorem that reduces it to a ratio of slopes, and derive the slopes of the $t$-test, the sign test and the signed-rank test for paired data. Then we do the same for the rank-sum test against the two-sample $t$-test. Next comes the cited result for the Kruskal–Wallis test. Finally two worked problems compute efficiencies in closed form. The article assumes the three test articles named above.

Asymptotic relative efficiency

The standard measure is the asymptotic relative efficiency (ARE) of one test to another. For alternatives close to the null hypothesis and large samples, it is the limiting ratio of the sample sizes the two tests need to reach the same power. An efficiency of 0.9 for test A relative to test B means that A needs about $1/0.9 = 1.11$ times as many observations as B to do as well. This article is the unit's home of the measure; the rank-sum and Kruskal–Wallis articles use it as defined here.

Consider a shift in location of a continuous distribution with density $f$ and variance $\sigma^2$, symmetric for the signed-rank and sign tests. The efficiencies relative to the $t$-test are

$$\text{ARE}(\text{signed-rank},\ t) = 12\,\sigma^2\left(\int f(x)^2\,dx\right)^2, \qquad \text{ARE}(\text{sign},\ t) = 4\,\sigma^2 f(0)^2,$$

where $f(0)$ is the density at the center of symmetry. These are the standard Pitman efficiencies.

Here is the idea behind them. A test is good at detecting a small shift if its statistic moves a lot when the center moves, compared with how much it varies by chance. The ratio of the two is the test's slope, and efficiency compares slopes.

The formal version. The general result is a theorem of asymptotic statistics, cited here and not proved (see van der Vaart, 1998, chapter 14, Theorem 14.19). The model is the shift family: $D = \Delta + E$, where $E$ has the symmetric density $f$, so the null hypothesis is $\Delta = 0$. A test rejects for large values of a statistic $T_n$ computed from $n$ observations. Suppose there are functions $\mu(\Delta)$ and $\sigma(\Delta)$ such that

$$\frac{\sqrt{n}\,\big(T_n - \mu(\Delta_n)\big)}{\sigma(\Delta_n)} \to N(0, 1) \quad \text{in distribution, along every sequence of shifts } \Delta_n = h/\sqrt{n},$$

with $\mu$ differentiable at 0 and $\sigma$ continuous at 0. So $\mu(\Delta)$ is the value the statistic settles on when the shift is $\Delta$, and $\sigma(0)/\sqrt{n}$ is its standard deviation near the null hypothesis. The slope of the test is $\mu'(0)/\sigma(0)$. The theorem says that, under further regularity conditions on the family of distributions stated there (among them $\mu'(0) > 0$), the efficiency of one such test relative to another is the squared ratio of their slopes.

The slopes are derived next. The convergence along shrinking shifts, which the theorem assumes, is not verified on this page. For the three statistics below it is part of the cited theory. Smooth shapes such as those in the table below are the setting the theorem is stated for. For the uniform density of the first worked problem, whose support moves with the shift, the slopes are computed in the same way, and the values 1 and $\tfrac13$ are the classical ones, but the cited theorem's conditions are not checked here.

The $t$-test. Take $T_n = \bar{d}$, the mean difference. Then $\mu(\Delta) = \Delta$, so $\mu'(0) = 1$. The standard deviation of $\bar{d}$ is $\sigma/\sqrt{n}$, so $\sigma(0) = \sigma$. The slope is $1/\sigma$. (The $t$ statistic divides by the sample standard deviation instead of $\sigma$. The two agree in the limit, which is again part of the cited theory.)

The sign test. Take $T_n = S/n$, the proportion of plus signs. Write $F$ for the distribution function of $E$. Then

$$\mu(\Delta) = P(D > 0) = P(E > -\Delta) = 1 - F(-\Delta) = F(\Delta),$$

the last step by symmetry. So $\mu'(0) = f(0)$. Under the null hypothesis $S$ is binomial with $p = \tfrac12$, so the variance of $S/n$ is $1/(4n)$ and $\sigma(0) = \tfrac12$. The slope is $2f(0)$, and the squared ratio to the $t$-test's slope is $4\sigma^2 f(0)^2$.

The signed-rank test. Take $T_n = W^+/\binom{n}{2}$. By the formula for $E(W^+)$ in the signed-rank article's section on what each test tests, its mean is $P(D_1 + D_2 > 0)$ plus a term of order $1/n$, so

$$\mu(\Delta) = P(D_1 + D_2 > 0) = P(E_1 + E_2 > -2\Delta).$$

Let $h$ be the density of $E_1 + E_2$. The derivative of $\mu$ at 0 is $2h(0)$: the factor 2 comes from the $-2\Delta$. The density of a sum of two independent variables is $h(s) = \int f(x)f(s - x)\,dx$, so $h(0) = \int f(x)f(-x)\,dx = \int f(x)^2\,dx$ by symmetry. Hence $\mu'(0) = 2\int f^2$. For the standard deviation, $\operatorname{Var}(W^+) = n(n+1)(2n+1)/24$ is $n^3/12$ to leading order, and $\binom{n}{2}^2$ is $n^4/4$ to leading order. So the variance of $T_n$ is $(n^3/12)/(n^4/4) = 1/(3n)$ to leading order, and $\sigma(0) = 1/\sqrt{3}$. The slope is $2\sqrt{3}\int f^2 = \sqrt{12}\int f^2$, and the squared ratio to $1/\sigma$ is $12\sigma^2\left(\int f^2\right)^2$, the first expression.

Fay and Proschan (2010, §5.3.4) give the same expression for the two-sample rank-sum test, and the next section derives its slopes.

Now the check on the normal case. For the normal density with standard deviation $\sigma$, $f(x)^2 = e^{-x^2/\sigma^2}/(2\pi\sigma^2)$. Since $\int e^{-x^2/\sigma^2}\,dx = \sigma\sqrt{\pi}$ (the normal integral of The Normal Distribution with $\sigma^2/2$ in place of $\sigma^2$), $\int f^2 = \sigma\sqrt{\pi}/(2\pi\sigma^2) = 1/(2\sigma\sqrt{\pi})$. Also $f(0) = 1/(\sigma\sqrt{2\pi})$. So

$$\text{ARE}(\text{signed-rank},\ t) = 12\sigma^2 \cdot \frac{1}{4\pi\sigma^2} = \frac{3}{\pi} = 0.955, \qquad \text{ARE}(\text{sign},\ t) = 4\sigma^2\cdot\frac{1}{2\pi\sigma^2} = \frac{2}{\pi} = 0.637.$$

Under normality the signed-rank test needs about 5% more pairs than the $t$-test for the same power, and the sign test about 57% more. The second number is the price of the median met in the sampling distribution of the mean, whose variance for normal data is $\pi/2$ times that of the mean. Wilcoxon (1945) cited an efficiency of 63 percent for the sign test from Cochran's work. He expected that his test, which uses the sizes of the differences as well as their signs, should do better, but wrote that he did not know its efficiency.

Example (the Laplace distribution). For the Laplace (double exponential) distribution, which has a sharp peak at its center and heavier tails than the normal, the sign test is the most efficient of the three: it scores 2.0 against the $t$-test, and the signed-rank test 1.5. The first worked problem derives both values. For heavier-tailed distributions the comparison with the $t$-test reverses. The script computed the two expressions by numerical integration for five symmetric shapes:

Shape of the differences Signed-rank vs $t$ Sign vs $t$
Normal 0.955 0.637
Logistic 1.097 0.822
Laplace (double exponential) 1.500 2.000
$t$ with 3 degrees of freedom 1.900 1.621
Contaminated normal (10% with three times the SD) 1.373 0.998

The signed-rank test loses at most a little when the data are normal, and gains a great deal when they have heavy tails or occasional wild values. The sign test is the best of the three for the Laplace distribution and the worst for the normal. The Laplace distribution's sharp peak at the center makes the sign of an observation unusually informative.

How far these large-sample numbers carry over to 20 pairs is measured by simulation in the signed-rank article's section on power in small samples.

The rank-sum test against the two-sample t-test

For two independent samples the measure is the same: the limiting ratio of the sample sizes two tests need for the same power against small shifts. Under a shift model with a continuous density $f$ of variance $\sigma^2$, the efficiency of the rank-sum test to the two-sample $t$-test is

$$\text{ARE} = 12\,\sigma^2\left(\int f(x)^2\,dx\right)^2$$

(Fay and Proschan, 2010, §5.3.4). It is the same expression as for the signed-rank test against the paired $t$-test. Here is the idea of the derivation: find each test's slope, the rate at which its statistic moves with the shift divided by its standard deviation, and square the ratio.

The formal version. The theorem is the one stated in the previous section (see van der Vaart, 1998, chapter 14, Theorem 14.19): if a statistic $T_N$ settles on $\mu(\Delta)$ under a shift $\Delta$ and has standard deviation $\sigma(0)/\sqrt{N}$ near the null hypothesis, the test's slope is $\mu'(0)/\sigma(0)$, and, under the regularity conditions stated there, the efficiency of one test relative to another is the squared ratio of their slopes. As there, the theorem and its condition of asymptotic normality along shrinking shifts are cited, and the slopes are derived here. The number of observations is $N = n_1 + n_2$, and the share of the first sample, $\lambda = n_1/N$, is held fixed as $N$ grows. Let $f$ be the common density of the two populations apart from the shift, so that $X$ has the distribution of $Y + \Delta$.

The $t$-test. Take $T_N = \bar{x} - \bar{y}$. Then $\mu(\Delta) = \Delta$ and $\mu'(0) = 1$. Its variance is

$$\sigma^2\left(\frac{1}{n_1} + \frac{1}{n_2}\right) = \frac{\sigma^2 N}{n_1 n_2} = \frac{\sigma^2}{N\lambda(1-\lambda)},$$

so $\sigma(0) = \sigma/\sqrt{\lambda(1-\lambda)}$ and the slope is $\sqrt{\lambda(1-\lambda)}/\sigma$.

The rank-sum test. Take $T_N = U/(n_1 n_2)$, whose mean is $\theta$. By the proof in What the Rank-Sum Test Tests, with $V = Y - Y'$ for two independent draws $Y$ and $Y'$ from the second population,

$$\mu(\Delta) = \theta(\Delta) = P(V < \Delta),$$

for continuous data. So $\mu'(0)$ is the density of $V$ at 0. The density of a difference of two independent variables with density $f$ is $\int f(y)f(y - v)\,dy$, which at $v = 0$ is $\int f(y)^2\,dy$. Hence $\mu'(0) = \int f^2$. Under the null hypothesis,

$$\operatorname{Var}\left(\frac{U}{n_1 n_2}\right) = \frac{n_1 n_2 (N+1)}{12\,n_1^2 n_2^2} = \frac{N+1}{12\,n_1 n_2} = \frac{N+1}{12\,N^2\lambda(1-\lambda)},$$

which is $1/\big(12N\lambda(1-\lambda)\big)$ to leading order. So $\sigma(0) = 1/\sqrt{12\lambda(1-\lambda)}$ and the slope is $\sqrt{12\lambda(1-\lambda)}\int f^2$.

The squared ratio of the two slopes is

$$\frac{12\lambda(1-\lambda)\left(\int f^2\right)^2}{\lambda(1-\lambda)/\sigma^2} = 12\,\sigma^2\left(\int f(x)^2\,dx\right)^2.$$

The share $\lambda$ cancels, so the efficiency does not depend on how the observations are divided between the groups. For the normal distribution, $\int f^2 = 1/(2\sigma\sqrt\pi)$, as computed in the previous section, so

$$\text{ARE} = 12\sigma^2 \cdot \frac{1}{4\pi\sigma^2} = \frac{3}{\pi} = 0.955.$$

When the data are normal, where the $t$-test is the best possible test, the rank-sum test needs only about 5% more observations to match it. For other shapes the script evaluated the integral numerically:

Shape ARE, rank-sum vs $t$
Normal 0.955
Uniform 1.000
Logistic 1.097
Contaminated normal (10% with three times the SD) 1.373
Laplace (double exponential) 1.500
$t$ with 3 degrees of freedom 1.900
Exponential 3.000

The efficiency can be far above 1 and never falls far below it. Fay and Proschan (2010, §5.3.4) give the smallest possible value over all shapes, $108/125 = 0.864$, attained by a parabolic density. The bound is a result of Hodges and Lehmann (1956), who showed that the efficiency never falls below 0.864. The second worked problem computes that value, and the uniform and exponential rows of the table, in closed form. They also show that for $t$ distributions the rank-sum test is the more efficient for 18 or fewer degrees of freedom. In their recommendations (§5.3.6) they cite a review of high-quality real measurements, which found that $t$ distributions with 3 to 10 degrees of freedom often fit better than the normal. They conclude that the rank-sum test "will often be asymptotically more powerful than the t-test for real high quality data".

The rank-sum article's section on power at 20 per group compares these numbers with a simulation.

Several groups: the Kruskal–Wallis test

Under a shift model, the asymptotic relative efficiency of the Kruskal–Wallis test to the $F$ test of one-way analysis of variance is the same expression as that of the rank-sum test to the $t$-test, $12\sigma^2\left(\int f^2\right)^2$. That is a cited result (Andrews, 1954), not derived here. It gives $3/\pi = 0.955$ for normal data, 1.9 for the $t$ distribution with 3 degrees of freedom and 3 for the exponential, the values in the table of the previous section.

The Kruskal–Wallis article's section on power compares the tests by simulation at small sample sizes.

Worked problems

The two problems are posed on this article's own results; none is taken from the course text. The first is checked by tools/experiments/applied-statistics/sign-and-signed-rank-tests.py in the site's repository and the second by wilcoxon-rank-sum-test.py beside it, which also compute the tables of this article by numerical integration.

Problem 1 (two efficiencies in closed form). Find the asymptotic relative efficiencies of the signed-rank test and the sign test to the $t$-test for the Laplace density $f(x) = e^{-|x|/b}/(2b)$ and for the uniform density on $[-a, a]$.

Solution. For the Laplace density the efficiencies are $\tfrac32$ and 2. For the uniform density they are 1 and $\tfrac13$. Each needs three ingredients: the variance $\sigma^2$, the integral $\int f^2$ and the central height $f(0)$.

Laplace. By symmetry, and with $\int_0^\infty x^2 e^{-x/b}\,dx = 2b^3$,

$$\sigma^2 = 2\int_0^\infty x^2\,\frac{e^{-x/b}}{2b}\,dx = \frac{2b^3}{b} = 2b^2, \qquad \int f^2 = 2\int_0^\infty \frac{e^{-2x/b}}{4b^2}\,dx = \frac{1}{2b^2}\cdot\frac{b}{2} = \frac{1}{4b}, \qquad f(0) = \frac{1}{2b}.$$

So

$$\text{ARE}(\text{signed-rank},\ t) = 12 \cdot 2b^2 \cdot \frac{1}{16b^2} = \frac32, \qquad \text{ARE}(\text{sign},\ t) = 4 \cdot 2b^2 \cdot \frac{1}{4b^2} = 2.$$

The scale $b$ cancels, as it must: an efficiency cannot depend on the units of measurement.

Uniform. Here $f = 1/(2a)$ on an interval of length $2a$, so

$$\sigma^2 = \int_{-a}^{a}\frac{x^2}{2a}\,dx = \frac{a^2}{3}, \qquad \int f^2 = \frac{2a}{4a^2} = \frac{1}{2a}, \qquad f(0) = \frac{1}{2a},$$

and

$$\text{ARE}(\text{signed-rank},\ t) = 12 \cdot \frac{a^2}{3} \cdot \frac{1}{4a^2} = 1, \qquad \text{ARE}(\text{sign},\ t) = 4 \cdot \frac{a^2}{3} \cdot \frac{1}{4a^2} = \frac13.$$

For flat data the signed-rank test matches the $t$-test in large samples, and the sign test needs three times as many pairs. The two shapes are at opposite ends: the sign test gains the most where the density is peaked at its center and loses the most where it is flat. The script's numerical integration agrees with all four values.

Problem 2 (three efficiencies in closed form). Find the asymptotic relative efficiency of the rank-sum test to the $t$-test under a shift model for the uniform density on $[0, 1]$, the exponential density $e^{-x}$ on $x \ge 0$, and the parabolic density $\tfrac34(1 - x^2)$ on $[-1, 1]$.

Solution. The efficiencies are 1, 3 and $108/125 = 0.864$. Each is $12\sigma^2\left(\int f^2\right)^2$.

Uniform. $\sigma^2 = \int_0^1 x^2\,dx - \left(\tfrac12\right)^2 = \tfrac13 - \tfrac14 = \tfrac1{12}$ and $\int f^2 = 1$, so the efficiency is $12 \cdot \tfrac1{12} \cdot 1 = 1$.

Exponential. The mean is $\int_0^\infty x e^{-x}\,dx = 1$ and $\int_0^\infty x^2 e^{-x}\,dx = 2$, so $\sigma^2 = 2 - 1 = 1$. Also $\int f^2 = \int_0^\infty e^{-2x}\,dx = \tfrac12$. The efficiency is $12 \cdot 1 \cdot \tfrac14 = 3$.

Parabolic. First, it is a density: $\tfrac34\int_{-1}^{1}(1 - x^2)\,dx = \tfrac34\left(2 - \tfrac23\right) = 1$. Its mean is 0 by symmetry, so

$$\sigma^2 = \frac34\int_{-1}^{1}(x^2 - x^4)\,dx = \frac34\left(\frac23 - \frac25\right) = \frac34 \cdot \frac{4}{15} = \frac15,$$

$$\int f^2 = \frac{9}{16}\int_{-1}^{1}(1 - 2x^2 + x^4)\,dx = \frac{9}{16}\left(2 - \frac43 + \frac25\right) = \frac{9}{16}\cdot\frac{16}{15} = \frac35,$$

and the efficiency is $12 \cdot \tfrac15 \cdot \tfrac{9}{25} = \tfrac{108}{125} = 0.864$. This is the smallest value the efficiency can take, as cited in the section on the rank-sum test. That no density gives less is Hodges and Lehmann's (1956) result, not proved here. The uniform density is not a smooth location family, and the cited efficiency theorem's conditions are not checked for it; its value 1 is the classical one. The script's numerical integrals give 1.000, 3.001 and 0.864.

What we have learned

The asymptotic relative efficiency of one test to another is the limiting ratio of the sample sizes they need for the same power against small shifts. By a cited theorem it is the squared ratio of the tests' slopes, and the slopes of the $t$, sign, signed-rank and rank-sum tests were derived here. Against the $t$-test the signed-rank and rank-sum tests have efficiency $12\sigma^2\left(\int f^2\right)^2$, which is $3/\pi = 0.955$ for normal data, far above 1 for heavy-tailed and skewed shapes, and never below $108/125 = 0.864$. The sign test has $4\sigma^2 f(0)^2$, which is $2/\pi = 0.637$ for normal data and 2 for the Laplace shape. The Kruskal–Wallis test has the rank-sum test's efficiency against the $F$ test, a cited result.

These are limits for large samples and small shifts. The simulations in the three test articles show how far they carry over to samples of 10 or 20. This article ends the unit. The next unit, Inference on Proportions, begins with the normal approximation to the binomial.

References

  • Andrews, F. C. (1954). Asymptotic behavior of some rank tests for analysis of variance. The Annals of Mathematical Statistics, 25(4), 724–736. record
  • Fay, M. P., and Proschan, M. A. (2010). Wilcoxon–Mann–Whitney or t-test? On assumptions for hypothesis tests and multiple interpretations of decision rules. Statistics Surveys, 4, 1–39. record · 4 pages
  • Hodges, J. L., and Lehmann, E. L. (1956). The efficiency of some nonparametric competitors of the t-test. The Annals of Mathematical Statistics, 27(2), 324–335. record
  • van der Vaart, A. W. (1998). Asymptotic Statistics. Cambridge: Cambridge University Press. Chapter 14 (relative efficiency of tests). record · 4 pages
  • Wilcoxon, F. (1945). Individual comparisons by ranking methods. Biometrics Bulletin, 1(6), 80–83. record · 4 pages