Ten patients each took two sleeping drugs. On the second drug every patient but one slept longer than on the first, and the remaining patient slept the same. The paired t-test summarized the ten differences by their mean, 1.58 hours, and their standard deviation, and found $p = 0.0028$. That $p$-value assumes the differences come from a normal population. Ten values cannot confirm it, and one of them, 4.6 hours, is nearly twice the next largest. Can we test whether the second drug helps without assuming anything about the shape of the differences?
R. A. Fisher, analyzing the same data in 1925, added a check that needed no such assumption. If the drugs were equally effective, each of the nine nonzero differences would be as likely to be negative as positive. All nine would then share the same sign by chance "only twice in 512 trials" (Fisher, 1925, §24; see Hypothesis Tests for a Mean). That calculation uses nothing but the signs of the differences. It does not assume that the differences are normal, or that their mean describes them well. It does not even assume a scale on which subtraction means anything beyond "more" or "less".
This article develops two tests built on that idea:
- The sign test counts how many differences are positive and refers the count to a binomial distribution.
- The Wilcoxon signed-rank test, proposed by Frank Wilcoxon (1945), also uses the order of the differences' sizes. It ranks the absolute differences and adds up the ranks that carry a plus sign.
First we state what each test's null hypothesis says, and work the sign test. Then we build the signed-rank test and find its exact null distribution by counting sign patterns, on drug 1 alone and on Wilcoxon's own example. Next comes the normal approximation, with corrections for ties and zero differences, on repeated NHANES blood pressure readings. Finally we ask what each test actually tests and what it costs, in efficiency and power, against the $t$-test. The article assumes The Paired Samples t-Test. It is the unit's home of exact null distributions by counting, the correction for ties and the asymptotic relative efficiency; the rank-sum test and the Kruskal–Wallis test build on all three.
Paired data and the null hypothesis
The setting is the one of the paired t-test. Either there are $n$ pairs of measurements, each reduced to a difference $d_i = x_{i2} - x_{i1}$, or there is a single sample $x_1, \dots, x_n$ to be compared with a hypothesized center $m_0$, in which case $d_i = x_i - m_0$. Write $D$ for the difference of a randomly chosen pair. The $t$-test asks whether $E(D) = 0$, and it needs $D$ to be roughly normal in small samples. The rank tests ask about the distribution of $D$ in other terms, and their null hypotheses differ:
- the sign test asks whether $D$ is as likely to be positive as negative, $P(D > 0) = P(D < 0)$; when $D$ is continuous this says that the median of $D$ is 0;
- the signed-rank test asks whether $D$ is symmetric about 0, that is, whether $D$ and $-D$ have the same distribution.
For a symmetric distribution the center of symmetry is both the median and (if it exists) the mean. So under symmetry all three tests are about the same number. When the distribution of the differences is skewed they part ways. The section on what each test tests shows by example that they can then reach different conclusions on the same population, each correctly.
Throughout, a difference of exactly zero carries no sign. The usual convention, Wilcoxon's, is to drop such pairs and let $n$ be the number of nonzero differences. An alternative due to Pratt is described with the normal approximation below.
The sign test
Here is the idea in one sentence: if the null hypothesis holds, each nonzero difference is a fair coin toss, so the number of plus signs is binomial.
Let $S$ be the number of positive differences among the $n$ nonzero ones. If the pairs are independent and $P(D > 0) = P(D < 0)$, then each nonzero difference is positive with probability $\tfrac{1}{2}$, independently of the others, so
$$S \sim \text{Binomial}\left(n, \tfrac{1}{2}\right), \qquad P(S = s) = \binom{n}{s}\frac{1}{2^n}, \quad s = 0, 1, \dots, n.$$
Large and small values of $S$ are both evidence against $H_0$. Because the binomial with $p = \tfrac12$ is symmetric, the two-sided $p$-value is twice the smaller tail:
$$p = \min\left\{1,\ 2\,P\big(S \le \min(s, n - s)\big)\right\} = \min\left\{1,\ \frac{2}{2^n}\sum_{i=0}^{\min(s,\,n-s)}\binom{n}{i}\right\}.$$
It is exact for every $n$. No approximation is involved, and no assumption beyond independent pairs and the null hypothesis itself. It is the exact binomial test of a proportion with $p_0 = \tfrac12$, which Inference on Proportions develops for any $p_0$.
Back to the ten sleep patients. Nine differences are positive, none is negative and one is zero, so $n = 9$, $s = 9$ and
$$p = 2\,P(S = 9) = 2 \cdot \frac{1}{512} = 0.0039.$$
This is Fisher's calculation: all nine signs alike, plus or minus, twice in 512. It is also the smallest two-sided $p$-value the sign test can produce with nine pairs, since the data are as extreme as nine signs can be. The test gives up what the paired $t$-test used, the sizes of the differences. Nothing in its result reflects that one patient gained 4.6 hours and another 0.8.
For large $n$ the binomial is approximated by the normal with mean $n/2$ and variance $n/4$. The continuity correction is one half, because $S$ moves in steps of 1:
$$z = \frac{|s - n/2| - \tfrac12}{\sqrt{n/4}}.$$
The approximation is examined on a larger sample below. First, can we keep some of the information about sizes that the sign test throws away?
The Wilcoxon signed-rank test
The sign test treats a difference of 0.8 hours and one of 4.6 hours alike. Wilcoxon's refinement keeps the order of the sizes, though not the sizes themselves. Rank the absolute values $|d_1|, \dots, |d_n|$ from 1 (smallest) to $n$ (largest). Tied values get the mean of the ranks they span, called their midrank. Then attach to each rank the sign of its difference. The test statistic is
$$W^+ = \sum_{i:\, d_i > 0} r_i,$$
the sum of the ranks of the positive differences, where $r_i$ is the rank of $|d_i|$. Its complement $W^- = \sum_{i:\,d_i<0} r_i$ carries the same information, because the ranks always add up to $1 + 2 + \cdots + n$:
$$W^+ + W^- = \frac{n(n+1)}{2}.$$
Wilcoxon (1945) worked with "the sum of rank numbers of one sign only, + or −, whichever is less", and tabulated its small values. Software usually reports one of the two sums, and the $p$-value is the same either way.
The null distribution, by counting
Here is the idea: under the null hypothesis the signs are fair coin tosses attached to fixed ranks, so we can list every outcome.
Under $H_0$ the distribution of $D$ is symmetric about 0. For a continuous symmetric distribution the sign of $D$ is then independent of its size $|D|$, and each sign is $+$ or $-$ with probability $\tfrac12$. The ranks depend only on the sizes. So, given the ranks, the signs attached to them are $n$ independent fair coin tosses. Every one of the $2^n$ patterns of signs is equally likely. The null distribution of $W^+$ is found by listing the patterns and adding up, for each, the ranks that carry a plus sign. This is the sign test's reasoning applied to ranks instead of counts, and like the sign test it is exact for any $n$.
For the sleep data the computation is short. Dropping the zero leaves nine differences. Their sizes and ranks are
| $\lvert d_i \rvert$ (hours) | 0.8 | 1.0 | 1.2 | 1.3 | 1.3 | 1.4 | 1.8 | 2.4 | 4.6 |
|---|---|---|---|---|---|---|---|---|---|
| Rank | 1 | 2 | 3 | 4.5 | 4.5 | 6 | 7 | 8 | 9 |
| Sign | + | + | + | + | + | + | + | + | + |
The two differences of 1.3 hours share ranks 4 and 5 and receive 4.5 each.
Check your understanding. The signed-rank test uses more information than the sign test, which gave $p = 0.0039$. Will its $p$-value for these nine differences be smaller?
It will not. All signs are positive, so $W^+ = 45$ and $W^- = 0$. Only one of the 512 sign patterns gives $W^+ = 45$ and only one gives $W^+ = 0$, so the exact two-sided $p$-value is $2/512 = 0.0039$, the same as the sign test's. When every difference has the same sign, the sizes cannot add anything. With nine pairs neither rank test can produce a $p$-value below $0.0039$, while the paired $t$-test, which uses the actual sizes, gives 0.0028.
A worked enumeration: drug 1 against no drug
All plus signs make a dull example, so we need data where the signs are mixed. Student's numbers are the extra hours of sleep each patient gained on a drug compared with nights without one. A more instructive question for the rank tests is whether the first drug, on its own, adds any sleep at all. That is, do the ten values for drug 1,
$$0.7,\ -1.6,\ -0.2,\ -1.2,\ -0.1,\ 3.4,\ 3.7,\ 0.8,\ 0.0,\ 2.0,$$
come from a distribution centered at zero? Their mean is 0.75 hours and their median 0.35. Five are positive, four negative and one zero.
The sign test. With $n = 9$ and $s = 5$, $P(S \ge 5) = 0.5$, so $p = 1$. Five plus signs out of nine is exactly what a fair coin would produce most often.
Check your understanding. Before ranking, look at the ten values. Will the signed-rank test find stronger evidence of a positive center than the sign test's $p = 1$, or weaker?
Stronger, though not strong. The negative values are the small ones (except $-1.6$), and the three largest values are positive, so the ranks lean further toward a positive center than the plain count of signs does. The rest of this section measures how far.
The signed-rank test. The nine nonzero values ranked by size are
| $\lvert x_i \rvert$ | 0.1 | 0.2 | 0.7 | 0.8 | 1.2 | 1.6 | 2.0 | 3.4 | 3.7 |
|---|---|---|---|---|---|---|---|---|---|
| Rank | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 |
| Sign | − | − | + | + | − | − | + | + | + |
so $W^- = 1 + 2 + 5 + 6 = 14$ and $W^+ = 45 - 14 = 31$.
To find the null distribution, the script enumerated all $2^9 = 512$ sign patterns of the ranks 1 to 9. It counted the patterns giving each value of $W^+$ from 0 to 45. The counts rise from 1 at each end to a maximum of 23 in the middle, and they are symmetric about the mean 22.5. The lower tail is
| $w$ | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Patterns with $W^+ = w$ | 1 | 1 | 1 | 2 | 2 | 3 | 4 | 5 | 6 | 8 | 9 | 10 |
| $P(W^+ \le w)$ | 0.0020 | 0.0039 | 0.0059 | 0.0098 | 0.0137 | 0.0195 | 0.0273 | 0.0371 | 0.0488 | 0.0645 | 0.0820 | 0.1016 |
For example, $W^+ = 3$ arises from two patterns: rank 3 alone positive, or ranks 1 and 2 positive. Continuing the count to $w = 14$ gives 92 patterns with $W^+ \le 14$, and by symmetry 92 with $W^+ \ge 31$, so
$$p = \frac{92 + 92}{512} = 0.359.$$
The table also gives the critical values. A two-sided test at $\alpha = 0.05$ with nine pairs rejects when $\min(W^+, W^-) \le 5$, which has actual size $2 \times 0.0195 = 0.039$. The next value, 6, would give 0.055. As with any test on a discrete statistic, the attainable levels are a finite list. In practice "$\alpha = 0.05$" means the largest attainable level that does not exceed 0.05.
The one-sample $t$-test on the same ten values gives $t = 1.326$ on 9 degrees of freedom and $p = 0.218$. The three tests agree that the data are consistent with drug 1 adding nothing. They rank the strength of the evidence in the order of the information they use:
- the sign test sees five against four ($p = 1$);
- the signed-rank test sees that the large values are mostly positive ($p = 0.359$);
- the $t$-test sees how large they are ($p = 0.218$).
Wilcoxon's own example
Does more information always mean a smaller $p$-value? Wilcoxon's own example says no. He illustrated the paired test with fifteen differences in height between cross-fertilized and self-fertilized corn (maize) plants grown in pairs. He took them from Fisher's The Design of Experiments (§17), where Fisher analyzes them as data from Charles Darwin's experiments on cross- and self-fertilization (Fisher, 1935, ch. III). In eighths of an inch, Fisher's unit, the differences were
$$6,\ 8,\ 14,\ 16,\ 23,\ 24,\ 28,\ 29,\ 41,\ -48,\ 49,\ 56,\ 60,\ -67,\ 75.$$
The two negative differences have ranks 10 and 14, so the smaller rank sum is 24. Wilcoxon's table showed that the probability of a total of 24 or less "is between 0.019 and 0.054 for 15 pairs", and he noted that Fisher's $t$-test gave 0.0497. The script recomputed all three tests:
- the signed-rank test, by enumerating the $2^{15} = 32{,}768$ sign patterns: exact two-sided $2\,P(W^- \le 24) = 0.0413$;
- the $t$-test: $t = 2.148$ and $p = 0.0497$, as Fisher found;
- the sign test, with 13 positive differences out of 15: $p = 0.0074$, the smallest of the three.
That reversal of the usual order is instructive. The two negative differences are among the largest in absolute size. The tests that weigh sizes, by value or by rank, are pulled toward the null by them, while the sign test counts them as two votes out of fifteen.
Recomputing Wilcoxon's Table II by enumeration reproduces his probabilities to within 0.01, and most to the third decimal. He described the tabulation as an exercise in counting partitions of integers. The number of sign patterns with $W^- = r$ is the number of ways to write $r$ as a sum of distinct ranks, which he related to tables of partitions. A computer makes the counting immediate. For small $n$ it can list the patterns. For larger $n$ it can build the distribution one rank at a time: if $P_{k}(w)$ is the distribution of the sum over the first $k$ ranks, then
$$P_{k}(w) = \tfrac12 P_{k-1}(w) + \tfrac12 P_{k-1}(w - r_k),$$
since the $k$-th rank either is or is not included, with probability one half each. The recursion works for midranks too, after doubling them so that they are whole numbers. That gives exact $p$-values in the presence of ties.
The normal approximation
Exact counting is cheap for nine pairs, but what about a few hundred? For larger $n$ the exact distribution is well approximated by a normal distribution with the same mean and variance. Here is the plan: write $W^+$ as a sum of independent pieces, one per rank, and add up their means and variances.
The formal version. Write $W^+ = \sum_{i=1}^n r_i I_i$, where $I_i = 1$ if the $i$-th rank carries a plus sign and 0 otherwise. Under $H_0$ the $I_i$ are independent with $P(I_i = 1) = \tfrac12$, so $E(I_i) = \tfrac12$ and $\operatorname{Var}(I_i) = \tfrac14$. The ranks are fixed numbers. Hence
$$E(W^+) = \frac12 \sum_{i=1}^n r_i, \qquad \operatorname{Var}(W^+) = \frac14 \sum_{i=1}^n r_i^2.$$
Without ties the ranks are $1, 2, \dots, n$, with $\sum i = n(n+1)/2$ and $\sum i^2 = n(n+1)(2n+1)/6$, so
$$E(W^+) = \frac{n(n+1)}{4}, \qquad \operatorname{Var}(W^+) = \frac{n(n+1)(2n+1)}{24},$$
and the standardized statistic is
$$z = \frac{\left|W^+ - \dfrac{n(n+1)}{4}\right| - \dfrac12}{\sqrt{\dfrac{n(n+1)(2n+1)}{24}}},$$
with the one-half a continuity correction, since $W^+$ moves in steps of 1. $W^+$ is a sum of independent bounded terms, none of which dominates, and that is why it is close to normal. The figure above shows how close it already is at $n = 9$.
Back to drug 1. There $E(W^+) = 22.5$, $\operatorname{Var}(W^+) = 9 \cdot 10 \cdot 19/24 = 71.25$, and
$$z = \frac{|31 - 22.5| - 0.5}{\sqrt{71.25}} = \frac{8}{8.441} = 0.948, \qquad p = 0.343,$$
against the exact 0.359. Without the continuity correction, $z = 1.007$ and $p = 0.314$. The correction matters most in small samples. At the exact critical values for 10, 20 and 30 pairs the script found:
| Pairs $n$ | Critical $\min(W^+, W^-)$ | Exact two-sided level | Normal, with correction | Normal, without |
|---|---|---|---|---|
| 10 | 8 | 0.0488 | 0.0528 | 0.0469 |
| 20 | 52 | 0.0484 | 0.0500 | 0.0479 |
| 30 | 137 | 0.0497 | 0.0507 | 0.0495 |
With ten or more pairs the approximation is good to about half a percentage point. Since software computes exact values, the approximation matters only for large samples.
The derivation assumed the ranks $1, \dots, n$. Real measurements are often recorded coarsely, so what happens when some of them tie?
Ties
Here is the idea in one sentence: midranks keep the total of the ranks but squeeze them toward their middle, so the mean of $W^+$ stays put and its variance shrinks.
When some absolute differences are tied and receive midranks, the mean $\tfrac12\sum r_i$ is unchanged. A midrank is the average of the ranks it replaces, so the total is still $n(n+1)/2$. The variance $\tfrac14\sum r_i^2$ is smaller. Replacing $t$ consecutive ranks $a+1, \dots, a+t$ by their mean $a + (t+1)/2$ reduces their sum of squares by their sum of squared deviations from that mean. That is $t$ times the variance of the integers $1, \dots, t$:
$$t \cdot \frac{t^2 - 1}{12} = \frac{t^3 - t}{12}.$$
Summing over the groups of ties and dividing by 4,
$$\operatorname{Var}(W^+) = \frac{n(n+1)(2n+1)}{24} - \frac{1}{48}\sum_{\text{tie groups}} (t^3 - t),$$
where $t$ is the number of values in each group of tied absolute differences. Kruskal and Wallis (1952, §3.1.2) derive the same reduction, $(t^3-t)/12$ in the sum of squares, for their test, and the rank-sum test and the Kruskal–Wallis test use it in this form. The correction is small unless the ties are extensive, but with measurements recorded coarsely they can be.
Zero differences
Dropping zero differences, as above, is Wilcoxon's convention and the default in most software. Its consequence is that pairs showing no change contribute nothing to the test. Pratt's alternative keeps the zeros in the ranking, where they occupy the smallest ranks, and then drops their ranks from the sum. The remaining ranks then start above 1. The mean and variance are computed from the ranks actually used, $\tfrac12\sum r_i$ and $\tfrac14\sum r_i^2$, as above. The scipy documentation describes the Pratt method as the more conservative of the two. When the zeros are few the choice hardly matters. When they are many, as in the next example, it can.
Repeated blood pressure readings
The sleep data have one tie and one zero. A sample with many of both shows the corrections at work. The NHANES examination of 2017 to 2018 took up to three systolic readings from each participant, recorded in steps of 2 mm Hg. The paired t-test article drew 40 adults at random (the script's seed 1) and analyzed the first reading minus the third. Their differences, in mm Hg, with counts, are
| Difference | −12 | −10 | −8 | −6 | −4 | −2 | 0 | 2 | 4 | 6 | 8 | 10 | 12 | 14 | 22 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Count | 1 | 1 | 2 | 2 | 6 | 6 | 7 | 3 | 4 | 1 | 1 | 1 | 1 | 3 | 1 |
The paired $t$-test gave $\bar{d} = 0.900$ mm Hg, $t = 0.790$ and $p = 0.434$. Seven differences are zero, and the rest have only eight distinct sizes, so this is a sample with heavy ties.
Sign test. There are 15 positive and 18 negative differences, $n = 33$. The exact two-sided $p$-value is 0.728; the normal approximation, $z = (|15 - 16.5| - 0.5)/\sqrt{33/4} = 0.348$, gives 0.728 as well.
Signed-rank test, zeros dropped. The 33 nonzero absolute differences take the values 2 (nine times), 4 (ten times), 6, 8, 10, 12, 14 and 22, with midranks 5, 14.5, 21, 24, 26.5, 28.5, 31 and 33. The positive differences hold ranks totaling $W^+ = 299$, and $W^- = 262$; together they make $33 \cdot 34/2 = 561$. Then:
- $E(W^+) = 33 \cdot 34/4 = 280.5$;
- without ties, $\operatorname{Var}(W^+) = 33 \cdot 34 \cdot 67/24 = 3132.25$;
- the tie groups have sizes 9, 10, 3, 3, 2, 2 and 3, so $\sum(t^3 - t)/48 = 37.375$ and $\operatorname{Var}(W^+) = 3094.875$, a standard deviation of 55.63;
- $z = (|299 - 280.5| - 0.5)/55.63 = 0.324$ and $p = 0.746$.
The exact conditional $p$-value, from the recursion over the observed midranks, is 0.748. Ignoring the tie correction changes $z$ only to 0.322.
With Pratt's treatment of zeros, the seven zeros take ranks 1 to 7 and the remaining ranks are shifted up. Then $W^+ = 404$ and $W^- = 388$, and $p = 0.919$.
All four rank-based analyses agree with the $t$-test that 40 adults cannot detect the difference. But they report it as weaker evidence than the $t$-test does ($p$ around 0.73 to 0.92 against 0.43). The reason is visible in the table. There are more negative differences than positive ones, but the positive ones include the largest: 14, 14, 14 and 22. The mean is pulled up by them; the counts of signs are not, and the ranks only partly.
That disagreement raises a question. If the tests can weigh the same sample so differently, are they testing the same hypothesis at all?
What each test tests
On symmetric data the three tests are three ways of asking whether the center is zero. On skewed or discrete data they ask three different questions, and the NHANES population of differences shows it. Among the 4,782 adults with a first and a third reading, the difference has
- mean 0.786 mm Hg;
- median exactly 0: 46.7% of adults have a positive difference, 15.8% a zero one and 37.5% a negative one, so neither side holds half;
- $P(D > 0 \mid D \ne 0) = 0.555$;
- for two adults chosen independently, $P(D_1 + D_2 > 0) = 0.521$ and $P(D_1 + D_2 < 0) = 0.374$.
Take the three null hypotheses in turn:
- The $t$-test's, $E(D) = 0$, is false: the mean is 0.79.
- The sign test's, $P(D > 0) = P(D < 0)$, is false too, even though the median is zero. The zeros that hold the median at 0 are exactly the observations the test discards.
- The signed-rank test's, symmetry about zero, is false as well.
What the signed-rank test responds to is the last line of the population summary, $P(D_1 + D_2 > 0)$. The signed-rank statistic is the number of positive pairwise averages. For nonzero differences without ties,
$$W^+ = \#\left\{(i, j): i \le j,\ \tfrac12(d_i + d_j) > 0\right\},$$
counting each difference with itself as well as with every other. To see why, take the difference of rank $r$. If it is positive, it gives a positive average with itself and with every difference of smaller absolute value, which is $r$ averages in all. If it is negative, it gives a negative average with all of them. Adding over the positive differences gives $\sum_{d_i>0} r_i$.
The averages $\tfrac12(d_i + d_j)$ are called Walsh averages. For drug 1 there are 45 of them, and 31 are positive, which is $W^+$. Under the null hypothesis half of these averages should be positive. So the test detects populations in which $P(D_1 + D_2 > 0)$ differs from $P(D_1 + D_2 < 0)$.
Check your understanding. The population median of the differences is exactly zero. If we draw larger and larger samples and apply the sign test at $\alpha = 0.05$, how often will it reject?
More and more often, as the table below shows: in about half of all samples of 400. Since all three nulls are false here, all three tests reject more often as the sample grows, but at different rates. The script drew 10,000 samples of each size from the population, for each of 30 seeds, and applied each test at $\alpha = 0.05$. The signed-rank test used the tie-corrected normal approximation and the sign test its exact distribution; the table gives medians over the seeds.
| Adults in the sample | Paired $t$ | Signed-rank | Sign |
|---|---|---|---|
| 25 | 0.098 | 0.091 | 0.048 |
| 50 | 0.159 | 0.149 | 0.085 |
| 100 | 0.278 | 0.259 | 0.143 |
| 200 | 0.504 | 0.469 | 0.271 |
| 400 | 0.800 | 0.764 | 0.506 |
The sign test rejects in half of the samples of 400 although the population median is exactly zero. That is not a malfunction. With continuous data the sign test is a test of the median. With discrete data it is a test of whether positive and negative differences are equally common, and here they are not. Suppose a report said "the median difference was significantly different from zero" on the strength of a sign test. It would be wrong about this population, while "positive differences were significantly more common than negative ones" would be right. The general lesson is that the choice among these tests is partly a choice of question. State the hypothesis in the terms of the test actually used.
The signed-rank test's symmetry assumption deserves the same care. It is not a technical condition to be checked and forgotten. If the differences are skewed, the signed-rank test can reject when the mean is zero, or fail to reject when the median is not, because it is testing neither. When the question is about the mean of a skewed variable, the $t$-test answers it more directly; its validity for the mean rests on the central limit theorem in moderate samples. A test on a transformed scale is the other option.
Suppose, then, that the differences are symmetric and all three tests are about the same center. Which one should we use?
Efficiency and robustness
The rank tests use less of the information in the data than the $t$-test does, and when the data are normal they pay for it in power. When the data are not normal they can be much more powerful. Both effects can be measured.
Asymptotic relative efficiency
The standard measure is the asymptotic relative efficiency (ARE) of one test to another. For alternatives close to the null hypothesis and large samples, it is the limiting ratio of the sample sizes the two tests need to reach the same power. An efficiency of 0.9 for test A relative to test B means that A needs about $1/0.9 = 1.11$ times as many observations as B to do as well. This article is the unit's home of the measure; the rank-sum and Kruskal–Wallis articles use it as defined here.
Consider a shift in location of a continuous distribution with density $f$ and variance $\sigma^2$, symmetric for the signed-rank and sign tests. The efficiencies relative to the $t$-test are
$$\text{ARE}(\text{signed-rank},\ t) = 12\,\sigma^2\left(\int f(x)^2\,dx\right)^2, \qquad \text{ARE}(\text{sign},\ t) = 4\,\sigma^2 f(0)^2,$$
where $f(0)$ is the density at the center of symmetry. These are the standard Pitman efficiencies.
Here is the idea behind them. A test is good at detecting a small shift if its statistic moves a lot when the center moves, compared with how much it varies by chance. The ratio of the two is the test's slope, and efficiency compares slopes.
The formal version. Suppose each test statistic is approximately normal near the null hypothesis. Then the efficiency of one test relative to another is the squared ratio of their slopes. A slope is the rate at which the statistic's mean moves with the shift, divided by its standard deviation, at the null (van der Vaart's Theorem 14.19, as presented in lecture notes by Duchi, 2021, and Bartlett, 2013). The three slopes are:
- The $t$-test: the slope is $1/\sigma$.
- The sign test: the slope is $2f(0)$. A shift $\Delta$ raises $P(D > 0)$ from $\tfrac12$ by about $f(0)\Delta$, and the proportion of plus signs has standard deviation $\tfrac12$ per observation. The squared ratio to the $t$-test's slope is $4\sigma^2 f(0)^2$.
- The signed-rank test: the slope is $\sqrt{12}\int f^2$. The Walsh-average form of $W^+$ in the previous section shows that $W^+/\binom{n}{2}$ estimates $P(D_1 + D_2 > 0)$. Its derivative in $\Delta$ at 0 is $2\int f(x)^2\,dx$, because by symmetry the density of $D_1 + D_2$ at 0 is $\int f(x)f(-x)\,dx = \int f^2$. Its null standard deviation is $1/\sqrt{3}$ per observation, from $\operatorname{Var}(W^+) \approx n^3/12$. The squared ratio to $1/\sigma$ is the first expression.
Fay and Proschan (2010, §5.3.4) give the same expression for the two-sample rank-sum test.
Now the check on the normal case. For the normal density with standard deviation $\sigma$, $\int f^2 = 1/(2\sigma\sqrt{\pi})$ and $f(0) = 1/(\sigma\sqrt{2\pi})$, so
$$\text{ARE}(\text{signed-rank},\ t) = 12\sigma^2 \cdot \frac{1}{4\pi\sigma^2} = \frac{3}{\pi} = 0.955, \qquad \text{ARE}(\text{sign},\ t) = 4\sigma^2\cdot\frac{1}{2\pi\sigma^2} = \frac{2}{\pi} = 0.637.$$
Under normality the signed-rank test needs about 5% more pairs than the $t$-test for the same power, and the sign test about 57% more. The second number is the price of the median met in the sampling distribution of the mean, whose variance for normal data is $\pi/2$ times that of the mean. Wilcoxon (1945) cited an efficiency of "63 percent" for the sign test from Cochran's work. He expected that his test, using "the magnitude of the differences as well as the signs", should do better, but wrote that its efficiency was "not known to me".
Check your understanding. The Laplace (double exponential) distribution has a sharp peak at its center and heavier tails than the normal. Which of the three tests would we expect to be most efficient for it?
The sign test, which scores 2.0 against the $t$-test. For heavier-tailed distributions the comparison reverses. The script computed the two expressions by numerical integration for five symmetric shapes:
| Shape of the differences | Signed-rank vs $t$ | Sign vs $t$ |
|---|---|---|
| Normal | 0.955 | 0.637 |
| Logistic | 1.097 | 0.822 |
| Laplace (double exponential) | 1.500 | 2.000 |
| $t$ with 3 degrees of freedom | 1.900 | 1.621 |
| Contaminated normal (10% with three times the SD) | 1.373 | 0.998 |
The signed-rank test loses at most a little when the data are normal, and gains a great deal when they have heavy tails or occasional wild values. The sign test is the best of the three for the Laplace distribution and the worst for the normal. The Laplace distribution's sharp peak at the center makes the sign of an observation unusually informative.
Power in small samples
Asymptotic efficiencies describe large samples and small effects. Do they hold with 20 pairs? The script checked by simulation, with this design:
- the shift was $0.660\,\sigma$, which gives the paired $t$-test power 0.80 for normal differences at $\alpha = 0.05$;
- each shape was scaled to have the same standard deviation;
- each test used its exact critical value: the signed-rank test rejects when $\min(W^+, W^-) \le 52$, an exact level of 0.0484, and the sign test when the smaller count is 5 or less, a level of 0.0414.
For each shape the script drew 10,000 samples for each of 30 seeds:
| Shape | Power, paired $t$ | Signed-rank | Sign | Level of $t$ when $H_0$ is true |
|---|---|---|---|---|
| Normal | 0.801 | 0.780 | 0.598 | 0.050 |
| Logistic | 0.803 | 0.815 | 0.689 | 0.049 |
| Laplace | 0.805 | 0.855 | 0.817 | 0.046 |
| $t$, 3 df | 0.854 | 0.929 | 0.895 | 0.043 |
| Contaminated normal | 0.811 | 0.879 | 0.779 | 0.045 |
The pattern follows the efficiencies. The signed-rank test gives up two percentage points of power under normality and gains five to eight under heavier tails. The rank tests' levels (0.048 and 0.041 for every shape, not shown in the table) do not depend on the shape at all, since their null distributions come from counting sign patterns. The $t$-test's level drifts slightly below 0.05 for heavy-tailed data, which makes it conservative rather than invalid. Its weakness with heavy tails is not that it rejects too often. It is that the standard deviation, inflated by the extreme values, swamps the shift the test is trying to detect.
When to use which
The trade-off is moderate in both directions, and a few points settle most cases.
- The paired $t$-test is the right tool for the mean difference, and with moderate samples it is valid for skewed data too. If the question is about the mean, a rank test answers a different question.
- The signed-rank test is a good default when outliers or heavy tails are plausible and the differences can be assumed roughly symmetric. Its cost under normality is about 5% of the sample size.
- The sign test needs the least: only the direction of each difference. It is the test to use when the data are signs (better or worse, preferred A or B), when the measurements are ordinal and differences of ranks mean nothing, or when a few differences are so extreme or so doubtful that even their ranks should not count.
- None of the three tests helps with dependence between pairs, a study design that confounds the comparison, or a sample too small to detect the effect of interest.
In information retrieval, Significance Testing applies both tests to per-query differences between search engines, where their results can disagree with the $t$-test's.
What we have learned
Under the null hypothesis each sign is a fair coin toss, so the sign test is an exact binomial test. Attaching the signs to the ranks of the sizes gives Wilcoxon's $W^+$, whose exact distribution comes from counting the $2^n$ equally likely sign patterns. Its mean and variance, reduced for ties, give a normal approximation that is already close at nine pairs. Under normality the signed-rank test needs about 5% more pairs than the $t$-test and the sign test about 57% more; for heavy tails both can need far fewer. But each test answers its own question, as the NHANES readings showed.
What if the two measurements come from different people, so that there are no pairs to subtract? The Wilcoxon rank-sum test ranks the two samples together and counts the same way, and the Kruskal–Wallis test extends it to several groups.
In Python
import numpy as np
from scipy import stats
drug1 = np.array([0.7, -1.6, -0.2, -1.2, -0.1, 3.4, 3.7, 0.8, 0.0, 2.0])
drug2 = np.array([1.9, 0.8, 1.1, 0.1, -0.1, 4.4, 5.5, 1.6, 4.6, 3.4])
d = np.round(drug2 - drug1, 1) # round so that equal differences tie exactly
print(stats.binomtest(9, 9).pvalue) # sign test: 9 of 9 nonzero positive, 0.0039
print(stats.wilcoxon(d).pvalue) # signed-rank, zero dropped, 0.0039
print(stats.wilcoxon(drug1)) # statistic 14 (= W-), p 0.359 (exact)
print(stats.wilcoxon(drug1, method='asymptotic', correction=True).pvalue) # 0.343
wilcoxon reports the smaller of $W^+$ and $W^-$ as its statistic, drops zeros by default (zero_method='wilcox'; 'pratt' gives Pratt's method) and applies the tie correction in its normal approximation. The rounding in the third line matters: computed in floating point, $1.9 - 0.7$ and $1.3$ may differ in the last digit, which would turn tied differences into untied ones with different ranks, a hazard the scipy documentation warns about. The outputs shown were checked with scipy 1.17.1, and every number in this article is computed by tools/experiments/applied-statistics/sign-and-signed-rank-tests.py in the site's repository, which computes the exact distributions itself.
References
- Cushny, A. R., and Peebles, A. R. (1905). The action of optical isomers. II. Hyoscines. The Journal of Physiology, 32(5–6), 501–510. record · 3 pages
- Bartlett, P. (2013). Lecture 24: relative efficiency of tests. Stat 210B, Theoretical Statistics, University of California, Berkeley. https://www.stat.berkeley.edu/~bartlett/courses/2013spring-stat210b/notes/24notes.pdf record
- Duchi, J. (2021). Asymptotic testing: basics and relative efficiencies (lecture slides 12). Stats 300B, Stanford University. https://web.stanford.edu/class/stats300b/Slides/12-testing-asymptotics.pdf record
- Fay, M. P., and Proschan, M. A. (2010). Wilcoxon–Mann–Whitney or t-test? On assumptions for hypothesis tests and multiple interpretations of decision rules. Statistics Surveys, 4, 1–39. record · 3 pages
- Fisher, R. A. (1925). Statistical Methods for Research Workers. Edinburgh: Oliver and Boyd. §24, Ex. 19. Online edition: Classics in the History of Psychology, York University. https://psychclassics.yorku.ca/Fisher/Methods/ record · 6 pages
- Fisher, R. A. (1935). The Design of Experiments. Edinburgh: Oliver and Boyd. Chapter III (§§14 to 17, Darwin's experiment on the growth of plants). record · 2 pages
- Kruskal, W. H., and Wallis, W. A. (1952). Use of ranks in one-criterion variance analysis. Journal of the American Statistical Association, 47(260), 583–621. record · 4 pages
- National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018: Demographic Variables and Sample Weights (DEMO_J) and Blood Pressure (BPX_J) data files. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 45 pages
- SciPy.
scipy.stats.wilcoxonandscipy.stats.binomtest. SciPy reference documentation. https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.wilcoxon.html record - Student (1908). The probable error of a mean. Biometrika, 6(1), 1–25. record · 11 pages
- Wilcoxon, F. (1945). Individual comparisons by ranking methods. Biometrics Bulletin, 1(6), 80–83. record · 3 pages