The Wilcoxon Rank-Sum Test is often described as a nonparametric test of whether two medians are equal. Is that what a rejection by the test shows? Only under a shift model. In general a rejection shows that $\theta = P(X > Y) + \tfrac12 P(X = Y)$ differs from one half, that is, that values in one group tend to be larger than values in the other. Two populations with the same median can have $\theta \ne \tfrac12$, and the test then rejects more and more often as the samples grow.
This article establishes what the test examines. It was part of the rank-sum article, which keeps the statistic, its exact and approximate null distributions and its power.
First we restate the setting and the convention for a shift. Then we prove what the test tests under a shift model, and show by simulation what it does without one: with equal medians and different shapes, and with equal centers and different spreads. Finally two worked problems give the Hodges–Lehmann estimate and interval for a shift, and the variance of $U$ when the two distributions differ. The article assumes The Wilcoxon Rank-Sum Test.
The setting and the shift convention
The setting is that of the rank-sum article. Two independent random samples are drawn: $x_1, \dots, x_{n_1}$ from a population with distribution function $F$, and $y_1, \dots, y_{n_2}$ from one with distribution function $G$. There are $N = n_1 + n_2$ observations in all. The Mann–Whitney statistic $U$ is the number of the $n_1 n_2$ pairs, one observation from each group, in which the $x$ is the larger, with ties counted as one half. The rank-sum article shows that $U/(n_1 n_2)$ is an unbiased estimate of
$$\theta = P(X > Y) + \tfrac12 P(X = Y),$$
and that $\theta = \tfrac12$ under the null hypothesis $H_0: F = G$.
A shift in location is the simplest case: $X$ has the distribution of $Y + \Delta$ for a constant $\Delta$. In terms of the distribution functions, $F(a) = P(Y + \Delta \le a) = G(a - \Delta)$, or equivalently
$$G(a) = F(a + \Delta).$$
This article keeps that convention throughout: $\Delta$ is the amount by which the first population lies above the second. With $\Delta > 0$, $P(X > a) = P(Y > a - \Delta) \ge P(Y > a)$ for every $a$, so $X$ is stochastically larger than $Y$.
The running example is the rank-sum article's 8 men and 8 women, drawn at random from the NHANES adults. Their systolic pressures, in mm Hg, are
- men: 140.7, 109.3, 124.7, 109.3, 143.3, 170.7, 124.0, 124.7;
- women: 115.3, 137.3, 140.0, 107.3, 119.3, 121.3, 112.7, 108.0.
For them $U = 46$ of 64 pairs, and the exact two-sided $p$-value is 0.161. For two samples of 8, the exact two-sided test at $\alpha = 0.05$ rejects when $\min(U, 64 - U) \le 13$, with exact level 0.0499. Every number in this article is computed by tools/experiments/applied-statistics/wilcoxon-rank-sum-test.py in the site's repository, the script of the rank-sum article.
What the test tests
The rank-sum test is often described as a test of whether two medians are equal. That is correct under an assumption that is easy to forget, and wrong without it.
Under a shift model the first population is the second moved up by a constant $\Delta$: $X$ has the distribution of $Y + \Delta$, that is, $G(a) = F(a + \Delta)$, as in the section on the setting. Then the median of $X$ is the median of $Y$ plus $\Delta$, and the same holds for the means. $\theta$ is above, at or below one half exactly as $\Delta$ is above, at or below zero.
Proof. Let $Y$ and $Y'$ be independent draws from the second population, so that $Y' + \Delta$ is a draw from the first, and put $V = Y - Y'$. Then $V$ is symmetric about 0, and
$$\theta(\Delta) = P(Y' + \Delta > Y) + \tfrac12 P(Y' + \Delta = Y) = P(V < \Delta) + \tfrac12 P(V = \Delta).$$
At $\Delta = 0$ this is $P(V < 0) + \tfrac12 P(V = 0) = \tfrac12$, by the symmetry of $V$. For $\Delta > 0$, subtract:
$$\theta(\Delta) - \tfrac12 = \tfrac12 P(V = 0) + P(0 < V < \Delta) + \tfrac12 P(V = \Delta) \ \ge\ \tfrac12 P(|V| < \Delta),$$
since by symmetry $P(|V| < \Delta) = P(V = 0) + 2P(0 < V < \Delta)$. The last probability is positive. To see it, cut the line into intervals of length $\Delta$, closed on the left and open on the right. At least one of them, say $I$, has $P(Y \in I) > 0$, and whenever $Y$ and $Y'$ both fall in $I$ they differ by less than $\Delta$. So $P(|V| < \Delta) \ge P(Y \in I)^2 > 0$, and $\theta(\Delta) > \tfrac12$. For a negative shift, the symmetry of $V$ gives
$$\theta(-\Delta) = P(V < -\Delta) + \tfrac12 P(V = -\Delta) = P(V > \Delta) + \tfrac12 P(V = \Delta) = 1 - \theta(\Delta) < \tfrac12.$$
The proof holds for any distribution, continuous or not. $\square$
The test of $F = G$ is then a test of $\Delta = 0$, which is at once a test of equal medians and of equal means. This is the setting in which the test was derived, and in which its efficiency is computed (Asymptotic Relative Efficiency of Rank Tests).
Without a shift model the test responds to $\theta$. Mann and Whitney (1947) proved that the test is consistent against every alternative in which one variable is stochastically larger than the other: its power tends to 1 as both samples grow. The underlying reason is that $U/(n_1 n_2)$ estimates $\theta$. Whenever $\theta \ne \tfrac12$, the statistic drifts away from its null mean faster than its standard deviation grows. For continuous data the second worked problem makes this exact. The mean of $U$ differs from its null value by $n_1 n_2\,|\theta - \tfrac12|$. Its variance is at most $n_1 n_2\,(N - 1)$, because in the variance formula $\theta(1-\theta)$, $\xi_x$ and $\xi_y$ are each at most 1, so the bracket is at most $1 + (n_2 - 1) + (n_1 - 1)$. With both samples a fixed share of $N$, the first grows like $N^2$ and the standard deviation no faster than $N^{3/2}$. The test rejects when $|U - n_1 n_2/2|$ exceeds about $1.96$ null standard deviations, a distance of order $N^{3/2}$. The true mean lies a distance of order $N^2$ from $n_1 n_2/2$. For large $N$, then, $U$ can fall in the acceptance region only by lying a distance of order $N^2$ from its own mean. By Chebyshev's inequality (Standard Scores, Chebyshev's Inequality and Grouped Data) the probability of that is at most the variance divided by the squared distance, a constant times $N^3/N^4$, which tends to 0. So the power tends to 1. Fay and Proschan (2010) list this among several interpretations of the same decision rule: under the assumption that $F = G$ when the null holds, rejecting means $\theta \ne \tfrac12$. Medians do not enter. Two consequences follow, and the script checked both by simulation (10,000 pairs of samples for each of 30 seeds; medians of the rejection rates).
Equal medians, different shapes. Take an exponential distribution shifted to have median 0 ($X = E - \ln 2$, with $E$ exponential with mean 1) and a standard normal $Y$. Both have median 0.
Example (equal medians, and a test that rejects). As the samples grow, the rank-sum test rejects the equality of these two populations more and more often, not about 5% of the time. $P(X > Y) = 0.557$, because the exponential's long right tail puts much of its probability above the normal's typical values, while its left side is cut off at $-0.69$. The rank-sum test detects that difference. It rejected in 9.4% of samples with 20 per group, 16.6% with 50 and 50.5% with 200. Each rejection is a correct statement that $\theta \ne \tfrac12$, and an incorrect one if reported as "the medians differ".
The NHANES populations give the converse. Shifting every woman's value up by 3.33 mm Hg makes the women's median equal the men's, 124.67 mm Hg. For these two populations $\theta = 0.496$, very close to one half. The rank-sum test rejected in 5.0%, 5.1% and 5.2% of samples of 50, 200 and 800 per group: it sees no difference, correctly. Welch's $t$-test compares means, 127.41 against 128.70 after the shift. It rejected in 6.0%, 9.5% and 25.1%, also correctly, because the means differ. The two tests are both valid; they answer different questions and so give different answers.
Equal centers, different spreads. If the two populations are normal with the same mean but standard deviations 1 and 3, then by symmetry $\theta = \tfrac12$ exactly, yet $F \ne G$. The rank-sum test is not a valid test of $\theta = \tfrac12$ here, as the simulation shows.
Example (10 observations at SD 1 and 40 at SD 3). With 10 observations at SD 1 and 40 at SD 3, the rank-sum test rejects too rarely, in 0.9% of samples, like the pooled $t$-test. The analysis of variance behaved the same way: it rejected too rarely when the largest groups had the largest spreads. The rates for three pairs of sample sizes:
| Sizes (SD 1, SD 3) | Rank-sum | Welch $t$ | Pooled $t$ |
|---|---|---|---|
| 25 and 25 | 0.065 | 0.050 | 0.053 |
| 10 and 40 | 0.009 | 0.050 | 0.001 |
| 40 and 10 | 0.135 | 0.051 | 0.257 |
The variance formula for $U$ assumed $F = G$. When the spreads differ, the true variance of $U$ is different. The second worked problem derives the variance of $U$ for any two continuous distributions and evaluates it for this example. With 10 observations at SD 1 and 40 at SD 3 the true variance is 0.58 times the null formula, with 40 and 10 it is 1.73 times, and with 25 and 25 it is 1.16 times. So in this example the variance is larger than the formula assumes when the more variable population has the smaller sample (and, by less, when the samples are equal), and smaller when it has the larger one. Those three ratios predict rejection rates of 0.010, 0.136 and 0.068, close to the simulated 0.009, 0.135 and 0.065. The test's level moves away from $\alpha$ in the same direction as the pooled $t$-test's, if less extremely. This is the Behrens–Fisher problem of the article on Welch's t-test in rank form. Fay and Proschan (2010, §5.1.2) call it the nonparametric Behrens–Fisher problem. They note that the rank-sum test is not valid for it, and describe a test with a variance estimate that does not assume $F = G$ (Brunner and Munzel's).
The practical point is that switching from the $t$-test to the rank-sum test does not protect against unequal variances. With equal group sizes the distortion is modest (0.065 here); with unequal sizes it can be serious.
The summary is a rule for reporting:
- Under a shift model, or when the populations are thought to have the same shape, the test compares locations and may be reported as comparing medians (or means).
- Otherwise it should be reported as showing, or failing to show, that values in one group tend to be larger than in the other, with $U/(n_1 n_2)$ as the estimate of how often.
In the rank-sum article's sample of 10 older and 30 younger adults, "older adults had higher blood pressure than younger adults in an estimated 82% of pairs" is both accurate and informative. Under a shift model, the median of the $n_1 n_2$ differences $x_i - y_j$ estimates the shift $\Delta$. Inverting the test gives a confidence interval for it, a construction due to Hodges and Lehmann (Fay and Proschan, 2010, §5.1.1). The first worked problem derives the interval and computes it for the 8 men and 8 women.
Worked problems
The two problems are posed here, on the 8 men and 8 women and on this article's simulation of unequal spreads; none is taken from the course text. The script wilcoxon-rank-sum-test.py named above checks every number.
Problem 1 (an estimate and an interval for the shift). Assume a shift model for the 8 men and 8 women of the section on the setting. Estimate the shift $\Delta$ by which men's pressures exceed women's, and find the interval of shifts that the exact two-sided rank-sum test at level 0.0499 does not reject.
Solution. The estimate is 7.3 mm Hg, and the interval runs from $-10.0$ to $28.0$ mm Hg.
To test a proposed shift $\Delta_0$, subtract it from every man's value. Under the shift model with $\Delta = \Delta_0$, the values $x_i - \Delta_0$ and $y_j$ come from one distribution, so the rank-sum test applies to them. Its statistic is
$$U(\Delta_0) = \#\{(i, j): x_i - \Delta_0 > y_j\} = \#\{(i, j): x_i - y_j > \Delta_0\},$$
the number of the 64 differences $x_i - y_j$ that exceed $\Delta_0$. (A proposed shift equal to one of the differences creates a tie. The counts below are for shifts that equal none of the differences, which is enough to locate the ends of the interval.) Order the differences, $D_{(1)} \le D_{(2)} \le \cdots \le D_{(64)}$. As $\Delta_0$ grows, $U(\Delta_0)$ steps down from 64 to 0.
The estimate. The null mean of $U$ is 32. The proposed shift that makes $U(\Delta_0)$ equal 32, and so makes the two samples look most alike, lies between $D_{(32)}$ and $D_{(33)}$. That is the median of the 64 differences. Here $D_{(32)} = 6.0$ and $D_{(33)} = 8.67$ (8.7 from the rounded values), so the estimate is their midpoint, 7.3 mm Hg. The difference of the two sample means is larger, 10.7 mm Hg.
The interval. For two samples of 8 the exact test rejects when $\min(U, 64 - U) \le 13$, with level 0.0499. So $\Delta_0$ is not rejected exactly when $14 \le U(\Delta_0) \le 50$. At least 14 differences exceed $\Delta_0$ exactly when $\Delta_0 < D_{(51)}$, the 14th largest. At most 50 exceed it exactly when at least 14 lie below it, that is, when $\Delta_0 > D_{(14)}$, the 14th smallest. The interval is therefore
$$D_{(14)} < \Delta < D_{(51)}: \qquad -10.0 < \Delta < 28.0 \ \text{mm Hg},$$
with confidence level $1 - 0.0499 = 95.0\%$ under the shift model. It contains 0, as it must, since the test of $\Delta_0 = 0$ gave $p = 0.161$. The interval is wide for the same reason the test did not reject: 16 people say little about a shift of a few mm Hg. The differences were computed from the unrounded pressures, each a mean of three readings; from the values rounded to 0.1 a reader will get the same results to within 0.1. As with the $p$-value 0.161, the level uses the distribution of $U$ without ties.
Problem 2 (the variance of U when the distributions differ). Derive the variance of $U$ for two samples from any two continuous distributions. Check that it reduces to $n_1 n_2 (N+1)/12$ when the distributions are the same. Evaluate it when $X$ is normal with standard deviation 1 and $Y$ is normal with the same mean and standard deviation 3, for the three pairs of sample sizes in the article's table, and predict the rejection rates.
Solution. The variance is
$$\operatorname{Var}(U) = n_1 n_2\Big[\theta(1 - \theta) + (n_2 - 1)\,\xi_x + (n_1 - 1)\,\xi_y\Big],$$
where $\xi_x = P(X > Y,\ X > Y') - \theta^2$ and $\xi_y = P(X > Y,\ X' > Y) - \theta^2$, with $X, X'$ independent draws from the first distribution and $Y, Y'$ from the second.
Derivation. Write $U = \sum_i \sum_j I_{ij}$, where $I_{ij} = 1$ if $x_i > y_j$ and 0 otherwise. The data are continuous, so there are no ties. The variance of a sum is the sum of the covariances of all ordered pairs of terms, as Random Variables, Expectation and Variance proves. The pairs of terms $(I_{ij}, I_{kl})$ are of four kinds:
- The same term ($i = k$, $j = l$): $n_1 n_2$ pairs, each contributing $\operatorname{Var}(I_{ij}) = \theta(1 - \theta)$, since $I_{ij}$ is a Bernoulli variable with probability $\theta$.
- The same $x$, different $y$'s ($i = k$, $j \ne l$): $n_1 n_2 (n_2 - 1)$ pairs, each with covariance $E(I_{ij} I_{il}) - \theta^2 = P(X > Y,\ X > Y') - \theta^2 = \xi_x$.
- The same $y$, different $x$'s ($i \ne k$, $j = l$): $n_1 n_2 (n_1 - 1)$ pairs, each with covariance $P(X > Y,\ X' > Y) - \theta^2 = \xi_y$.
- Nothing in common ($i \ne k$, $j \ne l$): the two terms involve four different observations, so they are independent and their covariance is 0.
Adding the four kinds gives the formula.
Identical distributions. Then $\theta = \tfrac12$. The event $X > Y,\ X > Y'$ says that $X$ is the largest of three independent draws from one continuous distribution, which has probability $\tfrac13$ because each of the three is equally likely to be the largest. Likewise $X > Y,\ X' > Y$ says that $Y$ is the smallest of three, with probability $\tfrac13$. So $\xi_x = \xi_y = \tfrac13 - \tfrac14 = \tfrac1{12}$, and
$$\operatorname{Var}(U) = n_1 n_2\left[\frac14 + \frac{(n_2 - 1) + (n_1 - 1)}{12}\right] = n_1 n_2 \cdot \frac{3 + N - 2}{12} = \frac{n_1 n_2 (N + 1)}{12}.$$
Standard deviations 1 and 3. By symmetry $\theta = \tfrac12$. Conditioning on the shared observation, $P(X > Y,\ X > Y') = E\big[P(Y < X \mid X)^2\big] = E\big[\Phi(X/3)^2\big]$, and $P(X > Y,\ X' > Y) = E\big[(1 - \Phi(Y))^2\big]$, where $\Phi$ is the standard normal distribution function. The script evaluated both expectations by numerical integration: 0.26594 and 0.42822. So
$$\xi_x = 0.01594, \qquad \xi_y = 0.17822.$$
The second is far larger. Two different $x$'s compared with the same $y$ usually agree, because a $y$ from the wide distribution tends to lie beyond both of them. The first is small: an $x$ from the narrow distribution near the center tells little about whether two different $y$'s lie below it.
| Sizes (SD 1, SD 3) | True $\operatorname{Var}(U)$ | Null formula | Ratio | Predicted rejection rate | Simulated |
|---|---|---|---|---|---|
| 25 and 25 | 3,068.6 | 2,656.25 | 1.155 | 0.068 | 0.065 |
| 10 and 40 | 990.3 | 1,700 | 0.583 | 0.010 | 0.009 |
| 40 and 10 | 2,937.6 | 1,700 | 1.728 | 0.136 | 0.135 |
For example, with 10 and 40, $\operatorname{Var}(U) = 400\,[0.25 + 39(0.01594) + 9(0.17822)] = 400 \times 2.4757 = 990.3$, against $400 \cdot 51/12 = 1{,}700$ from the null formula. The test divides by a standard deviation that is too large by the factor $1/\sqrt{0.583}$. If $U$ is approximately normal with its true variance, the test rejects when a standard normal variable exceeds $1.96/\sqrt{0.583} = 2.57$ in absolute value, which has probability 0.010. The other rows are computed the same way from their ratios: $1.96/\sqrt{1.155} = 1.82$ gives 0.068, and $1.96/\sqrt{1.728} = 1.49$ gives 0.136. The predictions agree with the simulated rates to within 0.004. The large covariance $\xi_y$ is multiplied by $n_1 - 1$, the size of the sample from the narrow distribution, less one. That is why the variance, and with it the rejection rate, is largest when the narrow population has the large sample and the wide population the small one.
What we have learned
A rejection by the rank-sum test means that $\theta = P(X > Y) + \tfrac12 P(X = Y)$ differs from one half. Under a shift model, $G(a) = F(a + \Delta)$, that is the same as $\Delta \ne 0$, so the test then compares medians and means, and the median of the differences $x_i - y_j$ estimates the shift. Without a shift model it does not compare medians: two populations with equal medians and different shapes are told apart more and more often as the samples grow. The test does not repair unequal spreads either. With equal centers, unequal spreads and unequal group sizes its level moves away from $\alpha$ in the same direction as the pooled $t$-test's, as the variance of $U$ under unequal distributions predicts.
When the shift model does hold, the remaining matter is power, measured in The Wilcoxon Rank-Sum Test and in Asymptotic Relative Efficiency of Rank Tests. For more than two groups, the Kruskal–Wallis test has the same reading: it tests whether values in some groups tend to be larger.
References
- Fay, M. P., and Proschan, M. A. (2010). Wilcoxon–Mann–Whitney or t-test? On assumptions for hypothesis tests and multiple interpretations of decision rules. Statistics Surveys, 4, 1–39. record · 4 pages
- Mann, H. B., and Whitney, D. R. (1947). On a test of whether one of two random variables is stochastically larger than the other. The Annals of Mathematical Statistics, 18(1), 50–60. record · 3 pages
- National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018: Demographic Variables and Sample Weights (DEMO_J) and Blood Pressure (BPX_J) data files. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 52 pages