One-Way Analysis of Variance compared the systolic blood pressures of 10 adults from each of four age groups in the NHANES survey of 2017 to 2018. It found $F = 5.716$ on 3 and 36 degrees of freedom, $p = 0.0026$. The $F$ test assumes that each group is a sample from a normal population and that the populations share a variance. Blood pressure in these age groups is right-skewed, and its spread nearly doubles from the youngest group to the oldest. The analysis of variance is fairly tolerant of the first problem and, with equal group sizes, of the second. Could we compare the four groups without needing either assumption?
William Kruskal and W. Allen Wallis (1952) proposed doing it with ranks. Given several samples, rank all the observations together from 1 to $N$ and add up the ranks in each sample. Then ask whether the rank sums differ more than random assignment of the ranks would make them. Their statistic, $H$, is to several groups what the Wilcoxon rank-sum test is to two, and they show that for two groups it is that test. The only assumptions they list for its use are that "the observations are all independent, that all those within a given sample come from a single population, and that the $C$ populations are of approximately the same form".
First we derive $H$ as the analysis of variance of the ranks, with its correction for ties, and work the age-group example by hand. Then we check the chi-squared approximation against exact counts and simulation, and show that for two groups $H$ is the square of the rank-sum test's $z$. Finally we examine what the test detects, how it behaves when the groups' spreads differ, how much power it has against the $F$ test, and how to find which groups differ. The article assumes One-Way Analysis of Variance and The Wilcoxon Rank-Sum Test.
Notation
There are $k$ groups (Kruskal and Wallis write $C$), of sizes $n_1, \dots, n_k$, with $N = \sum n_i$ observations in all. All $N$ observations are ranked together from 1 (smallest) to $N$ (largest), tied values receiving the mean of the ranks they span. For group $i$,
- $R_i$ is the sum of its ranks, and $\bar{R}_i = R_i/n_i$ its mean rank;
- the overall mean rank is $\bar{R} = (N+1)/2$, since the ranks $1, \dots, N$ add up to $N(N+1)/2$.
The null hypothesis is that the $k$ samples come from the same population, $H_0: F_1 = F_2 = \cdots = F_k$, where $F_i$ is the distribution of the measurement in population $i$. The alternative, as for the analysis of variance, is that they do not all agree. What kind of disagreement the test detects is taken up below.
The statistic
Here is the plan of the derivation. Under the null hypothesis the ranks are a known population, the integers 1 to $N$, so we know in advance how much a group's mean rank varies by chance. $H$ is the analysis of variance's between-groups sum of squares computed on the ranks, measured against that known variation instead of an estimate.
The ranks as a population
If $H_0$ holds and the measurements are continuous, all $N$ observations are independent draws from one distribution. Every assignment of the ranks $1, \dots, N$ to the $N$ positions is then equally likely. The ranks of group $i$ are a random sample of $n_i$ drawn without replacement from the integers $1, \dots, N$, as for the rank-sum test. That population of integers has mean $(N+1)/2$ and variance
$$\sigma_r^2 = \frac{1}{N}\sum_{j=1}^N j^2 - \left(\frac{N+1}{2}\right)^2 = \frac{(N+1)(2N+1)}{6} - \frac{(N+1)^2}{4} = \frac{N^2 - 1}{12}.$$
The mean of a sample of $n_i$ drawn without replacement from a population of $N$ has the population mean as its mean. Its variance is $(\sigma_r^2/n_i)(N - n_i)/(N - 1)$, the usual variance of a mean reduced by the finite population correction. So
$$E(\bar{R}_i) = \frac{N+1}{2}, \qquad \operatorname{Var}(\bar{R}_i) = \frac{N^2 - 1}{12\,n_i}\cdot\frac{N - n_i}{N - 1} = \frac{(N+1)(N - n_i)}{12\,n_i},$$
which is Kruskal and Wallis's formula (3.1).
H as an analysis of variance of the ranks
The natural measure of how far the mean ranks are from what $H_0$ predicts is the between-groups sum of squares of the analysis of variance, computed on the ranks:
$$SSB_r = \sum_{i=1}^k n_i\left(\bar{R}_i - \frac{N+1}{2}\right)^2.$$
In the analysis of variance this sum is divided by an estimate of the variance computed from the data. Here no estimate is needed. The total variation of the ranks is known in advance, because the ranks are always the integers 1 to $N$. Their total sum of squares is
$$SST_r = \sum_{j=1}^N\left(j - \frac{N+1}{2}\right)^2 = N\sigma_r^2 = \frac{N(N^2-1)}{12},$$
and the Kruskal–Wallis statistic is
$$H = (N - 1)\,\frac{SSB_r}{SST_r} = \frac{12}{N(N+1)}\sum_{i=1}^k n_i\left(\bar{R}_i - \frac{N+1}{2}\right)^2.$$
It is $N - 1$ times the share of the variation in the ranks that lies between groups, the rank version of the $\eta^2$ of the analysis of variance. Expanding the square, using $\sum n_i \bar{R}_i = \sum R_i = N(N+1)/2$, gives the computing formula of Kruskal and Wallis (1952, eq. 1.2) and of the NIST/SEMATECH e-Handbook (§7.4.1):
$$H = \frac{12}{N(N+1)}\sum_{i=1}^k \frac{R_i^2}{n_i} - 3(N+1).$$
Large values of $H$ mean that the groups occupy different parts of the joint ranking, and only large values lead to rejection. How large is large?
The chi-squared approximation
Here is the idea: $H$ is close to a sum of squared standardized mean ranks, and such a sum is approximately chi-squared.
The formal version. Each term $n_i(\bar{R}_i - \bar{R})^2$ divided by $(N+1)N/12$ is close to the square of a standardized mean rank. By the formula for $\operatorname{Var}(\bar{R}_i)$, the standardized deviation is $(\bar{R}_i - \bar{R})/\sqrt{(N+1)(N-n_i)/(12 n_i)}$. $H$ weights the squares by $(N - n_i)/N$, which accounts for the constraint that the rank sums must add up to $N(N+1)/2$. For groups that are not too small, the mean ranks are approximately normal. $H$ is then a sum of $k$ squared approximately normal deviations subject to one linear constraint, so it is approximately chi-squared with $k - 1$ degrees of freedom.
Kruskal and Wallis (1952, eq. 1.5) give this reading of $H$, and the approximation is the one used in practice: reject $H_0$ at level $\alpha$ when $H$ exceeds the $1-\alpha$ quantile of $\chi^2_{k-1}$. The NIST handbook suggests it once every group has more than four observations. The scipy documentation gives at least five per group as a typical rule.
Ties
Tied values receive midranks. This leaves every group's expected mean rank unchanged but lowers the total sum of squares of the ranks. As the signed-rank article derives, replacing $t$ consecutive ranks by their mean reduces the sum of squared deviations by $(t^3 - t)/12$. With ties, then,
$$SST_r = \frac{N(N^2 - 1)}{12} - \frac{1}{12}\sum_{\text{tie groups}}(t^3 - t),$$
and $H = (N-1)SSB_r/SST_r$ becomes the no-ties value divided by
$$C = 1 - \frac{\sum (t^3 - t)}{N^3 - N},$$
which is Kruskal and Wallis's correction (their eq. 1.3 and 1.4). Since $C < 1$, the correction always increases $H$. They note that it is usually small: with ten or fewer samples, and not more than a quarter of the observations tied, it does not change a chi-squared probability of 0.01 or more by more than ten percent.
Worked example: blood pressure in four age groups
Back to the four age groups. The sample is the one of the one-way ANOVA article: 10 adults drawn at random from each of four NHANES age groups (the script's seed 1). Each adult's value is the mean of up to three systolic readings. Ranking all 40 values together gives:
| Age group | Ranks | $R_i$ | $\bar{R}_i$ |
|---|---|---|---|
| 18–29 | 2, 5, 6, 13, 13, 15.5, 21, 21, 23, 28 | 147.5 | 14.75 |
| 30–44 | 1, 3, 4, 15.5, 18, 25, 26.5, 29, 32, 33 | 187.0 | 18.70 |
| 45–59 | 7, 8, 9, 10.5, 18, 18, 21, 26.5, 34, 36 | 188.0 | 18.80 |
| 60 and over | 10.5, 13, 24, 30, 31, 35, 37, 38, 39, 40 | 297.5 | 29.75 |
The rank sums add up to $820 = 40 \cdot 41/2$, as they must, and the overall mean rank is 20.5. The three younger groups have mean ranks a little below it and the oldest group one far above.
Check your understanding. The $F$ test gave $p = 0.0026$. Ranking pulls in the oldest group's highest values. Will the Kruskal–Wallis $p$-value be smaller or larger than the $F$ test's?
Larger, as the three steps below show: about ten times larger, though still below 0.05.
Step 1, the statistic. With $n_i = 10$,
$$\sum\frac{R_i^2}{n_i} = \frac{147.5^2 + 187.0^2 + 188.0^2 + 297.5^2}{10} = 18{,}057.55,$$
$$H = \frac{12}{40 \cdot 41}(18{,}057.55) - 3(41) = 132.128 - 123 = 9.128.$$
Step 2, ties. The 40 values contain six groups of ties, three pairs and three triples (each value is the mean of readings recorded in steps of 2 mm Hg, so coincidences are common), and
$$C = 1 - \frac{3(2^3 - 2) + 3(3^3 - 3)}{40^3 - 40} = 1 - \frac{90}{63{,}960} = 0.998593, \qquad H = \frac{9.128}{0.998593} = 9.141.$$
As Kruskal and Wallis predicted, the correction hardly matters. As a check on the derivation, the between-groups and total sums of squares of the ranks are $SSB_r = 1247.55$ and $SST_r = 5322.5$ (it would be $40(40^2 - 1)/12 = 5330$ without ties). Then $(N-1)SSB_r/SST_r = 39 \times 1247.55/5322.5 = 9.141$, as before.
Step 3, the $p$-value. The critical value of $\chi^2_3$ at $\alpha = 0.05$ is 7.815, so $H_0$ is rejected. The approximate $p$-value is $P(\chi^2_3 \ge 9.141) = 0.0275$.
The exact distribution of $H$ for these data is the distribution over all $40!/(10!)^4 \approx 4.7 \times 10^{21}$ ways to split the 40 ranks into four groups of ten. That is too many to list, but a random sample of them estimates it. Over 30 seeds of 10,000 random splits, the proportion giving $H \ge 9.141$ had median 0.0223 (interquartile range 0.0210 to 0.0232). The chi-squared $p$-value is slightly larger than the permutation one, which anticipates the next section.
The classical $F$ test gave $p = 0.0026$ on the same data, ten times smaller. The ranks discard the information that the oldest group's highest values are far above the rest: 156 and 144.7 mm Hg count only as 40 and 39. In a sample that is not badly skewed, that information is genuine. The two tests agree that the groups differ.
The F test on the ranks
Since $H$ is a function of the between-groups share of the rank variation, so is the ordinary $F$ statistic computed on the ranks. With $SSW_r = SST_r - SSB_r$,
$$F_r = \frac{SSB_r/(k-1)}{SSW_r/(N-k)} = \frac{H/(k-1)}{(N - 1 - H)/(N - k)}.$$
For the example, $F_r = (9.141/3)/(29.859/36) = 3.674$, with $p = 0.0209$ from the $F_{3,36}$ distribution. Running the usual analysis of variance on the ranks is therefore a close relative of the Kruskal–Wallis test. It uses the same statistic, transformed, and refers it to a different approximate distribution.
The exact distribution and the approximation
How good is the chi-squared approximation? For small samples the exact null distribution of $H$ can be found by listing every assignment of the ranks to groups. Kruskal and Wallis (1952, §2.1) illustrated their test with artificial data on the daily output of three bottle-cap machines (samples of 5, 3 and 4 days), for which $H = 5.656$. They reported an exact probability of 0.049 for a value this large, against 0.059 from $\chi^2_2$.
The script listed all $12!/(5!\,3!\,4!) = 27{,}720$ assignments of the ranks 1 to 12 to groups of 5, 3 and 4. It found 1,348 with $H \ge 5.656$, and so reproduced their exact value, $1348/27{,}720 = 0.0486$. The chi-squared $p$-value is 0.0591, and the analysis of variance of the original numbers gives $p = 0.051$, as they also reported.
For larger designs simulation shows how good the approximation is. For four groups of 10 continuous observations from one population, 10,000 samples for each of 30 seeds gave:
- $P(H > 7.815) = 0.0439$ (interquartile range 0.0429 to 0.0457), against the nominal 0.05;
- a mean of $H$ of 2.996 over all 300,000 samples, matching both the exact null mean $k - 1 = 3$ and the chi-squared mean;
- a variance of 5.39, against 6 for $\chi^2_3$. Kruskal and Wallis (1952, eqs. 6.1 and 6.2) give the exact null mean and variance of $H$. For four groups of 10 the variance is 5.43, so the shortfall is a property of $H$, not of the simulation.
At these sample sizes the chi-squared approximation is somewhat conservative. Its $p$-values run a little high, and a test at the nominal 0.05 has an actual level of about 0.044. Software that can compute exact or permutation $p$-values removes the question. For three groups of up to five observations each, Kruskal and Wallis provided exact tables.
Two groups: the rank-sum test
For $k = 2$ the statistic reduces to the rank-sum test. Kruskal and Wallis (1952, §3.1) show that the standardized mean rank of one sample, squared, is $H$. The two mean ranks are tied together by $n_1\bar{R}_1 + n_2\bar{R}_2 = N(N+1)/2$, so only one is free, and
$$H = \frac{\left(W - n_1(N+1)/2\right)^2}{n_1 n_2 (N+1)/12} = z^2,$$
where $W = R_1$ is the Wilcoxon rank-sum statistic and $z$ its standardized value without continuity correction.
Check your understanding. For the 20 men and 20 women of the two-sample t-test article, the rank-sum test without continuity correction gave $z = 0.609$ and $p = 0.543$. What will the Kruskal–Wallis test give on the same two groups?
$H = z^2 = 0.371$, with the same $p$. The script found $H = 0.37074$ (tie-corrected) and $z = 0.60888$, $z^2 = 0.37074$, with $p = 0.543$ from $\chi^2_1$. The relationship parallels $F = t^2$ for the analysis of variance of two groups.
What the test detects
The null hypothesis of the Kruskal–Wallis test is that all $k$ populations are identical. It is a valid test of that hypothesis whatever their common shape: under it, every assignment of ranks is equally likely, and that is all the exact distribution uses. When it rejects, it rejects because the groups' mean ranks differ. A group's mean rank depends on the probability that its observations exceed those of the other groups. In the analysis of variance's terms, the test responds to differences in location as measured by these probabilities, not by means.
Kruskal and Wallis listed among the assumptions for its use that the populations "are of approximately the same form". They described one of its most important applications as "detecting differences among the population means". Under that assumption, the populations differ, if at all, by a shift. Their means, medians and every other measure of location then differ by the same amounts, and the test is a test of each of them. The NIST handbook (§7.4.1) accordingly describes the test as comparing populations "at least as far as their central tendencies, or medians, are concerned".
When the shapes differ, it is a test of neither means nor medians in general, as the rank-sum article shows by example for two groups. The conclusion should then be stated in terms of the groups' tendency to have larger values.
When spreads differ
A rank test is sometimes recommended as a remedy for unequal variances. It is not one. The script repeated the experiment of the one-way ANOVA article: four normal populations with equal means and the standard deviations of the NHANES age groups (11.32, 15.36, 18.91 and 21.02 mm Hg). The population means and medians are equal, but the spreads are not.
Check your understanding. With group sizes 20, 15, 10 and 5, the smallest group is the most variable. Will the Kruskal–Wallis test reject too often or too rarely?
Too often, like the classical $F$ test, though by less. The proportion of samples in which each test rejected at $\alpha = 0.05$ (10,000 samples for each of 30 seeds, medians):
| Group sizes (from the least to the most variable group) | Classical $F$ | Welch's ANOVA | Kruskal–Wallis |
|---|---|---|---|
| 10, 10, 10, 10 | 0.057 | 0.051 | 0.049 |
| 20, 15, 10, 5 (smallest group most variable) | 0.116 | 0.059 | 0.075 |
| 5, 10, 15, 20 (largest group most variable) | 0.026 | 0.053 | 0.029 |
| 20, 15, 10, 5, all spreads equal | 0.049 | 0.057 | 0.044 |
With equal group sizes the Kruskal–Wallis test holds its level. With unequal sizes it fails in the same direction as the classical $F$ test, if by less: too often when the smallest group is the most variable, too rarely when the largest group is. Its null hypothesis, identical populations, is false here, since the spreads differ, and a test of identical populations is entitled to reject. But it is not a test that the centers differ. If that is the question, Welch's analysis of variance answers it better.
Power
Against the $F$ test, the asymptotic relative efficiency of the Kruskal–Wallis test behaves like that of the rank-sum test against the $t$-test. The script compared the three tests in two settings:
- four groups of 10 in a shift model, with group means $0, d, 2d, 3d$ standard deviations apart and $d = 0.493$ chosen to give the $F$ test power 0.80 for normal data;
- samples from the four NHANES age groups themselves.
| Populations | Classical $F$ | Welch's ANOVA | Kruskal–Wallis |
|---|---|---|---|
| Normal, shift model | 0.800 | 0.762 | 0.762 |
| $t$ with 3 df, shift model | 0.846 | 0.879 | 0.941 |
| Exponential, shift model | 0.805 | 0.818 | 0.924 |
| NHANES age groups, 10 per group | 0.770 | 0.765 | 0.760 |
| NHANES age groups, 5 per group | 0.390 | 0.326 | 0.349 |
For normal data the Kruskal–Wallis test loses about four percentage points of power at this size. Part of the loss comes from its chi-squared critical value being conservative. For heavy-tailed or strongly skewed data it gains ten or more. For the NHANES age groups, moderately skewed with unequal spreads, the three tests have nearly the same power with 10 per group.
Which groups differ
A significant $H$, like a significant $F$, says only that the groups are not all alike. Which ones differ? Dunn (1964) proposed comparing the groups' mean ranks from the joint ranking. Under $H_0$, the difference between the mean ranks of groups $i$ and $j$ has mean zero. From the variances and covariances of mean ranks drawn without replacement, its variance is
$$\operatorname{Var}(\bar{R}_i - \bar{R}_j) = \left[\frac{N(N+1)}{12} - \frac{\sum(t^3 - t)}{12(N-1)}\right]\left(\frac{1}{n_i} + \frac{1}{n_j}\right),$$
the bracket being $N/(N-1)$ times the variance of the ranks, reduced for ties (Dunn's equation 3). Each difference divided by its standard deviation is compared with the standard normal. To keep the family-wise error rate at $\alpha$ over $p$ comparisons, Dunn used the quantile $z_{1-\alpha/(2p)}$: a Bonferroni correction. For the worked sample the standard deviation of a difference is $\sqrt{(40 \cdot 41/12)(0.998593)(0.2)} = 5.224$, and the six comparisons are:
| Comparison | Difference in mean ranks | $z$ | $p$ | Bonferroni ($\times 6$) |
|---|---|---|---|---|
| 18–29 vs 30–44 | 3.95 | 0.756 | 0.450 | 1 |
| 18–29 vs 45–59 | 4.05 | 0.775 | 0.438 | 1 |
| 18–29 vs 60+ | 15.00 | 2.871 | 0.0041 | 0.025 |
| 30–44 vs 45–59 | 0.10 | 0.019 | 0.985 | 1 |
| 30–44 vs 60+ | 11.05 | 2.115 | 0.034 | 0.207 |
| 45–59 vs 60+ | 10.95 | 2.096 | 0.036 | 0.217 |
Only the youngest and oldest groups are shown to differ. Holm's procedure, which the Bonferroni article describes, gives the same conclusion (adjusted $p$-values 0.025, then 0.172 for the next two).
Dunn also discussed the alternative of ranking each pair of groups separately, which amounts to a rank-sum test for each pair. A separate ranking uses only the two groups being compared, while the joint ranking places every group on the same scale. On the worked sample the separate exact rank-sum tests give:
- $p = 0.0047$ for the youngest against the oldest;
- 0.043 and 0.025 for the two middle groups against the oldest;
- 0.40, 0.47 and 0.87 for the rest.
After Holm's adjustment only the first, at 0.028, stays below 0.05. With either method, and with these ten adults per group, the evidence separates the oldest group from the youngest and cannot place the two middle groups.
What we have learned
Ranking all the groups together and running the analysis of variance on the ranks gives Kruskal and Wallis's $H$. Because the ranks are always the integers 1 to $N$, their total variation is known, and no variance needs estimating. $H$ is approximately chi-squared with $k - 1$ degrees of freedom, a little conservatively for four groups of 10. For two groups it is the square of the rank-sum test's $z$. On the four age groups it gave $H = 9.141$ and $p = 0.0275$, and Dunn's comparisons separated only the youngest and oldest groups. (The $t$-based procedures of the Bonferroni article placed the oldest group above both the 18 to 29 and the 30 to 44 groups.) Like the rank-sum test, it tests whether values in some groups tend to be larger, not whether the means or medians differ, and it does not repair unequal spreads.
The whole unit has rested on one device: under the null hypothesis, signs are fair coin tosses and ranks are a random arrangement, so null distributions come from counting. The sign test was the simplest case, a binomial count of plus signs with $p = \tfrac12$. The next unit, Inference on Proportions, takes up counts of yes-or-no outcomes with any probability. It begins with the normal approximation to the binomial, the same approximation the sign test used for large samples.
In Python
from scipy import stats
# the worked sample of the one-way ANOVA article: 10 adults from each NHANES age group (mm Hg)
g1 = [114.7, 110.7, 110.0, 114.0, 118.7, 114.0, 96.7, 100.0, 110.0, 94.0] # 18-29
g2 = [116.0, 116.7, 131.3, 110.7, 112.7, 88.7, 96.0, 121.3, 95.0, 127.3] # 30-44
g3 = [108.0, 112.7, 137.0, 103.3, 116.7, 114.0, 108.7, 112.7, 106.0, 133.3] # 45-59
g4 = [144.7, 156.0, 108.7, 134.0, 110.0, 122.7, 144.0, 126.0, 142.0, 115.3] # 60+
print(stats.kruskal(g1, g2, g3, g4)) # H = 9.141, p = 0.0275 (tie-corrected)
print(stats.mannwhitneyu(g1, g4)) # one pairwise follow-up: U = 14, p = 0.0072
Rounding the values to 0.1 mm Hg keeps the same ranks and ties, so kruskal reproduces $H = 9.141$. With ties present mannwhitneyu uses the normal approximation, which gives 0.0072 for the youngest against the oldest group; the exact conditional value in the text, 0.0047, comes from the script. Dunn's comparisons are not in scipy and take a few lines from the mean ranks and the variance above. The outputs shown were checked with scipy 1.17.1, and every number in this article is computed by tools/experiments/applied-statistics/kruskal-wallis-test.py in the site's repository.
References
- Dunn, O. J. (1964). Multiple comparisons using rank sums. Technometrics, 6(3), 241–252. record · 2 pages
- Kruskal, W. H., and Wallis, W. A. (1952). Use of ranks in one-criterion variance analysis. Journal of the American Statistical Association, 47(260), 583–621. record · 4 pages
- National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018: Demographic Variables and Sample Weights (DEMO_J) and Blood Pressure (BPX_J) data files. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 45 pages
- NIST/SEMATECH. e-Handbook of Statistical Methods, §7.4.1 (How can we compare several populations with unknown distributions (the Kruskal-Wallis test)?). National Institute of Standards and Technology. https://www.itl.nist.gov/div898/handbook/ record · 30 pages
- SciPy.
scipy.stats.kruskal. SciPy reference documentation. https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.kruskal.html record