Multiple Comparisons and the Bonferroni Correction

Finding which group means differ after an analysis of variance without letting false alarms multiply. The family-wise error rate of all pairwise t-tests measured by simulation, the Bonferroni correction proved from the union bound and worked on NHANES blood pressures in four age groups, Holm's step-down improvement with its proof, Tukey's honestly significant difference from the studentized range, simultaneous confidence intervals, Scheffe's method, why a significant F test does not protect later comparisons, and what the corrections cost in power.

The one-way analysis of variance of systolic blood pressure in four NHANES age groups, 10 adults from each, gave $F = 5.716$ on 3 and 36 degrees of freedom, $p = 0.0026$. The four population means are not all equal, and that is all the $F$ test says. Is the oldest group different from each of the others? Are the three younger groups different from each other? Four groups make six pairs, and each pair can be tested. But six tests, each with a 5% chance of a false alarm, have together a much larger chance of producing at least one. How do we test all six pairs without letting the false alarms multiply?

Multiple comparison procedures answer that question. They control the error rate of the whole set of comparisons rather than of each one. This article is the course's home of that error rate, the family-wise error rate, and of the Bonferroni correction that controls it.

First we measure the problem by simulation. Then we derive three procedures in order of increasing refinement:

  • the Bonferroni correction, which the course uses and which rests on a single inequality;
  • Holm's step-down version of it, which rejects everything Bonferroni's does and sometimes more;
  • Tukey's procedure, built on the exact distribution of the largest difference among the means.

All three are worked on the same NHANES sample, with the simultaneous confidence intervals each one gives. Finally we ask whether a significant $F$ test makes the comparisons that follow it safe (it does not), and what the protection costs in power. The article assumes One-Way Analysis of Variance, whose worked sample and within-groups variance it reuses.

The problem

Back to the four age groups. The worked sample from One-Way Analysis of Variance has these group means, with the pooled within-groups variance $s_W^2 = 166.20$ on $n - k = 36$ degrees of freedom:

Age group $n_i$ $\bar{x}_i$ (mm Hg)
18–29 10 108.267
30–44 10 111.567
45–59 10 115.233
60 and over 10 130.333

To compare groups $i$ and $j$, the course uses the two-sample $t$ statistic with the within-groups variance in place of the two-group pooled variance:

$$t_{ij} = \frac{\bar{x}_i - \bar{x}_j}{\sqrt{s_W^2\left(\dfrac{1}{n_i} + \dfrac{1}{n_j}\right)}},$$

referred to a $t$ distribution with $n - k$ degrees of freedom. Using $s_W^2$ instead of the variance of the two groups alone is legitimate when all $k$ populations share a variance, the assumption the analysis of variance already makes. It gives the test 36 degrees of freedom instead of 18. Here every pair has the same standard error, $\sqrt{166.20\,(1/10 + 1/10)} = 5.765$ mm Hg.

The family-wise error rate (FWER) is the probability that a set of tests rejects at least one true null hypothesis. For one test it is the significance level. Suppose $m$ independent tests are each run at level $\alpha$, and all their null hypotheses are true. Then the family-wise error rate is $1 - (1 - \alpha)^m$, or 0.265 for six tests at 0.05. Type I and Type II Errors gives the argument, and Significance Testing works it for comparisons of search engines.

Check your understanding. The six pairwise tests after an analysis of variance are not independent. They share the group means (the comparisons of groups 1 and 2 and of groups 1 and 3 both involve $\bar{x}_1$), and they all share $s_W^2$. Will their actual family-wise error rate be above or below 0.265?

Below. Tests that share ingredients tend to reject together, so their false alarms overlap, and fewer samples contain at least one. The formula is only a guide; the simulation below puts the actual rate at 0.197 for four groups.

To measure it, the script drew $k$ normal samples of 10 with equal means, for $k$ from 2 to 8. It ran all $m = \binom{k}{2}$ pairwise $t$-tests at 0.05 and counted the samples in which at least one rejected (30 seeds of 10,000 samples each, medians reported):

Groups $k$ Pairs $m$ $1 - 0.95^m$ Unadjusted pairwise tests
2 1 0.050 0.050
3 3 0.143 0.119
4 6 0.265 0.197
5 10 0.401 0.275
6 15 0.537 0.353
8 28 0.762 0.494

The dependence makes the problem smaller than the independent-tests formula says, but not small. Imagine many studies in which age has no effect at all. With four groups, a fifth of them would report at least one "significant" difference between age groups; with eight groups, half would.

Family-wise error rate of all pairwise comparisons among k groups of 10 whose population means are all equal. Unadjusted t-tests at 0.05 (circles) find a false difference in 20% of samples with four groups and 49% with eight. Notice that they stay below the independent-tests formula (dotted), because the comparisons share means and a variance estimate. Tukey's procedure (triangles) holds the rate at exactly 0.05 and Bonferroni's (squares) slightly below it.
Family-wise error rate of all pairwise comparisons among k groups of 10 whose population means are all equal. Unadjusted t-tests at 0.05 (circles) find a false difference in 20% of samples with four groups and 49% with eight. Notice that they stay below the independent-tests formula (dotted), because the comparisons share means and a variance estimate. Tukey's procedure (triangles) holds the rate at exactly 0.05 and Bonferroni's (squares) slightly below it.

The usual goal is to keep the family-wise error rate at or below a chosen $\alpha$ whatever the true means are, including when some are equal and others are not. A procedure that does this controls the FWER in the strong sense. Controlling it only when all the means are equal is called weak control, and the difference matters later in the article.

The Bonferroni correction

How can a set of tests be held to $\alpha$ overall? The simplest answer is to test each comparison at a smaller level. If $m$ comparisons are made and each is tested at level $\alpha/m$, the family-wise error rate is at most $\alpha$. The proof needs one inequality.

The union bound. For any events $A_1, \ldots, A_m$,

$$P(A_1 \cup A_2 \cup \cdots \cup A_m) \le P(A_1) + P(A_2) + \cdots + P(A_m).$$

In words: the probability that at least one of several things happens is never more than the sum of their probabilities. Overlap between the events can only make the left side smaller. For two events this follows from $P(A_1 \cup A_2) = P(A_1) + P(A_2) - P(A_1 \cap A_2)$ and $P(A_1 \cap A_2) \ge 0$. For $m$ events it follows by induction, applying the two-event case to $A_1 \cup \cdots \cup A_{m-1}$ and $A_m$.

The correction. Let $A_i$ be the event that comparison $i$ rejects. If $m_0$ of the $m$ null hypotheses are true and each is tested at level $\alpha/m$, then

$$\text{FWER} = P\Big(\bigcup_{i \text{ true null}} A_i\Big) \le \sum_{i \text{ true null}} P(A_i) = m_0\cdot\frac{\alpha}{m} \le \alpha.$$

Nothing in the argument depends on how the tests are related. So the correction controls the family-wise error rate in the strong sense, for any set of tests, dependent or not. The NIST/SEMATECH e-Handbook of Statistical Methods (§7.4.7.3) states the same inequality in its complementary form, for the probability that a set of confidence intervals all cover their targets. For all pairwise comparisons among $k$ groups, $m = \binom{k}{2} = k(k-1)/2$, and the course's significance level for each comparison is

$$\alpha^* = \frac{\alpha}{\binom{k}{2}}.$$

The procedure can be stated equivalently with adjusted $p$-values: multiply each $p$-value by $m$, cap the result at 1, and compare it with $\alpha$. This is how software usually reports it (R Core Team, p.adjust documentation).

The worked example

With $k = 4$ there are $m = 6$ comparisons, and at $\alpha = 0.05$ each is tested at $\alpha^* = 0.05/6 = 0.00833$. Equivalently, a pair differs significantly when $|t_{ij}|$ exceeds $t_{36,\,1-0.05/12} = 2.7920$. With a standard error of 5.765, that means a difference in means of at least $2.7920 \times 5.765 = 16.10$ mm Hg. An unadjusted test at 0.05 would need $|t_{ij}| > 2.0281$, a difference of 11.69 mm Hg.

Comparison $\bar{x}_i - \bar{x}_j$ $t_{ij}$ $p$ Bonferroni $p$
60+ vs 18–29 22.067 3.827 0.0005 0.0030
60+ vs 30–44 18.767 3.255 0.0025 0.0148
60+ vs 45–59 15.100 2.619 0.0128 0.0769
45–59 vs 18–29 6.967 1.208 0.2348 1
45–59 vs 30–44 3.667 0.636 0.5288 1
30–44 vs 18–29 3.300 0.572 0.5706 1

Unadjusted, three comparisons are significant at 0.05: the oldest group against each of the others. With the Bonferroni correction, two remain. The oldest group differs from the 18–29 and 30–44 groups. Its difference of 15.1 mm Hg from the 45–59 group, with $p = 0.0128$, does not clear the threshold of 0.00833. None of the differences among the three younger groups is significant either way.

These data come from populations whose means all differ, so every non-rejection here is a Type II error. The correction has turned one correct detection into a miss. This is the price of protection, measured below. It is also a reminder of what "not significant" means. A reader may be tempted to conclude that the 45–59 and oldest groups have the same mean. The samples say only that they cannot establish the difference; the populations in fact differ by 9.29 mm Hg.

Simultaneous confidence intervals

The correction applies to intervals as well as tests. Build each interval for $\mu_i - \mu_j$ with confidence level $1 - \alpha/m$:

$$(\bar{x}_i - \bar{x}_j) \pm t_{n-k,\,1-\alpha/(2m)}\sqrt{s_W^2\left(\frac{1}{n_i} + \frac{1}{n_j}\right)}.$$

Each misses its target with probability $\alpha/m$. By the union bound, the probability that any of the $m$ intervals misses is at most $\alpha$, so they are simultaneous $100(1-\alpha)\%$ intervals (NIST handbook, §7.4.7.3). For the worked example, they are $\pm 2.7920 \times 5.765 = \pm 16.10$ mm Hg around each difference. The interval for the oldest group against the 45–59 group is $[-1.00, 31.20]$, which contains zero, matching the test. The one against the youngest group is $[5.97, 38.16]$.

Holm's step-down procedure

Bonferroni's correction is simple, but is it wasteful? It tests every comparison against the same threshold, $\alpha/m$, as if all $m$ null hypotheses might be true. Once a comparison has been rejected, that worst case is no longer plausible, and the threshold for the rest can be relaxed. Holm (1979) turned this into a procedure:

  1. Order the $p$-values from smallest to largest, $p_{(1)} \le p_{(2)} \le \cdots \le p_{(m)}$.
  2. Compare $p_{(1)}$ with $\alpha/m$, $p_{(2)}$ with $\alpha/(m-1)$, and in general $p_{(r)}$ with $\alpha/(m - r + 1)$.
  3. Reject the hypotheses in order until the first $p$-value that exceeds its threshold; that hypothesis and all later ones are retained.

The first threshold is Bonferroni's and the later ones are larger, so Holm's procedure rejects everything Bonferroni's does and sometimes more. Yet it still controls the family-wise error rate in the strong sense.

Here is the idea of the proof. With $m_0$ true null hypotheses, the procedure can only reach the first true one after rejecting false ones, and by then its threshold is at most $\alpha/m_0$. A Bonferroni correction for $m_0$ tests already holds the error rate to $\alpha$.

Proof. Suppose $m_0 \ge 1$ of the null hypotheses are true. Let $E$ be the event that every true null hypothesis has a $p$-value greater than $\alpha/m_0$. By the union bound, $E$ has probability at least $1 - m_0\cdot\alpha/m_0 = 1 - \alpha$. On $E$, consider the true null hypothesis with the smallest $p$-value, and let $r$ be its position in the ordering. Every hypothesis before it is false, and there are only $m - m_0$ false ones. So $r \le m - m_0 + 1$, and its threshold is $\alpha/(m - r + 1) \le \alpha/m_0$. Its $p$-value exceeds $\alpha/m_0$ on $E$, so the procedure stops there at the latest, and no true null hypothesis is rejected. Hence the probability of rejecting any true null hypothesis is at most $P(\text{not } E) \le \alpha$.

The argument, like Bonferroni's, makes no assumption about the dependence between the tests. R's documentation for p.adjust puts the practical conclusion bluntly: "There seems no reason to use the unmodified Bonferroni correction because it is dominated by Holm's method, which is also valid under arbitrary assumptions."

Check your understanding. In the Bonferroni table above, the third-smallest $p$-value is 0.0128, for the oldest group against the 45–59 group. Holm's threshold at rank 3 is $0.05/4 = 0.0125$. Will Holm's procedure detect this difference that Bonferroni's missed?

No, though only just: 0.0128 exceeds 0.0125 by 0.0003, so the procedure stops at rank 3. The full table:

Rank Comparison $p$ Threshold Decision
1 60+ vs 18–29 0.0005 $0.05/6 = 0.0083$ Reject
2 60+ vs 30–44 0.0025 $0.05/5 = 0.0100$ Reject
3 60+ vs 45–59 0.0128 $0.05/4 = 0.0125$ Stop
4–6 the three younger pairs 0.2348 to 0.5706 Retain

Holm's procedure makes the same decisions as Bonferroni's here. Its adjusted $p$-value for the oldest against the 45–59 group is $4 \times 0.0128 = 0.0513$, against Bonferroni's 0.0769. The adjusted $p$-value at rank $r$ is the largest of $(m - s + 1)\,p_{(s)}$ over $s \le r$, capped at 1, so that adjusted values never decrease down the ordering.

Tukey's honestly significant difference

The Bonferroni and Holm procedures use only the union bound, so they must allow for the worst case. Can a procedure built for pairwise comparisons do better? For that specific family, the exact distribution of the largest comparison is known. A procedure built on it, due to John Tukey, holds the family-wise error rate at or below $\alpha$ for any means. When the means are all equal and the group sizes are equal, it holds the rate at exactly $\alpha$.

Here is the idea in one sentence. All the pairwise differences are within a margin of their targets exactly when the errors in the group means, largest minus smallest, are within that margin. So a bound on the range of the errors covers every pair at once.

The studentized range. Let $Z_1, \ldots, Z_k$ be independent standard normal variables. Let $S$ be independent of them, with $\nu S^2$ chi-squared on $\nu$ degrees of freedom (the distribution is defined in Confidence Intervals for a Mean). The studentized range is

$$Q = \frac{\max_i Z_i - \min_i Z_i}{S},$$

the range of the $k$ values divided by an independent estimate of their standard deviation. Its distribution depends only on $k$ and $\nu$, and its upper $\alpha$ point is written $q_{k,\,\nu,\,1-\alpha}$. The NIST handbook (§7.4.7.1) gives the definition with an example. For 5 normal observations and 10 degrees of freedom, the 95th percentile is 4.65. So the range of five observations is at most 4.65 times an independent standard deviation estimate, with probability 0.95.

The connection to pairwise tests. Suppose every group has $n_0$ observations. Whatever the population means are, the standardized errors of the group means,

$$Z_i = \frac{\bar{X}_i - \mu_i}{\sigma/\sqrt{n_0}},$$

are independent standard normal variables. And $S = s_W/\sigma$ is independent of them, with $(n - k)S^2$ chi-squared on $n - k$ degrees of freedom, as shown in One-Way Analysis of Variance. The error in the estimated difference between groups $i$ and $j$ is

$$(\bar{X}_i - \bar{X}_j) - (\mu_i - \mu_j) = \frac{\sigma}{\sqrt{n_0}}(Z_i - Z_j),$$

and the largest of these errors over all pairs, measured in units of $s_W/\sqrt{n_0}$, is $(\max Z_i - \min Z_i)/S = Q$. So with probability exactly $1 - \alpha$, every pair satisfies

$$\big|(\bar{x}_i - \bar{x}_j) - (\mu_i - \mu_j)\big| \le q_{k,\,n-k,\,1-\alpha}\sqrt{\frac{s_W^2}{n_0}}.$$

These are Tukey's simultaneous confidence intervals for all pairwise differences. The procedure declares a pair different when its interval excludes zero, that is, when

$$|\bar{x}_i - \bar{x}_j| > \text{HSD} = q_{k,\,n-k,\,1-\alpha}\sqrt{\frac{s_W^2}{n_0}},$$

the honestly significant difference. The intervals cover all differences together with probability $1 - \alpha$ whatever the means are. So the procedure controls the family-wise error rate in the strong sense, and at exactly $\alpha$ when all the means are equal. In terms of the $t_{ij}$ statistics, it compares each $|t_{ij}|$ with $q_{k,\,n-k,\,1-\alpha}/\sqrt{2}$, since the standard error of a difference is $\sqrt{2}$ times $s_W/\sqrt{n_0}$. The NIST handbook writes the intervals in that form.

The worked example. For $k = 4$ groups and 36 degrees of freedom, $q_{4,\,36,\,0.95} = 3.8088$, and

$$\text{HSD} = 3.8088\sqrt{\frac{166.20}{10}} = 15.53 \text{ mm Hg}.$$

The differences of 22.07 and 18.77 mm Hg between the oldest group and the two youngest exceed it. The difference of 15.10 mm Hg between the oldest and the 45–59 group falls just short, with a Tukey-adjusted $p$-value of 0.0592. The decisions are the same as Bonferroni's and Holm's. Tukey's critical value on the $t$ scale, $3.8088/\sqrt{2} = 2.6932$, is a little smaller than Bonferroni's 2.7920. So its intervals are about 4% narrower: $[-0.43, 30.63]$ against $[-1.00, 31.20]$ for the oldest against the 45–59 group.

95% intervals for the six pairwise differences in mean blood pressure among the four age groups: unadjusted (top of each triple), Tukey's simultaneous intervals (middle) and Bonferroni's (bottom). Notice that the protected intervals are much wider than the unadjusted ones but only slightly different from each other. Three unadjusted intervals exclude zero; two Tukey and two Bonferroni intervals do, those comparing the oldest group with the two youngest.
95% intervals for the six pairwise differences in mean blood pressure among the four age groups: unadjusted (top of each triple), Tukey's simultaneous intervals (middle) and Bonferroni's (bottom). Notice that the protected intervals are much wider than the unadjusted ones but only slightly different from each other. Three unadjusted intervals exclude zero; two Tukey and two Bonferroni intervals do, those comparing the oldest group with the two youngest.

Unequal group sizes. With unequal $n_i$ the range argument no longer applies directly. The Tukey–Kramer procedure, the form for unequal sizes (Kramer, 1956), replaces $\sqrt{s_W^2/n_0}$ by $\sqrt{\tfrac{1}{2}s_W^2(1/n_i + 1/n_j)}$ for each pair. That is, it compares each $|t_{ij}|$ with $q_{k,\,n-k,\,1-\alpha}/\sqrt{2}$ as before. Its family-wise error rate is then below $\alpha$ rather than equal to it (NIST handbook, §7.4.7.1). Hayter (1984) proved the conjecture that this procedure is conservative.

Comparing the procedures

On the worked sample, all three procedures made the same decisions. How much do they differ in general? For all pairwise comparisons among $k$ groups of 10, the critical values on the scale of $|t_{ij}|$ are:

$k$ $m$ df Unadjusted $t$ Bonferroni $t$ Tukey $q/\sqrt{2}$ Scheffé
2 1 18 2.101 2.101 2.101 2.101
3 3 27 2.052 2.552 2.479 2.590
4 6 36 2.028 2.792 2.693 2.932
5 10 45 2.014 2.952 2.841 3.212
6 15 54 2.005 3.071 2.954 3.454
8 28 72 1.993 3.245 3.122 3.870

With two groups all four coincide, and every procedure is the two-sample $t$-test. With more groups, the protected critical values grow with the number of comparisons. They grow much faster than the unadjusted one shrinks with the extra degrees of freedom. Bonferroni's critical value is only 3 to 4% larger than Tukey's throughout this range. For all pairwise comparisons Tukey's procedure is the better choice, as the NIST handbook (§7.4.7.3) also concludes, but the difference is modest.

The simulation measured the family-wise error rate of each procedure under the complete null hypothesis, all means equal. Tukey's procedure held 0.050 for every $k$, as the theory says it must. Bonferroni's was below the nominal level, from 0.042 at $k = 3$ to 0.036 at $k = 8$. The union bound ignores the overlap between the events "comparison $i$ rejects", which is large when the comparisons are dependent, and the dependence grows with the number of groups. Holm's rate was the same as Bonferroni's, as it must be under the complete null hypothesis. There both procedures reject something exactly when the smallest $p$-value is below $\alpha/m$.

Scheffé's method answers a broader question. A contrast is any combination $\sum_i c_i\mu_i$ with $\sum_i c_i = 0$. A pairwise difference is one; the average of the two younger groups against the average of the two older ones is another. Scheffé's procedure gives intervals that cover all contrasts simultaneously, infinitely many of them. It uses the critical value $\sqrt{(k - 1)F_{k-1,\,n-k,\,1-\alpha}}$ with each contrast's own standard error (NIST handbook, §7.4.7.2). The last column of the table shows the cost: for pairwise comparisons only, it is the widest of the three. It is the right tool when the comparisons of interest are chosen after looking at the data and are not all pairwise.

Two other procedures are worth knowing by name:

  • Dunnett's procedure compares several treatments with a single control group. It needs fewer comparisons than all pairs, and so gives narrower intervals.
  • The Games–Howell procedure is a pairwise procedure built from Welch's unequal-variance standard errors, for groups that do not share a variance.

scipy provides both (scipy.stats.dunnett, and scipy.stats.tukey_hsd with equal_var=False).

Does a significant F test protect the comparisons?

A common belief says the corrections are unnecessary after a significant $F$ test: "the $F$ test already guarded against false alarms, so the pairwise tests can run unadjusted." The traditional practice built on it runs the analysis of variance first and makes unadjusted pairwise comparisons only if the $F$ test is significant. This is often called Fisher's least significant difference procedure. The NIST handbook (§7.4.7) also calls the $F$ test a "preliminary test" that indicates whether to investigate further, although the procedures it then describes are the protected ones of this article.

The idea is that the $F$ test acts as a gatekeeper. When all the means are equal it rejects only 5% of the time, so the pairwise tests that follow can produce a false alarm at most 5% of the time. In the simulation the gate held exactly that. Under the complete null hypothesis, the $F$-protected procedure had a family-wise error rate of 0.049 to 0.050 for every $k$.

Check your understanding. Now suppose eight groups of 10 fall into two clusters: four groups with mean 0 and four with mean 3 standard deviations. The 12 pairs within the clusters have equal means. Does the $F$ gate still keep the chance of a false difference among those 12 pairs near 5%?

No. The $F$ test rejects because of the real differences between the clusters; in the script it rejected in every sample. The gate is then always open, and the unadjusted tests among the equal means run with no protection at all. Among the 12 true null comparisons, the $F$-protected procedure produced at least one false rejection in 35.6% of samples. Bonferroni's did so in 1.8%, Holm's in 4.1% and Tukey's in 2.7%.

So the $F$ test gives only weak control. It protects against false alarms when nothing is going on, which is not the situation in which comparisons are usually made.

With exactly three groups the gatekeeping does work. If all three means are equal, the $F$ test protects. If not all are equal, at most one of the three pairs has equal means (two equal pairs would force the third). So at most one true null hypothesis is tested, at level $\alpha$. With four or more groups, a partial null hypothesis can leave several true null comparisons, and the argument breaks down.

What protection costs

Every procedure above buys its control of false alarms with something. Controlling the family-wise error rate means accepting more Type II errors, and the script measured how many. It simulated the design of the worked example:

  • four normal populations with the NHANES age-group means (113.18, 118.75, 127.56 and 136.85 mm Hg);
  • a common standard deviation of 17.06 mm Hg;
  • samples of 10 per group.

Every pair of means differs, so every rejection is correct. The table gives the share of samples in which each procedure detected each difference (30 seeds of 10,000 samples).

Comparison True difference (mm Hg) Unadjusted Bonferroni Holm Tukey
60+ vs 18–29 23.67 0.856 0.625 0.636 0.658
60+ vs 30–44 18.10 0.636 0.352 0.382 0.387
45–59 vs 18–29 14.39 0.451 0.200 0.231 0.227
60+ vs 45–59 9.29 0.220 0.069 0.089 0.082
45–59 vs 30–44 8.81 0.204 0.062 0.085 0.074
30–44 vs 18–29 5.57 0.110 0.027 0.037 0.032

Averaged over the six differences, unadjusted tests detected 41.3%, and the protected procedures between 22.2% (Bonferroni) and 24.3% (Holm and Tukey). Protection roughly halves the power of each comparison in this design. Holm's procedure and Tukey's gain a few points on Bonferroni's, and neither dominates the other. Tukey's is ahead for the largest differences, Holm's for the smaller ones. Once Holm's procedure has rejected the large differences, it tests the rest against relaxed thresholds.

The remedy for low power is the same as for a single test: a larger sample, planned with the multiple comparisons in mind. A study that intends to compare every pair among four groups should compute its sample size at the corrected significance level, not at 0.05. Ten adults per age group are far too few to resolve differences of 5 to 10 mm Hg between neighboring age groups, with or without correction.

Choosing a procedure

With the costs measured, the choice depends on which comparisons a study needs.

  • Decide the family in advance. The correction depends on the number of comparisons, so the set of comparisons should be fixed before the data are seen. The NIST handbook (§7.4.7) makes the same point for contrasts: chosen after seeing the results, their stated confidence levels no longer hold. A study interested only in the oldest group against each other group has three comparisons, not six, and a Bonferroni correction for three.
  • For all pairwise comparisons among groups of equal or similar size, Tukey's procedure gives exact simultaneous intervals and is slightly more powerful than Bonferroni's. With clearly unequal variances, the Games–Howell procedure is the pairwise analogue of Welch's analysis of variance.
  • For a small, prespecified set of comparisons, or tests of any kind, Holm's procedure is the default. It needs nothing but the $p$-values, makes no assumption about their dependence, and is never less powerful than Bonferroni's. Bonferroni's remains useful for its simplicity and for intervals, which Holm's procedure does not provide.
  • For contrasts chosen after looking at the data, Scheffé's method.
  • The $F$ test is not required first. Tukey's and Holm's procedures control the error rate on their own. The two kinds of test can also disagree: an $F$ test can be significant with no pairwise difference significant, since the $F$ test also responds to contrasts that are not pairwise.

Family-wise control is the right standard when a single false claim is costly, as when each comparison may change a clinical decision. Genomic studies, by contrast, screen thousands of hypotheses at once. There, demanding no false positives at all leaves almost no power, and a weaker criterion is used. The false discovery rate is the expected proportion of false discoveries among the rejected hypotheses, and the procedure of Benjamini and Hochberg (1995) controls it (R Core Team, p.adjust documentation). That answers a different question, about the proportion of errors among many findings rather than the chance of any error among a few. For comparing a handful of group means, the procedures above remain the standard.

What we have learned

Six unadjusted tests among four groups produce a false difference in about a fifth of samples when no difference exists. The Bonferroni correction tests each at $\alpha/m$ and holds the family-wise error rate to $\alpha$ by the union bound alone. Holm's step-down version rejects at least as much, and Tukey's studentized range gives exact simultaneous intervals for all pairs. On the age groups, all three found the oldest group above the two youngest and could not establish the other four differences. A significant $F$ test does not make unadjusted comparisons safe, and protection roughly halves the power of each comparison.

The tests of this unit, from the $F$ test to the pairwise $t$-tests and Tukey's range, all assume normal populations, and all but Welch's analysis of variance and the Games–Howell procedure also assume a common variance. Bonferroni's and Holm's corrections need only valid $p$-values, but here the $p$-values come from $t$-tests. For skewed data or outliers, a different idea is needed: compare ranks instead of values. The next topic, Nonparametric Methods, begins with the sign test and the Wilcoxon signed-rank test for paired data, and reaches several groups with the Kruskal–Wallis test.

In Python

import numpy as np
from scipy import stats

g1 = [114.7, 110.7, 110.0, 114.0, 118.7, 114.0, 96.7, 100.0, 110.0, 94.0]       # 18-29
g2 = [116.0, 116.7, 131.3, 110.7, 112.7, 88.7, 96.0, 121.3, 95.0, 127.3]        # 30-44
g3 = [108.0, 112.7, 137.0, 103.3, 116.7, 114.0, 108.7, 112.7, 106.0, 133.3]     # 45-59
g4 = [144.7, 156.0, 108.7, 134.0, 110.0, 122.7, 144.0, 126.0, 142.0, 115.3]     # 60+

res = stats.tukey_hsd(g1, g2, g3, g4)
print(res.pvalue.round(4))                 # res.pvalue[3, 2], 60+ vs 45-59: 0.059
print(res.confidence_interval(0.95).low)   # Tukey's simultaneous 95% intervals (low ends)

def bonferroni(p):
    return np.minimum(1, len(p) * np.asarray(p))

def holm(p):
    p = np.asarray(p)
    order = np.argsort(p)
    adj = np.minimum(1, (len(p) - np.arange(len(p))) * p[order])
    out = np.empty_like(adj)
    out[order] = np.maximum.accumulate(adj)
    return out

p = [0.0005, 0.0025, 0.0128, 0.2348, 0.5288, 0.5706]
print(bonferroni(p))   # 0.003, 0.015, 0.0768, 1, 1, 1
print(holm(p))         # 0.003, 0.0125, 0.0512, 0.7044, 1, 1

The $p$-values in the last lines are the rounded ones from the table, so the adjusted values differ from the table's in the last digit. The function tukey_hsd uses the Tukey–Kramer form when the group sizes differ and the Games–Howell procedure with equal_var=False. In R, p.adjust(p, method = "holm") and TukeyHSD do the same. Every number in this article is computed by tools/experiments/applied-statistics/multiple-comparisons-and-bonferroni.py in the site's repository, which implements the studentized range distribution by numerical integration and was checked against scipy 1.17.1.

References