The course page Multiple Comparisons and the Bonferroni Correction compares four NHANES age groups, 10 adults each, in all six pairs. Holm's procedure rejects the two smallest p-values and stops at the third: the oldest group against the 45–59 group has $p = 0.0128$, and Holm's threshold at rank 3 is $0.05/4 = 0.0125$. Yet Holm's threshold assumes that, after two rejections, four of the six pairwise hypotheses might still be true, and with four groups that cannot happen: once two groups differ, at most three pairs can have equal means. Is there a principle that says which procedures control the family-wise error rate, so that a procedure using this fact can be built and proved? There is, and it rejects the third pair.
The principle is closed testing (Marcus, Peritz and Gabriel, 1976). Test every intersection of the hypotheses, each at level $\alpha$ with any valid test, and reject a hypothesis only if every intersection that contains it has been rejected. The family-wise error rate is then at most $\alpha$, whatever the dependence between the tests. Holm's procedure from the course page is a closed testing procedure in disguise, and the refinements of this article are closed tests or reject no more than one.
Here is the route. First we define the closure and prove the principle. Then a shortcut lemma makes closures computable, and Holm's procedure falls out as the closure of Bonferroni tests. Next come closures built from Simes' test, the procedures of Hommel and Hochberg, and the fixed-sequence procedure. Then we turn to pairwise comparisons, where the hypotheses are logically related: we count which configurations of true hypotheses are possible, and prove Shaffer's procedure and the full closure over partitions of the groups. Three groups explain why Fisher's least significant difference works there and fails with four. Finally, closed testing gives a bound on the number of false discoveries in any set of hypotheses a reader chooses. The article assumes the course page above, whose union bound, family-wise error rate and proof of Holm's procedure it uses, and the unit's notation set out in Multiple Testing.
Intersection hypotheses and the closure
A family of null hypotheses $H_1, \ldots, H_m$ is given, each with a p-value. For a nonempty set of indices $I \subseteq \{1, \ldots, m\}$, the intersection hypothesis $H_I$ is the statement that every $H_i$ with $i \in I$ is true. It is true exactly when $I \subseteq \mathcal{I}_0$, the set of true null hypotheses. The single hypotheses are the intersections of one, $H_{\{i\}} = H_i$, and there are $2^m - 1$ intersections in all.
For three hypotheses the seven intersections are $H_1$, $H_2$, $H_3$, $H_1 \cap H_2$, $H_1 \cap H_3$, $H_2 \cap H_3$ and $H_1 \cap H_2 \cap H_3$. A local test is a test of one intersection hypothesis at level $\alpha$: it rejects $H_I$ with probability at most $\alpha$ whenever $H_I$ is true. Nothing more is asked of it. It may be a Bonferroni test, an $F$ test, a permutation test, or a test built for the dependence between these particular statistics.
The closed testing procedure. Choose a local test for every intersection. Reject $H_i$ if and only if the local test rejects $H_I$ for every $I$ that contains $i$.
In the three-hypothesis case, $H_1$ is rejected only if the local tests reject $H_1$, $H_1 \cap H_2$, $H_1 \cap H_3$ and $H_1 \cap H_2 \cap H_3$. The same rule applies to intersections: the procedure rejects $H_I$ when every $H_J$ with $J \supseteq I$ is rejected locally. The family-wise statements below concern the single hypotheses, and the last section uses the rest.
The closure principle
Here is the idea in one sentence. If the procedure makes any false rejection, it must have rejected the intersection of all the true null hypotheses, which is itself true, and a single level-$\alpha$ test does that with probability at most $\alpha$.
Theorem 1 (closure principle; Marcus, Peritz and Gabriel, 1976). If every local test has level $\alpha$, the closed testing procedure rejects at least one true null hypothesis with probability at most $\alpha$, whichever hypotheses are true and whatever the joint distribution of the tests.
Proof. If no null hypothesis is true there is nothing to prove. Otherwise $\mathcal{I}_0$ is nonempty and $H_{\mathcal{I}_0}$ is a true intersection hypothesis, so its local test rejects it with probability at most $\alpha$. Suppose the procedure rejects some true $H_i$, so $i \in \mathcal{I}_0$. By the rule, the local test rejected every intersection containing $i$, and $\mathcal{I}_0$ is one of them. So the event "some true null hypothesis is rejected" is contained in the event "the local test rejects $H_{\mathcal{I}_0}$", and its probability is at most $\alpha$. $\blacksquare$
The proof uses one local test only, the one for the true intersection, and makes no assumption about how the tests depend on one another. It also never uses the fact that the remaining local tests have level $\alpha$; that is needed only because $\mathcal{I}_0$ is unknown, so every intersection must be tested as though it might be the true one.
Example (why the condition is not "reject when the single test rejects"). The rule asks more than the local test of $H_i$. With Bonferroni local tests, $H_1 \cap H_2 \cap H_3$ is rejected when the smallest of the three p-values is at most $0.05/3$. If all three p-values exceed $0.0167$, the procedure rejects nothing, even though each might be below 0.05 on its own. The top of the lattice is a gate that every rejection passes through.
A shortcut, and Holm's procedure
A closure has $2^m - 1$ local tests, too many to compute for $m = 50$. For the commonest local tests, a hypothesis needs only $m$ of them.
Call a family of local tests symmetric if the test of $H_I$ depends on the p-values in $I$ only through their sorted values, by the same rule for every $I$ of the same size. Call it monotone if lowering p-values never turns a rejection into a non-rejection: if the sorted vector $u$ is at most $v$ in every coordinate and the rule rejects $v$, it rejects $u$. The Bonferroni test, which rejects $H_I$ when $\min_{i \in I} p_i \le \alpha/|I|$, is both.
Lemma 1 (shortcut). For symmetric, monotone local tests, the closure rejects $H_i$ if and only if, for every size $j = 1, \ldots, m$, the local test rejects the set $J_j(i)$ made of $i$ and the $j - 1$ other hypotheses with the largest p-values.
Here is the idea: among all sets of size $j$ containing $i$, the one filled with the largest other p-values is the hardest to reject, so if it falls, all of them fall.
Proof. The sets $J_j(i)$ contain $i$, so the closure's rule requires them to be rejected. For the converse, take any $I = \{i\} \cup A$ of size $j$, with $A$ drawn from the set $O$ of the other $m - 1$ indices, and let $B$ be the $j - 1$ indices of $O$ with the largest p-values. For two collections of $j$ numbers, the sorted vector of the first is at most that of the second in every coordinate exactly when, for every $x$, the first has at least as many members at or below $x$. (The $r$-th smallest is at most $x$ exactly when at least $r$ members are at or below $x$.) Write $N_A(x)$, $N_B(x)$ and $N_O(x)$ for the numbers of p-values of $A$, $B$ and $O$ at or below $x$. The $|O| - j + 1$ indices of $O$ outside $B$ have the smallest p-values, so $$N_B(x) = \max\big(0,\ N_O(x) - (|O| - j + 1)\big).$$ $A$ omits only $|O| - j + 1$ members of $O$, so $N_A(x) \ge N_O(x) - (|O| - j + 1)$, and $N_A(x) \ge 0$; hence $N_A(x) \ge N_B(x)$. Adding the same p-value $p_i$ to both collections adds the same amount to both counts, so the sorted p-values of $I$ are at most those of $J_j(i)$ in every coordinate. By monotonicity and symmetry, rejection of $J_j(i)$ implies rejection of $I$. $\blacksquare$
Theorem 2 (Holm's procedure is a closed test). With Bonferroni local tests, the closure rejects exactly the hypotheses that Holm's procedure rejects.
Proof. Let $H_i$ have rank $r$, so $p_i = p_{(r)}$ (ties ordered in any fixed way). By Lemma 1 we check the sets $J_j(i)$ for $j = 1, \ldots, m$. While $j - 1 \le m - r$, the $j - 1$ largest other p-values are those of ranks $m - j + 2$ to $m$, all at least $p_{(r)}$, so the smallest p-value in $J_j(i)$ is $p_{(r)}$, and the test rejects when $p_{(r)} \le \alpha/j$. The strictest of these conditions, at $j = m - r + 1$, is $p_{(r)} \le \alpha/(m - r + 1)$. For larger $j$, write $s = m - j + 1$, which runs over $1, \ldots, r - 1$. The set $J_j(i)$ then holds all ranks from $s$ to $m$, so its smallest p-value is $p_{(s)}$, and the test rejects when $p_{(s)} \le \alpha/(m - s + 1)$. So the closure rejects $H_{(r)}$ exactly when $p_{(s)} \le \alpha/(m - s + 1)$ for every $s \le r$, which is Holm's rule: step down through the ordered p-values and stop at the first that exceeds its threshold. $\blacksquare$
The course page proves Holm's procedure directly. Theorem 2 shows where it comes from, and Theorem 1 then gives its error control at once. The script checks Theorem 2 and Lemma 1 by computing closures by brute force over all $2^m - 1$ intersections, for 2,400 random vectors of p-values with $m$ from 2 to 7 (seed 1, p-values rounded to three decimals so that ties occur). Every one agreed.
Closures of Simes tests: Hommel's and Hochberg's procedures
The Bonferroni test of an intersection looks only at the smallest p-value. Simes' test (Simes, 1986) looks at all of them: it rejects $H_I$, with $|I| = j$ and sorted p-values $u_{(1)} \le \cdots \le u_{(j)}$, if $u_{(r)} \le r\alpha/j$ for at least one $r$. It rejects whenever the Bonferroni test does (take $r = 1$), and also when several p-values are moderately small together. The False Discovery Rate and the Benjamini–Hochberg Procedure proves that Simes' test has level exactly $\alpha$ when the p-values of $I$ are independent and uniform under $H_I$, and at most $\alpha$ when they are independent and valid. Under dependence it can exceed $\alpha$, as that article also shows. Sarkar (1998) proved it conservative for a class of positively dependent statistics; Goeman and Solari (2011, §4.2) summarize the conditions as including identically distributed, nonnegatively correlated test statistics.
Simes' test is symmetric and monotone, so Lemma 1 applies, and the closure of Simes tests needs only $m^2$ local tests. Hommel (1988), in the abstract of his paper, describes his procedure as an application of the closed testing principle of Marcus, Peritz and Gabriel to Simes' test; we follow the usual name, Hommel's procedure, for this closure. A simpler rule, Hochberg's procedure (Hochberg, 1988), steps up: find the largest $k$ with $p_{(k)} \le \alpha/(m - k + 1)$ and reject $H_{(1)}, \ldots, H_{(k)}$. It has Holm's thresholds but reads them from the top, so it rejects everything Holm's does.
Theorem 3. Every hypothesis that Hochberg's procedure rejects is rejected by the closure of Simes tests. Hochberg's procedure therefore controls the family-wise error rate whenever Simes' test is valid for every intersection of true null hypotheses, in particular when the true null p-values are independent.
Proof. Let $k \ge 1$ be Hochberg's number of rejections, with $p_{(k)} \le \alpha/(m - k + 1)$, and let $H_i$ be one of the rejected, of rank at most $k$. Take any $J$ containing $i$, of size $j$. $J$ omits $m - j$ hypotheses, so it contains at least $k - (m - j)$ of the $k$ smallest p-values, and it contains at least one of them, $p_i$. Let $r \ge \max(1, k - m + j)$ be the number it contains. Then the $r$-th smallest p-value of $J$ is at most $p_{(k)} \le \alpha/(m - k + 1)$, and Simes' test rejects $H_J$ if $\alpha/(m - k + 1) \le r\alpha/j$, that is, if $j \le r(m - k + 1)$. Write $d = m - k \ge 0$. If $j \le d + 1$, this holds because $r \ge 1$. If $j > d + 1$, then $r \ge j - d$, and $$r(d + 1) \ge (j - d)(d + 1) = j + d(j - d - 1) \ge j,$$ since $d \ge 0$ and $j - d - 1 > 0$. So every $H_J$ containing $i$ is rejected, and the closure rejects $H_i$. The second statement follows from Theorem 1. $\blacksquare$
So the three procedures are nested: Holm's rejections are among Hochberg's, and Hochberg's among Hommel's. Holm's needs no assumption about dependence; the other two need Simes' test to be valid. In the script's 2,400 random vectors, the Simes closure rejected more than Hochberg's procedure in 66.
Example (the three procedures on one family). For three p-values 0.02, 0.03 and 0.04 at $\alpha = 0.05$, Holm's procedure rejects nothing: $0.02 > 0.05/3$. Hochberg's rejects all three, since the largest, 0.04, is below $0.05/1$. For 0.02, 0.03 and 0.06, Hochberg's rejects nothing ($0.06 > 0.05$, $0.03 > 0.025$, $0.02 > 0.0167$), but the Simes closure rejects the first hypothesis. Worked problem 1 computes both closures by hand. On the fifteen p-values of Benjamini and Hochberg's (1995, §3.2) clinical example, all three procedures reject the same three hypotheses.
Fixed sequences
A study often knows in advance which hypotheses matter most. Order them before seeing the data, $H_1, H_2, \ldots, H_m$, and test each at the full level $\alpha$ in that order, stopping at the first that is not rejected. This fixed-sequence procedure spends no level on multiplicity at all.
Theorem 4. The fixed-sequence procedure is the closure of the local tests that reject $H_I$ when the p-value of the first hypothesis of $I$ in the order is at most $\alpha$. It controls the family-wise error rate.
Proof. If $H_I$ is true, its first hypothesis is true, and its valid p-value is at most $\alpha$ with probability at most $\alpha$, so each local test has level $\alpha$. The closure rejects $H_i$ when every $I$ containing $i$ has a first member with p-value at most $\alpha$. The first member of such an $I$ can be any index $j \le i$ (take $I = \{j, i\}$, or $I = \{i\}$ for $j = i$), and only those. So $H_i$ is rejected exactly when $p_j \le \alpha$ for every $j \le i$, which is the procedure. Theorem 1 gives the error control. $\blacksquare$
Example (the four age groups in a planned order). Suppose the protocol for the four age groups had ordered the comparisons by the gap in age from the oldest group: 60+ against 18–29, then against 30–44, then against 45–59, then 45–59 against 18–29, 45–59 against 30–44 and 30–44 against 18–29. The first three p-values, 0.0005, 0.0025 and 0.0128, are each below 0.05, and the fourth, 0.2348, is not. So the procedure rejects three hypotheses, one more than Holm's. The gain is bought with the order: had the protocol put a younger pair first, its p-value of 0.5706 would have stopped the procedure at once, with nothing rejected. The order must be fixed before the data are seen, or the proof's local tests are not the ones used.
Pairwise comparisons: logically related hypotheses
Back to the four age groups. The six pairwise hypotheses $\mu_a = \mu_b$ are not free to be true or false independently. If $\mu_1 = \mu_2$ and $\mu_2 = \mu_3$, then $\mu_1 = \mu_3$. Holm's procedure ignores this and so allows for configurations that cannot occur.
Theorem 5 (possible numbers of true pairwise hypotheses). With $k$ groups, the sets of true pairwise hypotheses that can occur are exactly those of the form "all pairs within the blocks of a partition of the groups". The possible numbers of true hypotheses form the set $$S(k) = \Big\{\sum_{\text{blocks } b} \binom{|b|}{2} : \text{partitions of the } k \text{ groups}\Big\},$$ which satisfies $S(0) = S(1) = \{0\}$ and $S(k) = \bigcup_{j=1}^{k}\big\{\binom{j}{2} + x : x \in S(k - j)\big\}$.
Proof. For any means $\mu_1, \ldots, \mu_k$, "has the same mean as" is an equivalence relation on the groups, so its classes partition them, and $\mu_a = \mu_b$ is true exactly when $a$ and $b$ are in the same class. Conversely, every partition occurs: give each block its own distinct mean. A block of size $j$ contributes $\binom{j}{2}$ true pairs. For the recursion, the block containing the first group has some size $j$ between 1 and $k$, contributing $\binom{j}{2}$, and the other $k - j$ groups are partitioned in any way, contributing a member of $S(k - j)$. $\blacksquare$
For four groups the partitions give $S(4) = \{0, 1, 2, 3, 6\}$: all equal (6), three equal (3), two pairs (1 + 1 = 2), one pair (1), all different (0). Four or five true hypotheses are impossible. The script enumerates the partitions and checks the recursion for $k$ up to 8. For six groups, $$S(6) = \{0, 1, 2, 3, 4, 6, 7, 10, 15\}.$$
Shaffer's procedure. Shaffer (1986) used these counts in Holm's step-down scheme. The description here follows García and Herrera (2008), who state her "static" procedure for comparing classifiers. Define $t_r$ as the largest number of hypotheses that can be true when at least $r - 1$ are false: $$t_r = \max\{s \in S(k) : s \le m - r + 1\}.$$ Step down through the ordered p-values, rejecting $H_{(r)}$ while $p_{(r)} \le \alpha/t_r$, and stop at the first failure. Holm's procedure is the case $t_r = m - r + 1$. For four groups ($m = 6$) the divisors are $6, 3, 3, 3, 2, 1$ instead of Holm's $6, 5, 4, 3, 2, 1$.
Theorem 6. Shaffer's procedure controls the family-wise error rate in the strong sense, whatever the dependence between the tests.
Proof. It follows the course page's proof of Holm's procedure. If no hypothesis is true there is nothing to prove. Otherwise let $m_0 \ge 1$ be the number of true hypotheses, and $E$ the event that every true hypothesis has $p > \alpha/m_0$. By the union bound, $E$ fails with probability at most $m_0 \cdot \alpha/m_0 = \alpha$. On $E$, let $r$ be the rank of the first true hypothesis. The $r - 1$ hypotheses before it are false, so $m_0 \le m - r + 1$, and $m_0$ belongs to $S(k)$ by Theorem 5. So $m_0 \le t_r$, and the threshold at rank $r$ is $\alpha/t_r \le \alpha/m_0$, which the p-value at rank $r$ exceeds on $E$. The procedure stops at rank $r$ or earlier, rejecting only false hypotheses. $\blacksquare$
Example (the third pair). On the four age groups, Holm's procedure stopped at rank 3 with $p = 0.0128$ against $0.05/4 = 0.0125$. Shaffer's threshold at rank 3 is $0.05/t_3 = 0.05/3 = 0.0167$, so it rejects. At rank 4 the p-value is 0.2348, far above $0.05/3$, and it stops. Shaffer's procedure concludes that the oldest group differs from each of the other three, and it rejects no other pair.
The full closure over partitions. Shaffer's procedure uses only the counts. The closure principle can use the whole structure. An intersection of pairwise hypotheses asserts that certain pairs are equal, and by transitivity it asserts that the groups fall into blocks with equal means. So every intersection hypothesis is the hypothesis of some partition $P$: the means are equal within each block of $P$. Test it locally with the Bonferroni test over the $c(P)$ pairs that $P$ makes equal: reject when the smallest of their p-values is at most $\alpha/c(P)$. When the hypothesis of $P$ is true, all $c(P)$ pairs are true, and by the union bound the test has level $\alpha$. The closure then rejects the pair $(a, b)$ when every partition that puts $a$ and $b$ in the same block is rejected. Theorem 1 gives:
Theorem 7. The closure over partition hypotheses with Bonferroni local tests controls the family-wise error rate for all pairwise comparisons, whatever the dependence between the tests. $\blacksquare$
García and Herrera (2008) give this procedure as Bergmann and Hommel's (1988), in the form "retain every hypothesis in some exhaustive set $I$ with $\min_{i \in I} p_i > \alpha/|I|$", where an exhaustive set is the set of pairs a partition makes equal; the two descriptions are the same rule. Four groups have 15 partitions, of which 14 put at least two groups together; six groups have 203, of which 202 do.
Shaffer's procedure is not itself shown here to be a closed test; Theorem 6 proved its error control directly. It is, however, never more powerful than the closure over partitions.
Proposition 1. Every pair that Shaffer's procedure rejects is rejected by the closure over partitions.
Proof. Suppose Shaffer's procedure rejects $H_{(1)}, \ldots, H_{(K)}$, and take a pair of rank $r \le K$ and any partition $P$ that puts its two groups in one block, with $c = c(P)$ equal pairs. Let $s$ be the smallest rank among those $c$ pairs, so $s \le r \le K$. The $s - 1$ pairs of smaller rank are not among them, so $c \le m - s + 1$; and $c$ belongs to $S(k)$ by Theorem 5. Hence $c \le t_s$. Since $s \le K$, Shaffer's procedure passed rank $s$: $p_{(s)} \le \alpha/t_s \le \alpha/c$. The smallest p-value among the pairs of $P$ is $p_{(s)}$, so the local test of $P$ rejects. This holds for every partition containing the pair, so the closure rejects it. $\blacksquare$
The script checks Proposition 1 in every simulated sample below.
Example (the closure on the four age groups). For the oldest group against the 45–59 group, five partitions put the two together. Each is rejected:
| Partition (blocks with equal means) | Pairs $c(P)$ | Smallest p-value | Level $0.05/c(P)$ |
|---|---|---|---|
| all four groups | 6 | 0.0005 | 0.0083 |
| {30–44, 45–59, 60+}, {18–29} | 3 | 0.0025 | 0.0167 |
| {18–29, 45–59, 60+}, {30–44} | 3 | 0.0005 | 0.0167 |
| {18–29, 30–44}, {45–59, 60+} | 2 | 0.0128 | 0.0250 |
| {45–59, 60+} | 1 | 0.0128 | 0.0500 |
So the closure rejects the pair, and in all it rejects the same three hypotheses as Shaffer's procedure.
Six age groups
A larger family shows the differences between the procedures. The script drew 15 adults from each of six NHANES age bands (seed 1) and compared all 15 pairs with Welch's $t$-test of The Welch t-Test, since the variances grow with age. The populations' mean systolic pressures are 113.18 (ages 18–29), 116.99 (30–39), 122.04 (40–49), 130.10 (50–59), 134.25 (60–69) and 139.79 mm Hg (70 and over); the samples' are 110.1, 116.9, 127.2, 139.5, 131.9 and 137.3. The seven smallest p-values, with the thresholds of the step-down procedures:
| Rank | Pair | $p$ | Holm $\alpha/(m - r + 1)$ | Shaffer $\alpha/t_r$ |
|---|---|---|---|---|
| 1 | 18–29 vs 50–59 | 0.0001 | 0.0033 | 0.0033 |
| 2 | 18–29 vs 60–69 | 0.0001 | 0.0036 | 0.0050 |
| 3 | 18–29 vs 70+ | 0.0003 | 0.0038 | 0.0050 |
| 4 | 30–39 vs 50–59 | 0.0019 | 0.0042 | 0.0050 |
| 5 | 18–29 vs 40–49 | 0.0033 | 0.0045 | 0.0050 |
| 6 | 30–39 vs 70+ | 0.0055 | 0.0050 | 0.0050 |
| 7 | 30–39 vs 60–69 | 0.0096 | 0.0056 | 0.0071 |
The other eight p-values run from 0.0874 to 0.7876. Holm's and Shaffer's procedures both stop at rank 6 and reject five pairs. Shaffer's divisor at rank 6 is $t_6 = 10$, the same as Holm's $m - r + 1 = 10$. After five rejections, ten true pairs are still possible: five of the six groups could share a mean, with the sixth different from all of them.
The closure over partitions rejects all seven pairs with $p \le 0.05$, because it looks at which pairs the data contradict and not only at how many hypotheses might be true. For 30–39 against 70+, every partition with ten equal pairs that contains this pair also makes equal a pair with p-value 0.0019 or less, far below its level $0.05/10$. The hardest partition containing the pair puts 30–39, 40–49, 60–69 and 70+ together: six equal pairs, whose smallest p-value is 0.0055, below $0.05/6 = 0.0083$. For 30–39 against 60–69, the hardest partition has blocks {30–39, 40–49, 60–69} and {50–59, 70+}. It makes four pairs equal, the smallest of their p-values is 0.0096, and $0.0096 \le 0.05/4 = 0.0125$.
Error rates and power, measured
How much do the refinements gain, and do they hold their level? The script drew samples of six groups of 15 from the NHANES age bands, 1,000 samples for each of 30 seeds, and reports medians with interquartile ranges in parentheses.
In the first design three groups came from the adults aged 18–29 and three from those aged 60–69 (seeds $600 + s$). The six pairs within a band are true null hypotheses, because their groups are drawn from the same population; the nine pairs across bands are false.
| Procedure | FWER | Share of the 9 differences detected |
|---|---|---|
| Unadjusted, each at 0.05 | 0.2215 (0.2123 to 0.2258) | 0.9412 |
| Holm | 0.0270 (0.0240 to 0.0320) | 0.6791 |
| Shaffer | 0.0305 (0.0260 to 0.0358) | 0.6984 |
| Closure over partitions | 0.0385 (0.0323 to 0.0445) | 0.7649 |
In the second design the six groups came from the six age bands (seeds $700 + s$), so every pair differs, though some only by a few mm Hg. The share of the 15 differences detected was 0.4917 unadjusted, 0.2487 for Holm's procedure, 0.2618 for Shaffer's and 0.2903 for the closure.
In both designs the share detected rose from Holm's procedure to Shaffer's to the closure. Holm's correction cost 0.262 of the unadjusted share in the first design and the closure won back 0.086 of it; in the second the cost was 0.243 and the closure won back 0.042. All three protected procedures stayed below 0.05 in the partial null design, the closure closest to it. These are observations for these designs and sample sizes; Theorems 6 and 7 are what guarantee the level in general.
Three groups, four groups, and Fisher's least significant difference
The course page shows that running unadjusted pairwise $t$-tests only after a significant $F$ test, Fisher's least significant difference procedure, protects the family-wise error rate with three groups, and shows it failing with eight groups in two clusters of four. Closed testing explains why three groups are safe, and a short argument shows that four are already too many.
Theorem 8. With three groups, Fisher's least significant difference procedure is the closed testing procedure whose local test of the complete null hypothesis is the $F$ test and whose local test of each single pairwise hypothesis is its $t$-test at level $\alpha$.
Proof. With three groups, any intersection of two or more pairwise hypotheses asserts that all three means are equal: two different pairs among three groups share a group, and transitivity joins the third. So the intersection hypotheses are the three single pairs and the complete null $\mu_1 = \mu_2 = \mu_3$. The closure rejects a pair exactly when the local test of the complete null rejects and the pair's own test rejects, which is Fisher's rule. The $F$ test has level $\alpha$ under the complete null, and each $t$-test has level $\alpha$ under its own hypothesis, as the course pages prove. Theorem 1 gives family-wise control. $\blacksquare$
With four groups, the intersection $\mu_1 = \mu_2$ and $\mu_3 = \mu_4$ is a hypothesis of its own, a partition with two blocks, and it is not the complete null. Fisher's procedure tests it with the $F$ test of the complete null, which is not a level-$\alpha$ test of it: when the two blocks differ, the $F$ test rejects with high probability even though the partition hypothesis is true. The course page's example of eight groups in two clusters of four is this case, and there the procedure's family-wise error rate reached 35.6%. An invalid local test does not by itself prove that the procedure exceeds its level, so here is the argument for four groups.
Proposition 2. With four normal groups of equal size and common variance, $\mu_1 = \mu_2 = 0$ and $\mu_3 = \mu_4 = \delta$, the family-wise error rate of Fisher's procedure exceeds $\alpha$ once $\delta$ is large enough.
Proof. The true null pairs are (1, 2) and (3, 4). Let $A_{12}$ and $A_{34}$ be the events that their unadjusted $t$-tests reject. The procedure makes a false rejection when the $F$ test rejects and $A_{12} \cup A_{34}$ occurs, so its error rate is at least $P(A_{12} \cup A_{34}) - P(F \text{ does not reject})$. As $\delta$ grows the second term tends to 0, while the first does not depend on $\delta$. Given the pooled standard deviation $s_W$, the differences $\bar x_1 - \bar x_2$ and $\bar x_3 - \bar x_4$ are independent normal variables with mean 0, independent of $s_W$, so $P(A_{12} \cap A_{34}) = E[g(s_W)^2]$, where $g(s) = P(A_{12} \mid s_W = s)$, and $E[g(s_W)] = \alpha$. Since $0 < g < 1$, $E[g^2] < E[g] = \alpha$, and $$P(A_{12} \cup A_{34}) = 2\alpha - E[g(s_W)^2] > \alpha.$$ So for large $\delta$ the error rate is above $\alpha$. Since a false rejection needs $A_{12}$ or $A_{34}$, the error rate is also at most $P(A_{12} \cup A_{34})$, so as $\delta$ grows it tends to $2\alpha - E[g(s_W)^2]$. $\blacksquare$
For the course's design, 10 adults per group and 36 degrees of freedom, the script computes the limit $2\alpha - E[g^2] = 0.0967$ at $\alpha = 0.05$: nearly twice the nominal level. The closure over partitions repairs the procedure by testing such intersections with tests that are valid for them.
How many of the discoveries are false?
Closed testing rejects intersections as well as single hypotheses. Goeman and Solari (2011) observed that those rejections answer a question family-wise control does not. For any set of hypotheses a reader chooses to call discoveries, how many of them can be true nulls?
For a set $R$ of hypotheses, let $t(R)$ be the size of the largest subset $I \subseteq R$ whose intersection hypothesis $H_I$ is not rejected by the closure, with $t(R) = 0$ if every subset is rejected.
Theorem 9 (Goeman and Solari, 2011). With probability at least $1 - \alpha$, simultaneously for every set $R$, the number of true null hypotheses in $R$ is at most $t(R)$.
Proof. Let $E$ be the event that the local test does not reject $H_{\mathcal{I}_0}$; it has probability at least $1 - \alpha$ (if no hypothesis is true, every bound holds trivially). On $E$, the closure rejects no $H_I$ with $I \subseteq \mathcal{I}_0$, because $\mathcal{I}_0$ contains $I$ and is not rejected locally. For any $R$, the set $R \cap \mathcal{I}_0$ is such an $I$, so it is not rejected by the closure, and its size, the number of true nulls in $R$, is at most $t(R)$. The event $E$ does not depend on $R$, so the bound holds for all $R$ at once. $\blacksquare$
Because the bound holds simultaneously, $R$ may be chosen after seeing the data, for instance as "the hypotheses with $p \le 0.05$", without losing the guarantee. Selecting after looking is the subject of Selective Inference and the Garden of Forking Paths; here the simultaneity makes it harmless.
For the pairwise family the closure fails to reject $H_I$ exactly when some intersection containing $I$ escapes its local test, and every intersection is the hypothesis of a partition. So $t(R)$ is the largest number of pairs of $R$ that a single partition makes equal, among the partitions whose own local test does not reject. On the six-group sample, the seven pairs with $p \le 0.05$ have $t = 0$: with 95% confidence none of them is a true null, which agrees with the closure rejecting all seven. For all 15 pairs, $t = 7$. The partition with the most equal pairs whose local test does not reject is {18–29, 30–39}, {40–49, 50–59, 60–69, 70+}, with $1 + 6 = 7$ pairs whose smallest p-value, 0.0874, exceeds $0.05/7$. So at least $15 - 7 = 8$ of the 15 pairs differ, with 95% confidence, although only seven have been identified individually. The bound says how many differences exist even where it cannot say which.
Example (the Benjamini–Hochberg discoveries are not all false). Take Simes local tests, and let $R$ be the $k \ge 1$ hypotheses rejected by the Benjamini–Hochberg procedure at level $\alpha$, so that the $k$-th smallest p-value is at most $k\alpha/m$. Any set $J \supseteq R$ of size $j \le m$ contains those $k$ p-values, so its $k$-th smallest is at most $k\alpha/m \le k\alpha/j$, and Simes' test rejects $H_J$. So the closure rejects $H_R$ and $t(R) < k$. When Simes' test is valid for the true nulls (for example, when their p-values are independent), Theorem 9 applies, and with confidence $1 - \alpha$ at least one of the Benjamini–Hochberg discoveries is a true discovery. Goeman and Solari (2011, §4.2) note the same correspondence. It is a modest statement next to the false discovery rate, but it holds simultaneously with every other bound of Theorem 9.
What is proved here, what is cited, what the experiment shows
Proved here. The closure principle (Theorem 1). The shortcut for symmetric, monotone local tests (Lemma 1). Holm's procedure as the closure of Bonferroni tests (Theorem 2), Hochberg's rejections as a subset of the Simes closure (Theorem 3), the fixed-sequence procedure as a closure (Theorem 4). The possible numbers of true pairwise hypotheses and their recursion (Theorem 5), the error control of Shaffer's procedure (Theorem 6) and of the closure over partitions (Theorem 7). Fisher's least significant difference with three groups as a closed test (Theorem 8), and its failure with four (Proposition 2). Shaffer's rejections as a subset of the closure over partitions (Proposition 1). The simultaneous bound on true nulls in any set (Theorem 9), and its consequence for the Benjamini–Hochberg discoveries. The level of Simes' test under independence is proved in The False Discovery Rate and the Benjamini–Hochberg Procedure.
Cited. The principle's origin (Marcus, Peritz and Gabriel, 1976), as Goeman and Solari (2011) credit it; Shaffer's (1986) and Bergmann and Hommel's (1988) procedures, as García and Herrera (2008) describe them; Hommel's (1988) procedure as the closure of Simes' tests, from his abstract; Hochberg's (1988) procedure by its published description; Sarkar's (1998) conditions for Simes' test under dependence; Goeman and Solari's (2011) bound, here reproved.
What the experiments show. The brute-force check of Lemma 1 and Theorems 2 to 4 on 2,400 random vectors. The decisions of each procedure on the four and six age groups. The error rates and power in the two simulated designs drawn from NHANES, which are observations for those designs and that sample size; the check of Proposition 1 in every simulated sample; and the limit 0.0967 of Proposition 2 for groups of 10, by numerical integration.
Worked problems
The four problems are posed here, on small exact cases and on the article's NHANES samples; the article's script checks each number.
Problem 1 (three p-values by hand). Three hypotheses have independent p-values 0.02, 0.03 and 0.06. Compute the closure with Bonferroni local tests and with Simes local tests at $\alpha = 0.05$, and compare with Hochberg's procedure.
Solution. The Bonferroni closure rejects nothing; the Simes closure rejects $H_1$ only; Hochberg's procedure rejects nothing.
Bonferroni: the top intersection $H_1 \cap H_2 \cap H_3$ needs $\min p \le 0.05/3 = 0.0167$, and $0.02$ is above it. Every hypothesis lies under the top, so nothing is rejected, as Holm's procedure says.
Simes: the top intersection has sorted p-values 0.02, 0.03, 0.06 and critical values $0.0167, 0.0333, 0.05$. The second p-value, 0.03, is below 0.0333, so it is rejected. For the pairs, the critical values are 0.025 and 0.05. $H_1 \cap H_2$ (0.02, 0.03): $0.02 \le 0.025$, rejected. $H_1 \cap H_3$ (0.02, 0.06): $0.02 \le 0.025$, rejected. $H_2 \cap H_3$ (0.03, 0.06): $0.03 > 0.025$ and $0.06 > 0.05$, not rejected. Singly, $H_1$ and $H_2$ are rejected at 0.05 and $H_3$ is not. $H_1$ is the only hypothesis whose four intersections are all rejected. $H_2$ fails at $H_2 \cap H_3$.
Hochberg: $p_{(3)} = 0.06 > 0.05$, $p_{(2)} = 0.03 > 0.025$, $p_{(1)} = 0.02 > 0.0167$: nothing. The shortcut of Lemma 1 for $H_1$ checks only $\{1\}$, $\{1, 3\}$ and $\{1, 2, 3\}$, the sets filled with the largest other p-values, and these are the ones that decide it. With 0.04 in place of 0.06, Hochberg's procedure rejects all three hypotheses at its first step, and so does the Simes closure, by Theorem 3.
Problem 2 (five groups). Find $S(5)$ from the recursion and Shaffer's divisors $t_1, \ldots, t_{10}$ for all pairwise comparisons of five groups.
Solution. $S(5) = \{0, 1, 2, 3, 4, 6, 10\}$ and the divisors are $10, 6, 6, 6, 6, 4, 4, 3, 2, 1$.
With $S(0) = S(1) = \{0\}$, the recursion gives $S(2) = \{0, 1\}$, $S(3) = \{0, 1, 3\}$ and $S(4) = \{0, 1, 2, 3, 6\}$. Then, by the size $j$ of the block containing the first group (writing $c + S$ for the set $S$ with $c$ added to each member), $$S(5) = S(4) \cup (1 + S(3)) \cup (3 + S(2)) \cup (6 + S(1)) \cup (10 + S(0)) = \{0,1,2,3,6\} \cup \{1,2,4\} \cup \{3,4\} \cup \{6\} \cup \{10\}.$$ The same set comes from the block sizes of the seven kinds of partition of five groups: $5$ gives 10 pairs, $4+1$ gives 6, $3+2$ gives $3 + 1 = 4$, $3+1+1$ gives 3, $2+2+1$ gives 2, $2+1+1+1$ gives 1 and $1+1+1+1+1$ gives 0. So 5, 7, 8 and 9 true pairs are impossible.
There are $m = 10$ pairs, and $t_r$ is the largest member of $S(5)$ that is at most $11 - r$. For $r = 1$ the bound is 10, so $t_1 = 10$. For $r = 2$ to 5 the bounds are 9, 8, 7 and 6, and the largest member at most each is 6. For $r = 6$ and 7 the bounds are 5 and 4, giving 4. For $r = 8$, 9 and 10 the bounds are 3, 2 and 1, which are members themselves. Holm's divisors would be $10, 9, \ldots, 1$; Shaffer's are smaller at steps 2, 3, 4 and 6 and equal at the others.
Problem 3 (three groups under Fisher's procedure). Take the course's samples of the three older age groups, 30–44, 45–59 and 60+. The one-way $F$ statistic is 4.993 on 2 and 27 degrees of freedom ($p = 0.0143$), and with $s_W^2 = 198.15$ the pairwise $t$ statistics are 0.582 (30–44 vs 45–59), 2.981 (30–44 vs 60+) and 2.399 (45–59 vs 60+), with p-values 0.5651, 0.0060 and 0.0236. Apply Fisher's procedure, and compare with Holm's.
Solution. Both reject the same two hypotheses: 60+ differs from 30–44 and from 45–59.
By Theorem 8 the closure has four hypotheses: the complete null, rejected by the $F$ test ($0.0143 \le 0.05$), and the three pairs, each rejected when its own p-value is at most 0.05. So 30–44 vs 60+ and 45–59 vs 60+ are rejected, and 30–44 vs 45–59 is not. Holm's procedure compares 0.0060 with $0.05/3 = 0.0167$ and 0.0236 with $0.05/2 = 0.025$, rejecting both, and then 0.5651 with 0.05, retaining it. Here Fisher's procedure, valid because there are three groups, and Holm's agree; Fisher's needs the $F$ test to reject first, Holm's needs the smaller thresholds.
Problem 4 (how many of the fifteen pairs differ). On the six-group sample, the closure over partitions rejected seven pairs. Show that, with 95% confidence, at least eight of the fifteen pairs differ.
Solution. By Theorem 9, with 95% confidence the number of true pairs among all 15 is at most $t$, the largest number of equal pairs in a partition whose local test does not reject. We show $t = 7$.
First, $t \ge 7$. The partition {18–29, 30–39}, {40–49, 50–59, 60–69, 70+} makes $1 + 6 = 7$ pairs equal, and the smallest of their p-values is 0.0874, above $0.05/7 = 0.0071$. So its local test does not reject.
Second, $t \le 7$. The members of $S(6)$ above 7, from the list after Theorem 5, are 10 and 15, so a partition with more than seven equal pairs either has a block of five groups (10 pairs) or puts all six together (15 pairs). All six together includes the pair 18–29 vs 50–59 with $p = 0.0001 \le 0.05/15$. A block of five omits one group. If it omits 18–29, it contains 30–39 vs 50–59 with $p = 0.0019 \le 0.05/10$; if it omits any other group, it contains 18–29 and at least one of 50–59 and 60–69, with $p = 0.0001$. So every such local test rejects.
Hence $t = 7$, and at least $15 - 7 = 8$ pairs differ, with 95% confidence. The seven pairs rejected individually are among them, and an eighth exists, though the data cannot say which. The script confirms $t = 7$ by checking all 202 partitions.
Summary
- Closed testing rejects a hypothesis when every intersection containing it is rejected by a level-$\alpha$ local test; the family-wise error rate is then at most $\alpha$, by a one-line argument that uses only the test of the true intersection.
- For symmetric, monotone local tests the closure needs $m$ tests per hypothesis. Bonferroni local tests give Holm's procedure; Simes local tests give Hommel's, which rejects everything Hochberg's does and needs Simes' test to be valid.
- Ordering the hypotheses in advance gives the fixed-sequence procedure, a closure that spends no level on multiplicity.
- Logical relations shrink the closure. For pairwise comparisons the possible numbers of true hypotheses are $S(k)$; Shaffer's procedure and the closure over partitions use them, and on the four age groups both reject the difference Holm's procedure missed.
- Fisher's least significant difference is a closed test with three groups; with four, its gate is not a valid test of every intersection.
- The closure's rejected intersections bound the number of false discoveries in any chosen set, simultaneously.
What we have learned
Every procedure in this article is a closed test or a shortcut of one, except Hochberg's and Shaffer's, each of which rejects no more than a closed test (Theorem 3 and Proposition 1); Holm's procedure from the course page is a closed test too. The closure principle explains both why they work and how to improve them: choose better local tests, or remove intersections that cannot occur. On the six age groups the closure over partitions found seven differences where Holm's procedure found five, and its error rate stayed below 0.05 in the simulated partial null.
Family-wise control remains strict when the hypotheses number in thousands. The false discovery rate of The False Discovery Rate and the Benjamini–Hochberg Procedure relaxes it, and q-Values and the Share of True Nulls estimates how many of the hypotheses are true at all.
References
- Benjamini, Y., and Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B, 57(1), 289–300. https://doi.org/10.1111/j.2517-6161.1995.tb02031.x record · 6 pages
- Bergmann, B., and Hommel, G. (1988). Improvements of general multiple test procedures for redundant systems of hypotheses. In P. Bauer, G. Hommel and E. Sonnemann (Eds.), Multiple Hypothesenprüfung / Multiple Hypotheses Testing (pp. 100–115). Springer. https://doi.org/10.1007/978-3-642-52307-6_8 (Cited as García and Herrera, 2008, describe it.) record
- García, S., and Herrera, F. (2008). An extension on "Statistical comparisons of classifiers over multiple data sets" for all pairwise comparisons. Journal of Machine Learning Research, 9, 2677–2694. https://www.jmlr.org/papers/v9/garcia08a.html record
- Goeman, J. J., and Solari, A. (2011). Multiple testing for exploratory research. Statistical Science, 26(4), 584–597. https://doi.org/10.1214/11-STS356 record
- Hochberg, Y. (1988). A sharper Bonferroni procedure for multiple tests of significance. Biometrika, 75(4), 800–802. https://doi.org/10.1093/biomet/75.4.800 record · 2 pages
- Hommel, G. (1988). A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika, 75(2), 383–386. https://doi.org/10.1093/biomet/75.2.383 record
- Marcus, R., Peritz, E., and Gabriel, K. R. (1976). On closed testing procedures with special reference to ordered analysis of variance. Biometrika, 63(3), 655–660. https://doi.org/10.1093/biomet/63.3.655 record · 2 pages
- National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018: Demographic Variables and Sample Weights (DEMO_J) and Blood Pressure (BPX_J) data files. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 95 pages
- Sarkar, S. K. (1998). Some probability inequalities for ordered MTP2 random variables: a proof of the Simes conjecture. The Annals of Statistics, 26(2), 494–504. https://doi.org/10.1214/aos/1028144846 record
- Simes, R. J. (1986). An improved Bonferroni procedure for multiple tests of significance. Biometrika, 73(3), 751–754. https://doi.org/10.1093/biomet/73.3.751 record · 2 pages
- Shaffer, J. P. (1986). Modified sequentially rejective multiple test procedures. Journal of the American Statistical Association, 81(395), 826–831. https://doi.org/10.1080/01621459.1986.10478341 record