Multiple Testing

The theory of testing many hypotheses at once, beyond the family-wise corrections of the course. The false discovery rate and the Benjamini–Hochberg procedure with its proof; the closure principle, from which Holm's procedure and its refinements follow; Storey's q-values and the estimated share of true null hypotheses; and what selecting hypotheses, models or reports after seeing the data does to p-values and intervals. Worked on all pairwise comparisons of NHANES age groups and on thousands of term–category tests in the Reuters stories.

A study that tests one hypothesis at level 0.05 has a 5% chance of a false alarm when the hypothesis is true. A study that tests a thousand has a near certainty of several. Multiple testing is the theory of what can still be promised in that situation, and of the procedures that keep the promise. The course page Multiple Comparisons and the Bonferroni Correction answers the question for a handful of comparisons among group means: it controls the family-wise error rate, the probability of any false rejection, with the Bonferroni correction, Holm's step-down procedure and Tukey's studentized range. This unit takes the subject further in two directions. When the hypotheses number in the thousands and many are expected to be false, it controls a weaker and more useful quantity, the false discovery rate. And when family-wise control is the right standard, it derives Holm's procedure and its refinements from a single principle, closed testing, which also gives procedures more powerful than Holm's.

What it assumes. The units on Hypothesis Testing and Analysis of Variance, in particular Type I and Type II Errors and the course page on multiple comparisons, whose union bound and proof of Holm's procedure the unit uses by link. The probability needed is the union bound, conditional probability and independence; nothing beyond Random Variables and Distribution Functions and Conditional Expectation. Permutation Tests supplies the label-permutation argument that one experiment uses to build a world in which every null hypothesis is true.

What is proved and what is cited. Proved across the unit: that the false discovery rate never exceeds the family-wise error rate and equals it when every null hypothesis is true; that the Benjamini–Hochberg procedure has false discovery rate exactly $\pi_0\alpha$ for independent tests with uniform null p-values and at most that for valid ones; Simes' test as its special case; the Benjamini–Yekutieli correction for arbitrary dependence, with an example in which the uncorrected procedure reaches $1.5\alpha$; the closure principle and its strong control of the family-wise error rate; Holm's, Hommel's and the fixed-sequence procedures as closed tests, and Hochberg's and Shaffer's as procedures that reject no more than a closed test; the failure of Fisher's least significant difference with four groups; the simultaneous upper bound that closed testing gives on the number of false discoveries in any set of hypotheses; the positive false discovery rate as a posterior probability in the two-groups model, the conservativeness of Storey's estimate of $\pi_0$ with its exact bias, and the equality of q-values and Benjamini–Hochberg adjusted p-values when $\pi_0$ is taken as one; the winner's curse, and the truncated-normal pivot behind selective p-values and intervals, with the polyhedral lemma for the best of several models. Cited rather than proved: the control of the false discovery rate under positive regression dependence (Benjamini and Yekutieli, 2001), the validity of Simes' test under positive dependence, Efron's empirical null, and the original papers' simulation studies. Where a result is measured rather than proved, the articles say so and give the experiment.

Notation. The family has $m$ null hypotheses $H_1, \ldots, H_m$ with p-values $p_1, \ldots, p_m$. Ordered, they are $p_{(1)} \le p_{(2)} \le \cdots \le p_{(m)}$, and $H_{(i)}$ is the hypothesis with the $i$-th smallest. The true null hypotheses form the unknown index set $\mathcal{I}_0$, with $m_0 = |\mathcal{I}_0|$ members and share $\pi_0 = m_0/m$. A p-value is valid if $P(p_i \le u) \le u$ for every $u$ between 0 and 1 whenever $H_i$ is true. A procedure rejects $R$ hypotheses, of which $V$ are true nulls (false discoveries) and $S = R - V$ are not. The false discovery proportion is $V/\max(R, 1)$, the false discovery rate (FDR) is its expectation, and the family-wise error rate (FWER) is $P(V \ge 1)$. The target level is written $\alpha$ for both error rates. The letter $q$ is kept for Storey's q-values. Adjusted p-values are written $\tilde p_i$.

Data. Two running examples carry the unit. The first is all pairwise comparisons of systolic blood pressure among age groups of the adults in the National Health and Nutrition Examination Survey (NHANES) 2017–2018: the four groups of 10 from the course's One-Way Analysis of Variance, six age groups of 15, and simulations that draw groups from the survey's age bands, so that some pairs of groups come from the same population and the true null hypotheses are known. The second is the Reuters-21578 news stories of the course's Feature Selection article: for each of the 18,135 terms of the training stories, a test of whether the term is more common in stories about grain, thousands of tests on one dataset. The selective-inference article adds six language models scored on random samples of questions from the HELM MMLU benchmark of Statistics of Model Evaluation. Every number is computed by a script in tools/experiments/statistics/.

Articles

Error rates for many hypotheses

  • The False Discovery Rate and the Benjamini–Hochberg Procedure: the false discovery rate defined and compared with the family-wise error rate; the step-up procedure and its adjusted p-values; the proof that its false discovery rate is exactly $\pi_0\alpha$ under independence, with Simes' test as a corollary; dependence, with the Benjamini–Yekutieli correction proved and an example that attains its bound; on the Reuters terms, how many associations each procedure finds, how often it finds any in a world of permuted labels, and how many of its findings recur in the test stories.
  • q-Values and the Share of True Nulls: the positive false discovery rate as the posterior probability of the null in the two-groups model; the q-value as a tail-area posterior probability, and its equality with the Benjamini–Hochberg adjusted p-value when the share of true nulls is taken as one; the local false discovery rate, whose average is the tail rate; Storey's estimate of the share of true nulls, proved conservative, with its bias, variance and behavior under dependence; and the adaptive procedure, on two-sided Fisher tests of 878 Reuters terms against grain.

Family-wise control from one principle

  • Closed Testing: the closure principle and its proof; Holm's procedure as the closure of Bonferroni tests and the shortcut that makes closures computable; Simes-based procedures (Hommel's and Hochberg's); logically related hypotheses, where all pairwise comparisons admit Shaffer's and Bergmann and Hommel's improvements on Holm; fixed sequences; Fisher's least significant difference as a closed test with three groups, and its failure with four; and the simultaneous bound on the number of false discoveries in any chosen set.

When the hypotheses are chosen after looking

  • Selective Inference and the Garden of Forking Paths: what choosing a result after seeing the data does to its estimate, p-value and interval; the winner's curse proved; the file-drawer effect and the garden of forking paths measured on NHANES with random labels; selective p-values and intervals from the truncated-normal pivot, for a significant result and, through the polyhedral lemma, for the best of several models; simultaneous intervals and sample splitting as the other repairs, compared on six language models scored on MMLU.

Where the unit leads

The false discovery rate is the standard error rate for screening: genes, features, and terms for a classifier, which is where Feature Selection meets this unit. Closed testing is the standard for confirmatory studies with several endpoints, and for comparing many systems: Comparing Systems Across Many Tasks uses Holm's procedure over all pairs of systems. When the endpoints are several measurements on the same subjects, Hotelling's T-Squared Test offers one test of the mean vector in place of corrected separate tests, and compares the power of the two. The pairwise refinements of Closed Testing apply when each pairwise hypothesis is the equality of one parameter per system, such as mean scores or mean ranks, so that equality is transitive; the signed-rank hypotheses used there are not of that form. Model choice after looking at the data, the subject of the last article, returns in Prediction Error and Model Choice.

References

Each article lists its own references. The two papers that set the unit's two directions: record