A resampling method answers a question about the sampling distribution of a statistic by recomputing the statistic on many samples made from the data. The bootstrap draws them with replacement from the observations, a permutation test rearranges the observations among groups, and the jackknife leaves one observation out at a time. Each replaces a derivation that may be impossible, such as the standard error of a median or the null distribution of a statistic under an unknown population, with arithmetic that a computer does in seconds. The library's The Bootstrap Method and the course's Significance Testing use these methods as tools. This unit is the theory beneath them: why they work, what they assume, and where they fail.
The unit is organized around two ideas. The first is the plug-in principle: a quantity that depends on the unknown population distribution $F$ is estimated by the same quantity computed from the empirical distribution $\hat F_n$, which puts probability $1/n$ on each observation. The bootstrap is that principle applied to a sampling distribution, and it is consistent when $\hat F_n$ is close to $F$ and the quantity depends smoothly on the distribution. The second is exchangeability: when the null hypothesis makes every rearrangement of the data equally likely, the rearrangements give an exact null distribution with no large-sample approximation at all. The first idea gives intervals that are correct in large samples; the second gives tests that are exact at every sample size.
What it assumes. The course units on Sampling Distributions, Confidence Intervals and Hypothesis Testing, and from the Probability Theory unit the laws of large numbers, the proof of the central limit theorem by characteristic functions, Slutsky's theorem and the delta method, all of which the consistency proofs use by link.
What is proved and what is cited. Proved across the unit: the Glivenko–Cantelli theorem; that the empirical distribution is the nonparametric maximum likelihood estimate; consistency of the bootstrap for the mean, the studentized mean and smooth functions of a mean under a finite variance alone; the failure of the bootstrap for the sample maximum; the exactness of the percentile and BCa intervals under the transformation models that motivate them, and the transformation invariance of both; the exactness of permutation tests under the randomization hypothesis, and the validity of Monte Carlo permutation p-values; the jackknife's bias correction, the Efron–Stein identity, which shows that its sum of squares overestimates on average the variance of the statistic computed from $n - 1$ observations, and its failure for the median; and the results on dependent and clustered data stated in that article. Cited rather than proved: the strong law of large numbers under a finite mean and Lévy's continuity theorem, both cited elsewhere on the site; the Dvoretzky–Kiefer–Wolfowitz inequality; and the second-order accuracy of the BCa interval, which rests on Edgeworth expansions this unit does not develop (for the studentized interval the unit gives only a heuristic and a measurement). Where a result is measured rather than proved, the articles say so and give the experiment.
Notation. The sample is $x_1, \ldots, x_n$, i.i.d. draws from a distribution function $F$; the empirical distribution function is $\hat F_n$ (or $\hat F$ when $n$ is fixed). A parameter is a functional $\theta = T(F)$ with plug-in estimate $\hat\theta = T(\hat F_n)$. A bootstrap sample is $x_1^*, \ldots, x_n^*$, drawn with replacement; its statistic is $\hat\theta^*$, and $B$ Monte Carlo replicates are $\hat\theta^{*1}, \ldots, \hat\theta^{*B}$. $P_*$, $E_*$ and $\operatorname{Var}_*$ are probability, expectation and variance under resampling with the data held fixed. The bootstrap distribution function of $\hat\theta^*$ is $\hat G$. Leave-one-out estimates are $\hat\theta_{(i)}$, with mean $\hat\theta_{(\cdot)}$. Intervals have level $1 - \alpha$ with $\alpha/2$ in each tail, and $z_p = \Phi^{-1}(p)$, so $z_{0.975} = 1.96$. The normal distribution is written in words, or as $N(\mu, \sigma)$ with the standard deviation second, as everywhere on the site.
Data. The running example of the bootstrap articles is high-sensitivity C-reactive protein (CRP) in the 5,146 adults of the National Health and Nutrition Examination Survey 2017–2018 with a measurement. With skewness 7.9, it is the population on which the $t$ interval fails, as The Central Limit Theorem and Confidence Intervals for a Mean show for skewed populations. It is treated as a finite population and sampled with replacement, so every method can be checked against the truth; the running sample is 30 adults. The second example is the area under the ROC curve of HbA1c as a screen for diabetes from ROC Curves for Diagnostic Tests. The other articles use Darwin's maize and Student's sleep data, small samples of NHANES adults and their blood pressures, and weekly death counts. Every number is computed by a script in tools/experiments/statistics/.
Articles
The bootstrap
- The Plug-In Principle and Why the Bootstrap Works: parameters as functions of a distribution; the empirical distribution as the nonparametric maximum likelihood estimate; the Glivenko–Cantelli theorem; when plug-in estimates are consistent; the bootstrap as a plug-in estimate of a sampling distribution, with the exact bootstrap mean and variance; consistency for the mean, the studentized mean and smooth functions of a mean, proved under a finite variance; the sample maximum as a proved failure. On CRP, the bootstrap of the studentized mean was two to three times closer to the true distribution than the normal approximation at samples of 100 and 400, and less at smaller sizes.
- Bootstrap Confidence Intervals: Percentile, Basic, Studentized and BCa: the four intervals derived from what each assumes about the sampling distribution; their exactness under their models and their invariance or lack of it; the BCa formula derived from its transformation model, with the acceleration from the jackknife; first- and second-order accuracy; coverage of all four measured on CRP at sample sizes from 10 to 400, and intervals for the HbA1c AUC.
Tests by rearrangement
- Permutation Tests: the exactness of permutation p-values under the randomization hypothesis, proved; sign flips for paired data and relabelings for two samples; the course's sign, signed-rank, rank-sum and Kruskal–Wallis tests and Fisher's exact test as permutation tests; when the permutation distribution misstates the variance of a difference in means, and the studentized test that repairs it; Monte Carlo p-values and intervals by inverting the test.
Estimates from leaving data out
- The Jackknife: Quenouille's bias correction and Tukey's variance estimate, with what each is proved to do; leaving one out as a derivative, the influence function and the infinitesimal jackknife; the jackknife covariance of least-squares coefficients as a robust variance; the Efron–Stein identity, which shows that the jackknife sum of squares overestimates, on average, the variance of the statistic computed from $n - 1$ observations; and its proved failure for the median.
Data that are not independent
- Resampling Dependent and Clustered Data: why resampling single observations fails for dependent data; block bootstraps for series and the cluster bootstrap, with its relation to the cluster-robust variance.
Where the unit leads
The bootstrap reappears wherever an estimate is too complicated for a formula. Statistics of Model Evaluation uses the paired and cluster bootstraps to compare models on the same benchmark questions. Multiple Testing uses permutation and resampling to control error rates across many tests at once. Prediction Error and Model Choice treats cross-validation, the resampling method for prediction error, whose bootstrap competitors are worked in The Bootstrap Method.
References
Each article lists its own references. A text that treats the unit's subject as a whole, at the level of these articles and with Efron's own account of the intervals: record
- Efron, B., and Hastie, T. (2016). Computer Age Statistical Inference: Algorithms, Evidence, and Data Science. Cambridge University Press. Chapter 10, "The Jackknife and the Bootstrap", and Chapter 11, "Bootstrap Confidence Intervals". record · 3 pages