The running sample of the Resampling Methods unit is thirty adults' C-reactive protein (CRP), drawn from the National Health and Nutrition Examination Survey 2017–2018 (National Center for Health Statistics, 2020), with mean 5.68 mg/L and standard error $s/\sqrt{30} = 1.592$. The $t$ interval for the population mean is 2.42 to 8.93. Yet for this population the studentized mean of thirty does not have anything like Student's distribution: its true 2.5% and 97.5% points are −5.06 and 1.29, not ±2.05 (The Plug-In Principle and Why the Bootstrap Works). Over thousands of samples of 30, the $t$ interval misses the true mean far more often than 5% of the time, and almost always on the same side. Four bootstrap intervals computed from the same thirty values run from 3.11 to 9.15, 2.21 to 8.25, 3.31 to 12.17 and 3.58 to 10.76. Which of them, if any, deserves the name "95% interval"?
A bootstrap confidence interval is an interval for a parameter $\theta$ whose endpoints are read off the bootstrap distribution of an estimate $\hat\theta$, instead of off a normal or $t$ table. The four standard constructions differ in which bootstrap distribution they read and how. They agree in large samples, where they cover with probability tending to the nominal level (proved below for three of them for a mean; the BCa interval's accuracy is cited). They disagree in small samples from skewed populations, which is where the choice matters.
First we set out what an interval has to get right, and the Monte Carlo conventions. Then we derive the basic interval from the bootstrap distribution of the error $\hat\theta - \theta$, the percentile interval from a transformation model, and the studentized interval from a resampled pivot, and prove that all three have the right coverage in large samples for a mean. Next we derive the BC and BCa intervals in full from Efron's model, with the acceleration constant from the jackknife. Then we state what is known about their accuracy, measure the coverage of all of them on CRP at six sample sizes, and apply them to the area under an ROC curve. The article assumes The Plug-In Principle and Why the Bootstrap Works, whose consistency theorems it uses, and Confidence Intervals for a Mean, whose $t$ interval is the baseline throughout. The library's The Bootstrap Method computes the percentile interval in practice; this article is its theory.
What an interval has to get right
An interval $[L, U]$ computed from the data has coverage $P(L \le \theta \le U)$, and nominal level $1 - \alpha$ if it is meant to cover with probability $1 - \alpha$. The intervals here are equal-tailed: they aim for $P(\theta < L) = \alpha/2$ and $P(\theta > U) = \alpha/2$ separately. Overall coverage can hide a lopsided failure. An interval that misses on the left 0.5% of the time and on the right 9.5% of the time has coverage 90%, and it also places the parameter wrongly: it says "probably not above $U$" with far more confidence than the data justify. The coverage tables below therefore report the misses on each side.
Every construction starts from a root whose distribution is approximately known: $\hat\theta - \theta$, a studentized version $(\hat\theta - \theta)/\widehat{\operatorname{se}}$, or the estimate on a transformed scale. If the root's quantiles $k_p$ were known exactly,
$$P\big(k_{\alpha/2} \le \hat\theta - \theta \le k_{1-\alpha/2}\big) = 1 - \alpha \quad\text{would give}\quad \big[\hat\theta - k_{1-\alpha/2},\ \hat\theta - k_{\alpha/2}\big],$$
an interval with exact coverage, obtained by solving the two inequalities for $\theta$. The order of the endpoints flips: the upper quantile of the root sets the lower endpoint. Everything below is a way of estimating such quantiles by the bootstrap.
Monte Carlo conventions. From $B$ bootstrap replicates $\hat\theta^{*1}, \ldots, \hat\theta^{*B}$, the bootstrap distribution function is estimated by $\hat G_B(t) = \#\{b : \hat\theta^{*b} \le t\}/B$, and its $p$-quantile by $\hat\theta^*_{(p)}$, the $\lceil pB\rceil$-th smallest replicate (Efron and Hastie, 2016, eqs. 11.16 to 11.18, define the bootstrap percentiles from $\hat G$ in the same way). With $B = 2{,}000$ and $\alpha = 0.05$ the endpoints are the 50th and 1,950th smallest replicates. Efron and Hastie (2016, Sections 11.2 and 11.4) recommend $B$ on the order of 2,000 for intervals, against 200 or fewer for a standard error, because the endpoints depend on the tails. The coverage experiment and the AUC example use $B = 2{,}000$. The intervals quoted for the running sample use $B = 200{,}000$ resamples (seed 2026), so that they are close to the ideal bootstrap intervals and their Monte Carlo error does not distract. With $B = 2{,}000$ that error is visible: five seeds (1 to 5) gave studentized upper limits from 11.66 to 12.61 and BCa upper limits from 10.16 to 11.21, against 12.17 and 10.76 from 200,000 resamples. The script checked its percentile, basic and BCa limits for the running sample against SciPy's scipy.stats.bootstrap (version 1.15.2) with the same number of resamples, which gave the same limits to two decimals.
The basic interval: resampling the error
Here is the idea in one sentence. In the bootstrap world $\hat\theta$ plays the part of $\theta$, so the spread of $\hat\theta^* - \hat\theta$ around 0 estimates the spread of $\hat\theta - \theta$ around 0, and the exact-pivot recipe above can be applied with the bootstrap's quantiles.
The $p$-quantile of $\hat\theta^* - \hat\theta$ is $\hat\theta^*_{(p)} - \hat\theta$. Substituting it for $k_p$ gives the basic interval
$$\big[\hat\theta - (\hat\theta^*_{(1-\alpha/2)} - \hat\theta),\ \hat\theta - (\hat\theta^*_{(\alpha/2)} - \hat\theta)\big] = \big[2\hat\theta - \hat\theta^*_{(1-\alpha/2)},\ 2\hat\theta - \hat\theta^*_{(\alpha/2)}\big].$$
SciPy calls it the "basic" or "reverse" interval, and the second name describes it: it is the bootstrap percentile range reflected through $\hat\theta$. For the running sample the 2.5% and 97.5% points of the bootstrap means were 3.107 and 9.148, so the basic interval is $2(5.678) - 9.148 = 2.21$ to $2(5.678) - 3.107 = 8.25$.
The reflection is the logic of the method, not a defect. If the bootstrap means spread further above $\bar x$ than below, the basic interval reads that as evidence that $\bar X$ tends to overshoot $\mu$ by more than it undershoots, and it extends the interval further below $\bar x$ than above. For a right-skewed population that is backwards. A sample mean from such a population usually falls a little below $\mu$, because most samples miss the rare large values, and occasionally far above it. To cover $\mu$, an interval should reach further up than down. The basic interval does the opposite, and the coverage table below shows the cost.
The percentile interval: trusting the shape
The percentile interval uses the bootstrap quantiles directly:
$$\big[\hat\theta^*_{(\alpha/2)},\ \hat\theta^*_{(1-\alpha/2)}\big].$$
For the running sample it is 3.11 to 9.15. It reaches further above $\bar x = 5.68$ than below, following the skewness of the bootstrap means. Why should reading the bootstrap distribution directly, without reflecting it, be right? The basic interval's reasoning says it is backwards. Efron's answer is a transformation model, under which the percentile interval is exact on every scale, while the basic interval is exact only on the one scale where the model holds.
Theorem 1 (the percentile interval under a symmetric transformation model). Suppose there is a continuous, strictly increasing function $m$ such that $\hat\phi = m(\hat\theta)$ and $\phi = m(\theta)$ satisfy: $\hat\phi - \phi$ has a continuous distribution $H$, symmetric about 0, the same for every $\theta$. Suppose the bootstrap reproduces the model exactly: given the data, $\hat\phi^* - \hat\phi$ has distribution $H$, where $\hat\phi^* = m(\hat\theta^*)$. Then the ideal percentile limit $\hat G^{-1}(p)$ satisfies $P\big(\theta < \hat G^{-1}(p)\big) = p$ for every $0 < p < 1$, so the percentile interval has coverage exactly $1 - \alpha$, whatever $m$ is.
Here is the idea. On the scale of $\phi$ the percentile interval is $\hat\phi \pm$ (a quantile of $H$), which is exact because $H$ is symmetric; and percentiles do not care which increasing scale they are computed on.
Proof. Step 1: percentiles commute with increasing maps. Let $\hat G$ be the bootstrap distribution function of $\hat\theta^*$ and $\hat K$ that of $\hat\phi^* = m(\hat\theta^*)$. For a continuous, strictly increasing $m$ and continuous distributions, $\hat\theta^* \le t$ exactly when $m(\hat\theta^*) \le m(t)$, so $\hat K(m(t)) = \hat G(t)$, and the $p$-quantiles correspond: $\hat K^{-1}(p) = m\big(\hat G^{-1}(p)\big)$.
Step 2: on the $\phi$ scale. By hypothesis $\hat\phi^* = \hat\phi + V^*$ with $V^*$ distributed as $H$, so $\hat K^{-1}(p) = \hat\phi + h_p$, where $h_p$ is the $p$-quantile of $H$, and $H(h_p) = p$ because $H$ is continuous.
Step 3: coverage. By Steps 1 and 2, $\theta < \hat G^{-1}(p)$ exactly when $\phi < \hat\phi + h_p$, that is, when $\hat\phi - \phi > -h_p$. Since $\hat\phi - \phi$ has the continuous distribution $H$, symmetric about 0, this has probability $1 - H(-h_p) = H(h_p) = p$. $\square$
The theorem needs only that such an $m$ exists, not that anyone knows it (Efron and Hastie, 2016, Section 11.2, make the same argument with a normal $H$). Step 1 also proves the property that makes the percentile interval attractive in practice. The percentile interval is transformation invariant: the percentile interval for $m(\theta)$ is $m$ applied to the percentile interval for $\theta$. Computed for $\ln\mu$ from the same resamples, it is $(\ln 3.107, \ln 9.148) = (1.1336, 2.2135)$ exactly, because the logarithm does not change the order of the replicates. The basic interval is not invariant, because reflecting through $\hat\theta$ does not commute with a nonlinear $m$.
A common justification of the percentile interval, that the bootstrap values vary around $\hat\theta$ roughly as $\hat\theta$ varies around $\theta$, is the basic interval's logic, and it points the other way for skewed data; Theorem 1 is the argument that supports the percentile interval. Theorem 1 does not say that the percentile interval is right for the mean of CRP. It says what would make it right: a scale on which the estimate is unbiased and its error has a fixed symmetric shape. For a skewed mean there is no such scale exactly, and the percentile interval's errors in the table below measure how far from it CRP is. The BCa interval below widens the model.
Example (the two intervals cannot both be right). For the running sample, the percentile interval is 3.11 to 9.15 and the basic interval 2.21 to 8.25, the same length placed differently. Under a symmetric bootstrap distribution they coincide; under a skewed one they move in opposite directions. The basic interval assumes the error $\hat\theta - \theta$ has the skewness of $\hat\theta^* - \hat\theta$; the percentile interval assumes there is a scale on which the error is symmetric, with the skewness of the bootstrap distribution coming from the transformation. For means of the right-skewed CRP population, the coverage experiment below favors the percentile interval at every sample size, by 1.4 to 5.1 points of coverage, although both lose nearly all their misses on the same side.
The studentized interval: resampling a pivot
The basic interval resamples the error $\hat\theta - \theta$, whose spread depends on the unknown $\sigma$. Student's insight was to divide by an estimate of the spread. The studentized root $T = (\hat\theta - \theta)/\widehat{\operatorname{se}}$ is close to a pivot, a quantity whose distribution does not depend on the unknown population. The studentized or bootstrap-$t$ interval keeps Student's form and replaces his table by the bootstrap distribution of
$$T^* = \frac{\hat\theta^* - \hat\theta}{\widehat{\operatorname{se}}^*},$$
where $\widehat{\operatorname{se}}^*$ is the standard error formula applied to the bootstrap sample. With $t^*_{(p)}$ its Monte Carlo quantiles, the exact-pivot recipe gives
$$\big[\hat\theta - t^*_{(1-\alpha/2)}\,\widehat{\operatorname{se}},\ \hat\theta - t^*_{(\alpha/2)}\,\widehat{\operatorname{se}}\big].$$
Efron and Hastie (2016, Section 11.5) give the same construction (their eq. 11.53). For a mean, $\widehat{\operatorname{se}} = s/\sqrt n$ and $\widehat{\operatorname{se}}^* = s^*/\sqrt n$. For the running sample the 2.5% and 97.5% points of $T^*$ were −4.076 and 1.485, against ±2.045 from the $t$ table with 29 degrees of freedom. The interval is
$$\big[5.678 - 1.485 \times 1.592,\ 5.678 + 4.076 \times 1.592\big] = [3.31,\ 12.17].$$
It is long and lopsided: it reaches 6.49 above $\bar x$ and 2.36 below. That is what the true distribution of $T$ calls for, with its 2.5% point at −5.06. A sample whose mean is low usually lacks large values and so also has a small $s$, which makes $T$ very negative; the upper endpoint has to allow for it.
The studentized interval is not transformation invariant: computed on the log scale, with the delta-method standard error $s/(\sqrt n\,\bar x)$ for $\ln\bar X$ and its bootstrap counterpart, and transformed back, it gives 3.37 to 13.13 for the running sample, not 3.31 to 12.17. It needs a formula for the standard error, applied in every resample; when no formula exists, a second level of bootstrap inside each resample can supply one, at $B$ times the cost. And a resample with an unusually small $\widehat{\operatorname{se}}^*$ produces an extreme $T^*$, so the interval can be very long when the sample is small (Efron and Hastie, 2016, Section 11.5, note the lack of invariance and the need for a standard error formula).
All three are right in large samples
For the mean, the consistency theorems of The Plug-In Principle give all three intervals the right coverage in the limit.
Theorem 2 (large-sample coverage for a mean). Let $X_1, X_2, \ldots$ be i.i.d. with mean $\mu$ and variance $0 < \sigma^2 < \infty$, and let the intervals be computed from the ideal bootstrap distributions ($B$ infinite). Then the basic, percentile and studentized intervals for $\mu$ at level $1 - \alpha$ each have $P(\mu < L) \to \alpha/2$ and $P(\mu > U) \to \alpha/2$, so their coverage tends to $1 - \alpha$.
Here is the idea. Each endpoint is $\bar X_n$ plus a bootstrap quantile divided by $\sqrt n$. The bootstrap quantiles converge to normal quantiles with probability 1, and then Slutsky's theorem turns each one-sided miss into a normal probability.
Proof. Step 1: bootstrap quantiles converge. Let $\hat H_n$ be the bootstrap distribution function of $\sqrt n(\bar X^* - \bar X_n)$ and $\hat h_p$ its $p$-quantile. By Theorem 5 of the plug-in article, with probability 1, $d_K(\hat H_n, K_n) \to 0$, where $K_n$ is the distribution function of $\sqrt n(\bar X_n - \mu)$; and $K_n$ converges uniformly to $\Phi(\cdot/\sigma)$ by the central limit theorem. So $d_K(\hat H_n, \Phi(\cdot/\sigma)) \to 0$ with probability 1. The limit is continuous and strictly increasing, so the argument of that article's example on quantiles (with $\hat H_n$ in place of $\hat F_n$) gives $\hat h_p \to \sigma z_p$ with probability 1, for each $0 < p < 1$; almost sure convergence implies convergence in probability.
Step 2: the basic interval. Its endpoints are $L = \bar X_n - \hat h_{1-\alpha/2}/\sqrt n$ and $U = \bar X_n - \hat h_{\alpha/2}/\sqrt n$. So $\mu < L$ exactly when $\sqrt n(\bar X_n - \mu) - \hat h_{1-\alpha/2} > 0$. By the central limit theorem and Slutsky's theorem (the CLT article), $\sqrt n(\bar X_n - \mu) - \hat h_{1-\alpha/2}$ converges in distribution to a normal with mean $-\sigma z_{1-\alpha/2}$ and standard deviation $\sigma$, whose probability of exceeding 0 is $1 - \Phi(z_{1-\alpha/2}) = \alpha/2$. The limit is continuous at 0, so $P(\mu < L) \to \alpha/2$ (Lemma 1 of that article handles the strict inequality). In the same way $\mu > U$ exactly when $\sqrt n(\bar X_n - \mu) - \hat h_{\alpha/2} < 0$, with limiting probability $\Phi(z_{\alpha/2}) = \alpha/2$.
Step 3: the percentile interval. Its endpoints are $\bar X_n + \hat h_{\alpha/2}/\sqrt n$ and $\bar X_n + \hat h_{1-\alpha/2}/\sqrt n$. Now $\mu < L$ exactly when $\sqrt n(\bar X_n - \mu) + \hat h_{\alpha/2} > 0$, and the limit is a normal with mean $\sigma z_{\alpha/2}$ and standard deviation $\sigma$, whose probability of exceeding 0 is $\Phi(z_{\alpha/2}) = \alpha/2$. The upper side is the same with $z_{1-\alpha/2}$. The symmetry of the normal limit is used here: it makes $z_{\alpha/2} = -z_{1-\alpha/2}$, so that reading the bootstrap quantiles directly or reflected gives the same limit.
Step 4: the studentized interval. By Theorem 6 of the plug-in article, the bootstrap distribution of $T^*$ converges uniformly to $\Phi$ with probability 1, so its quantiles $\hat t_p \to z_p$, as in Step 1. Since $\mu < L = \bar X_n - \hat t_{1-\alpha/2}\,s_n/\sqrt n$ exactly when $\sqrt n(\bar X_n - \mu)/s_n - \hat t_{1-\alpha/2} > 0$, and the studentized mean converges in distribution to the standard normal (Corollary 1 of the CLT article), the limiting probability is $1 - \Phi(z_{1-\alpha/2}) = \alpha/2$. The upper side is the same. $\square$
The theorem is a statement about limits. It does not distinguish the three intervals, and it says nothing about $n = 30$. The distinction comes from the rate at which the misses approach $\alpha/2$, which is the subject of the section on accuracy below.
Correcting the percentile interval: BC and BCa
Theorem 1 asked for a scale on which $\hat\phi$ is unbiased with a fixed spread. Efron's BCa interval asks for less: a scale on which $\hat\phi$ may be biased, by a fixed number of its standard deviations, and on which its standard deviation may change linearly with the parameter. Efron and Hastie (2016, eq. 11.38) write the model as
$$\hat\phi = \phi + \sigma_\phi\,(Z - z_0), \qquad \sigma_\phi = 1 + a\phi, \qquad Z \text{ standard normal},$$
for every value of $\phi = m(\theta)$, where $m$ is a continuous, strictly increasing transformation, $z_0$ and $a$ are constants, and $\sigma_\phi > 0$ at the true $\phi$. The bias correction $z_0$ measures median bias: $P(\hat\phi < \phi) = P(Z < z_0) = \Phi(z_0)$, so $z_0 = 0$ means the estimate falls below the truth half the time. The acceleration $a$ is the rate at which the standard deviation of $\hat\phi$ changes with $\phi$. The model includes Theorem 1's normal case ($z_0 = a = 0$). The BCa limit at level $p$ is
$$\hat\theta_{\mathrm{BCa}}[p] = \hat G^{-1}\Big(\Phi\Big(z_0 + \frac{z_0 + z_p}{1 - a(z_0 + z_p)}\Big)\Big),$$
Efron and Hastie's eq. 11.39, and the interval is $\big[\hat\theta_{\mathrm{BCa}}[\alpha/2],\ \hat\theta_{\mathrm{BCa}}[1 - \alpha/2]\big]$. It is a percentile interval read at adjusted levels. With $a = 0$ the adjusted level is $\Phi(2z_0 + z_p)$, the BC interval (their eq. 11.33); with $z_0 = a = 0$ it is the percentile interval. Efron and Hastie state that the BCa limits are exact under the model and refer to Section 3 of Efron (1987) for the algebra (Section 11.7, note 4). Here it is.
Theorem 3 (the BCa interval is exact under its model). Under the model above, suppose that the ideal bootstrap reproduces it at the estimate: given the data, $\hat\phi^* = \hat\phi + \sigma_{\hat\phi}(Z^* - z_0)$ with $Z^*$ standard normal, whenever $\sigma_{\hat\phi} = 1 + a\hat\phi > 0$. Fix $0 < p < 1$ and write $w = z_0 + z_p$, and suppose $1 - aw > 0$. On the outcomes with $\sigma_{\hat\phi} \le 0$, where no such bootstrap distribution exists, define the limit on the $\phi$ scale by the closed form $(\hat\phi + w)/(1 - aw)$ of Step 2 below. Then $P\big(\theta < \hat\theta_{\mathrm{BCa}}[p]\big) = p$.
The convention costs almost nothing. Under the model, $1 + a\hat\phi = 1 + a\phi + a\sigma_\phi(Z - z_0) = \sigma_\phi\big(1 + a(Z - z_0)\big)$, so for $a > 0$ the condition $\sigma_{\hat\phi} > 0$ fails only when $Z \le z_0 - 1/a$, an event of probability $\Phi(z_0 - 1/a)$, below $10^{-22}$ for $a = z_0 = 0.1$ (for $a < 0$ the event is $Z \ge z_0 - 1/a$). Steps 1 and 2 use $\sigma_{\hat\phi} > 0$; Step 3 uses only $\sigma_\phi > 0$.
Here is the plan. Compute the bootstrap distribution function on the $\phi$ scale in closed form, evaluate it at the exact confidence limit for $\phi$, and check that the level it returns is the BCa formula. Transformation invariance then carries the result to $\theta$.
Proof. Step 1: the bootstrap distribution. On the $\phi$ scale, $\hat K(t) = P_*(\hat\phi^* \le t) = P_*\big(Z^* \le z_0 + (t - \hat\phi)/\sigma_{\hat\phi}\big) = \Phi\big(z_0 + (t - \hat\phi)/\sigma_{\hat\phi}\big)$. Its quantile at level $\beta$ is $\hat K^{-1}(\beta) = \hat\phi + \sigma_{\hat\phi}\big(\Phi^{-1}(\beta) - z_0\big)$.
Step 2: the BCa limit on the $\phi$ scale. With $\beta = \Phi\big(z_0 + w/(1 - aw)\big)$, Step 1 gives
$$\hat\phi[p] = \hat\phi + (1 + a\hat\phi)\,\frac{w}{1 - aw} = \frac{\hat\phi(1 - aw) + w + a\hat\phi w}{1 - aw} = \frac{\hat\phi + w}{1 - aw}.$$
Step 3: its coverage. Since $1 - aw > 0$, $\phi < \hat\phi[p]$ exactly when $\phi(1 - aw) < \hat\phi + w$, that is, $\phi - \hat\phi < w(1 + a\phi) = w\,\sigma_\phi$. By the model, $\phi - \hat\phi = -\sigma_\phi(Z - z_0)$, and $\sigma_\phi > 0$, so the condition is $-(Z - z_0) < w$, that is, $Z > z_0 - w = -z_p$. This has probability $1 - \Phi(-z_p) = \Phi(z_p) = p$.
Step 4: back to $\theta$. The BCa limit is a bootstrap percentile at a level $\beta$ that depends only on $z_0$, $a$ and $p$. By Step 1 of Theorem 1, percentiles commute with the increasing map $m$, so $\hat\theta_{\mathrm{BCa}}[p] = m^{-1}(\hat\phi[p])$, and $\theta < \hat\theta_{\mathrm{BCa}}[p]$ exactly when $\phi < \hat\phi[p]$. $\square$
The script checked Theorem 3 by simulation for three choices of $(a, z_0, \phi)$, among them $a = 0.1$, $z_0 = 0.1$ and $\phi = 1$: over 400,000 draws from the model (seed 5), the limits at levels 0.025 and 0.975, computed from the exact bootstrap distribution, fell above $\phi$ in 2.46% and 97.54% of draws, the same in all three cases. The agreement to the last digit is Step 3 at work: whatever $a$, $z_0$ and $\phi$ are, the event is exactly $Z > -z_p$, so the three checks count the same draws of $Z$, and 0.0246 is the share of those 400,000 standard normal draws above 1.96. In the check, $1 + a\hat\phi$ was positive in every draw. The BCa interval, like the percentile interval, is transformation invariant, because it is a percentile interval at levels that do not depend on the scale (Efron and Hastie, 2016, Section 11.7, note 4).
Estimating $z_0$. Under the model, $\hat K(\hat\phi) = \Phi(z_0)$ by Step 1, so $z_0 = \Phi^{-1}\big(\hat G(\hat\theta)\big)$ on any increasing scale. The estimate is
$$\hat z_0 = \Phi^{-1}\big(\#\{b : \hat\theta^{*b} < \hat\theta\}/B\big),$$
the normal quantile of the share of replicates below the estimate (Efron and Hastie, 2016, eqs. 11.31 and 11.32); the script counts a replicate exactly equal to $\hat\theta$ as one half. For the running sample 108,227 of the 200,000 bootstrap means were below 5.678 and 15 were equal to it; counting each tie as one half, the share is 0.5412, so $\hat z_0 = \Phi^{-1}(0.5412) = 0.1034$.
Estimating $a$. The acceleration is not a property of the bootstrap distribution and must be computed separately. In one-sample nonparametric problems Efron and Hastie (2016, eq. 11.40) estimate it from the jackknife: with $\hat\theta_{(i)}$ the estimate computed with observation $i$ left out and $\hat\theta_{(\cdot)}$ the average of the $n$ of them,
$$\hat a = \frac{\sum_{i=1}^n \big(\hat\theta_{(\cdot)} - \hat\theta_{(i)}\big)^3}{6\Big[\sum_{i=1}^n \big(\hat\theta_{(\cdot)} - \hat\theta_{(i)}\big)^2\Big]^{3/2}}.$$
The Jackknife treats the leave-one-out estimates in general. For the mean the formula has a closed form.
Theorem 4 (the acceleration of a mean). For the mean, $\hat a = \hat\gamma/(6\sqrt n)$, where $\hat\gamma = \frac1n\sum_i (x_i - \bar x)^3/\hat\sigma_n^3$ is the plug-in skewness of the sample and $\hat\sigma_n^2 = \frac1n\sum_i (x_i - \bar x)^2$.
Proof. Leaving out $x_i$ gives $\hat\theta_{(i)} = (n\bar x - x_i)/(n - 1)$. Their average is $\hat\theta_{(\cdot)} = (n^2\bar x - n\bar x)/(n(n-1)) = \bar x$, so $\hat\theta_{(\cdot)} - \hat\theta_{(i)} = \bar x - (n\bar x - x_i)/(n-1) = (x_i - \bar x)/(n - 1)$. Substituting, the factors $(n-1)^{-3}$ in the numerator and denominator cancel:
$$\hat a = \frac{\sum_i (x_i - \bar x)^3}{6\big[\sum_i (x_i - \bar x)^2\big]^{3/2}} = \frac{n\hat\gamma\hat\sigma_n^3}{6\,(n\hat\sigma_n^2)^{3/2}} = \frac{\hat\gamma}{6\sqrt n}. \qquad \square$$
The skewness of a mean of $n$ observations is the population skewness divided by $\sqrt n$ (The Central Limit Theorem proves it), so $\hat a$ is one sixth of the estimated skewness of $\bar X$. Efron and Hastie (2016, Section 11.7, note 5) describe the same link in one-parameter exponential families, where $a$ is one sixth of the skewness of the score. Why should skewness show up as a changing standard deviation? For a right-skewed population, samples with large means are the ones that caught large values, and those samples also have large standard deviations. Confidence Intervals for a Mean measured the resulting correlation between sample mean and standard deviation: 0.88 in samples of 10 from a lognormal population. That dependence is what $a > 0$ encodes.
For the running sample, $\hat\gamma = 3.2226$, so $\hat a = 3.2226/(6\sqrt{30}) = 0.0981$. The adjusted levels are $\Phi(0.1034 + w/(1 - 0.0981w))$ with $w = 0.1034 \pm 1.96$, which are 0.0712 and 0.9964, so the BCa interval runs from the 7.12% point to the 99.64% point of the bootstrap means: 3.58 to 10.76. Compared with the percentile interval, 3.11 to 9.15, both ends move up, and the upper end moves much further. The BC interval, with $a = 0$, is 3.30 to 9.60.
Example (BCa at the edge of the bootstrap distribution). The upper BCa level 0.9964 asks for a point with 0.36% of the bootstrap distribution above it, about seven replicates out of 2,000. The endpoint is then estimated from very few replicates, and its Monte Carlo error is larger than that of a 97.5% point. As the skewness grows, the levels move further into the tail; Efron and Hastie (2016, Section 11.4) note that the BC and BCa intervals work most dependably when $\lvert z_0\rvert$ and $\lvert a\rvert$ are small, of the order of 0.2 or less. Here both are about 0.1.
How accurate: first and second order
Theorem 2 says the misses tend to $\alpha/2$; the question is how fast. Efron and Hastie (2016, eqs. 11.36 and 11.37) call an interval first-order accurate if the probability that $\theta$ lies below a nominal one-sided limit at level $p$ is $p + O(1/\sqrt n)$, and second-order accurate if it is $p + O(1/n)$. A first-order error shrinks by half only when $n$ is multiplied by 4; a second-order error halves when $n$ doubles. They state that the standard interval $\hat\theta \pm z\,\widehat{\operatorname{se}}$ is typically first-order accurate, that neither the percentile nor the BC interval is second-order accurate (though both tend to do better than the standard interval), and that the BCa interval is second-order accurate, its second-order accuracy having been established by Hall (1988). Hall's abstract reports a comparison of seven bootstrap methods in which the percentile-$t$ (the studentized interval here) and the accelerated bias correction emerge as the most promising, and it confirms Efron's conjecture that the BCa interval is second-order correct in a variety of multivariate settings. These results are cited, not proved, here: their proofs rest on Edgeworth expansions of the distributions involved, which this unit does not develop.
Where the extra accuracy comes from can be seen in the expansion that The Central Limit Theorem quotes for the standardized mean of a continuous population (CRP's recorded values make it a lattice distribution, so this is a guide here, not a theorem about the experiment): its distribution function is $\Phi(z) + \gamma(1 - z^2)\phi(z)/(6\sqrt n)$ plus terms of smaller order, where $\gamma$ is the population skewness. The normal approximation drops the $1/\sqrt n$ term entirely, so its error is of that order. A bootstrap distribution of a root with a fixed limit carries the same kind of term with $\hat\gamma$ in place of $\gamma$, and $\hat\gamma - \gamma$ is itself of order $1/\sqrt n$, so what is left over is of order $1/n$. That is the heuristic behind the studentized interval's advantage; it is not a proof, and this article cites no source that proves second-order accuracy for the studentized interval. It also explains why the bootstrap did not help the unstudentized root in the plug-in article's experiment: there the bootstrap distribution's scale is $\hat\sigma_n$, and the error $\hat\sigma_n - \sigma$, of order $1/\sqrt n$, enters at first order. The percentile and basic intervals use exactly that unstudentized distribution. The BCa interval recovers second-order accuracy by correcting the percentile levels for the bias and the skewness at first order, through $\hat z_0$ and $\hat a$.
"Second order" is an asymptotic statement. It does not promise that a BCa or studentized interval is accurate for thirty values of CRP, whose skewness of 7.9 makes the constants in front of $1/\sqrt n$ and $1/n$ large. The experiment measures what the orders mean at the sample sizes people use.
Coverage on C-reactive protein
The population makes the coverage of every interval computable. For each sample size from 10 to 400, the script ran 30 independent runs (NumPy seeds $10{,}000n$ to $10{,}000n + 29$). Each run drew 2,000 samples from the CRP population and, for each sample, computed the $t$ interval and the four bootstrap intervals from the same $B = 2{,}000$ resamples. Coverage is the share of the 2,000 samples whose interval contains $\mu = 4.17$; the tables give the median over the 30 runs. The interquartile ranges of the coverages over runs are at most 0.015 wide. The comparisons between methods are more precise than that, because all methods are computed on the same samples, so their differences are measured run by run: every ordering stated below holds in at least 24 of the 30 runs (for example, BCa above the $t$ interval at $n = 10$ in 25 runs, and in all 30 at every larger $n$).
| $n$ | $t$ | percentile | basic | studentized | BCa |
|---|---|---|---|---|---|
| 10 | 0.770 | 0.734 | 0.683 | 0.889 | 0.774 |
| 20 | 0.785 | 0.777 | 0.731 | 0.883 | 0.818 |
| 50 | 0.828 | 0.829 | 0.798 | 0.879 | 0.847 |
| 100 | 0.868 | 0.876 | 0.844 | 0.909 | 0.889 |
| 200 | 0.900 | 0.904 | 0.882 | 0.928 | 0.916 |
| 400 | 0.919 | 0.923 | 0.909 | 0.940 | 0.929 |
Almost every miss is on the same side. The next table gives the share of samples in which the whole interval lay below $\mu$, so that $\mu > U$; the nominal value is 0.025.
| $n$ | $t$ | percentile | basic | studentized | BCa |
|---|---|---|---|---|---|
| 10 | 0.230 | 0.262 | 0.316 | 0.109 | 0.209 |
| 20 | 0.215 | 0.219 | 0.269 | 0.111 | 0.164 |
| 50 | 0.172 | 0.167 | 0.202 | 0.107 | 0.131 |
| 100 | 0.130 | 0.117 | 0.154 | 0.077 | 0.085 |
| 200 | 0.097 | 0.087 | 0.116 | 0.051 | 0.058 |
| 400 | 0.073 | 0.064 | 0.087 | 0.037 | 0.043 |
The opposite misses, with the interval wholly above $\mu$, were rare for every method: at most 0.007 for the $t$ interval and at most 0.028 for any interval, at any $n$. Six observations from this experiment:
- No interval reaches 95% at any of these sample sizes. At $n = 400$ the best, the studentized interval, covers 94.0%; the $t$ interval covers 91.9%. CRP's skewness of 7.9 is extreme, and the coverage reflects it.
- The studentized interval is best at every $n$, and by a wide margin at small $n$: 88.9% at $n = 10$ against 77.0% for the $t$ interval. It pays in length. Its median length at $n = 10$ is 7.44 mg/L against 5.05 for the $t$ interval, and the extra length is placed above $\bar x$, where the misses are.
- BCa is second at every $n$, and improves on the percentile interval at every $n$, by 4.1 points at $n = 20$ and 0.6 points at $n = 400$. At $n = 10$ it is barely better than the $t$ interval, 77.4% against 77.0%. Its lower misses, 0.017 to 0.028, are the closest of any interval to the nominal 0.025 up to $n = 200$; at $n = 400$ the studentized interval's 0.024 is closer.
- The percentile interval is close to the $t$ interval, a little worse at $n \le 20$ (in all 30 runs) and a little better from $n = 50$ (in 24 to 30 of the runs).
- The basic interval is worst at every $n$, as the reflection argument predicted: it extends the interval downward when the bootstrap distribution says the error runs upward.
- From $n = 100$ to 400 the rates differ. Subtract the nominal 0.025 from the upper misses at $n = 100$, 200 and 400. For the studentized interval the excess is 0.052, 0.026 and 0.012; for BCa, 0.060, 0.033 and 0.018. In this window each roughly halved with each doubling of $n$, as an error of order $1/n$ would. For the $t$ interval the excess is 0.105, 0.072 and 0.048, for the percentile interval 0.092, 0.062 and 0.039, and for the basic interval 0.129, 0.091 and 0.062; each shrank by a factor of about 0.7, close to $1/\sqrt 2 = 0.71$, as an error of order $1/\sqrt n$ would. That is consistent with the first- and second-order rates of the previous section, but it is three sample sizes over a factor of 4, and the orders are statements about limits. Below $n = 100$ the pattern does not hold: the studentized interval's excess was 0.084, 0.086 and 0.082 at $n = 10$, 20 and 50, flat while $n$ grew fivefold. The figure shows both regimes.
The intervals in these tables are those of Confidence Intervals for a Mean's warning, made concrete: for a strongly skewed population the $t$ interval's misses pile up on one side, and resampling cannot create the long tail that a small sample lacks. What the studentized and BCa intervals do is use what the sample does show, the correlation between a sample's mean and its spread, to move the interval's upper end.
A second example: the area under an ROC curve
The area under the ROC curve of HbA1c as a screen for a diabetic fasting glucose is 0.8894, from 115 cases and 1,860 non-cases (ROC Curves for Diagnostic Tests). It is a plug-in estimate of $P(X > Y) + \frac12 P(X = Y)$ (The Plug-In Principle, Problem 3), and it is bounded by 1, so its sampling distribution can be expected to lean away from the bound. The bootstrap here resamples the cases and the non-cases separately, each with replacement, keeping both group sizes fixed as the study design did. The studentized interval uses DeLong's standard error (DeLong, DeLong and Clarke-Pearson, 1988, as computed in the ROC article), recomputed in every resample. For each of 30 seeds (0 to 29) the script drew 2,000 resamples; the medians of the endpoints over the 30 seeds are:
| Interval | Lower | Upper |
|---|---|---|
| DeLong, $\hat\theta \pm 1.96\,\widehat{\operatorname{se}}$ | 0.855 | 0.924 |
| percentile | 0.853 | 0.922 |
| basic | 0.857 | 0.926 |
| BC | 0.852 | 0.921 |
| studentized | 0.847 | 0.920 |
The interquartile ranges of the endpoints over seeds are at most 0.003 wide. The median bootstrap standard error is 0.0176, against DeLong's 0.0177. The bias correction is small and negative, with median $\hat z_0 = -0.026$ (interquartile range $-0.040$ to $-0.007$): slightly more than half of the bootstrap AUCs lie above 0.8894.
All five intervals agree to within about 0.01 at each end, as Theorem 2's large-sample reasoning suggests for samples of this size. The differences that remain have the expected direction. The bootstrap distribution of the AUC is skewed to the left, toward lower values, away from the bound at 1. The percentile, BC and studentized intervals therefore sit slightly lower than DeLong's symmetric interval, and the basic interval, reflecting the skew, sits slightly higher; its upper end, 0.926, is the highest of the five. The BCa interval is not computed here: Efron and Hastie's jackknife estimate of $a$ is given for one-sample problems, and the AUC is a two-sample statistic. With 115 cases, the case sample limits the precision, as the ROC article showed: almost all of the AUC's variance comes from the cases.
Which interval to use
No interval is best for every statistic, and the theory and the experiment together support a short list rather than a rule.
- When a standard error formula exists, the studentized interval is the one that resamples a near-pivot, and on CRP it was the most accurate at every sample size. It is not transformation invariant, so it should be computed on a scale where the statistic is roughly symmetric and its standard error stable, when such a scale is known; it can be very long in small samples.
- When no formula exists, the BCa interval corrects the percentile interval for bias and for a changing standard error at no extra cost in resamples, beyond the $n$ recomputations of the jackknife. It is transformation invariant, and Hall (1988) found it, with the studentized interval, the most promising of the methods he compared. Efron and Hastie (2016) advise care when $\lvert\hat z_0\rvert$ or $\lvert\hat a\rvert$ is large, say above 0.2.
- The percentile interval is the simplest. It is invariant but not second-order accurate, and it is what The Bootstrap Method and Significance Testing use. For roughly symmetric statistics it agrees with the others.
- The basic interval has the weakest case on skewed data; its virtue is that its logic, the bootstrap distribution of the error, is the easiest to state.
Two things hold for all of them. $B$ should be in the thousands for intervals (Problem 3 measures why). And no interval repairs a sample that does not represent the population: with $n = 10$ from CRP, the best of them covered 88.9%.
Example (a common misreading). "The bootstrap is distribution-free, so a bootstrap interval is exact for any population." Theorem 2 is a large-sample result, and the table shows how far from exact the intervals are at practical sizes for a skewed population: at $n = 50$ the $t$ interval covered 82.8% and the percentile interval 82.9%, nearly the same. The bootstrap removes the need for a formula. It does not remove the need for a sample large enough to show the population's shape.
What is proved here, what is cited, what the experiment shows
Proved here: the basic, percentile and studentized intervals as solutions of the exact-pivot recipe; the exactness of the percentile interval under a symmetric transformation model, and its transformation invariance; that the basic, percentile and studentized intervals for a mean have one-sided misses tending to $\alpha/2$ under a finite variance; the exactness of the BCa limits under Efron's model, with the BC and percentile limits as special cases; the formula $\hat z_0 = \Phi^{-1}(\hat G(\hat\theta))$ from the model; and that the jackknife acceleration of a mean is the plug-in skewness over $6\sqrt n$.
Taken from other pages of the site: the bootstrap consistency theorems for the mean and the studentized mean, and the argument for quantiles, from The Plug-In Principle and Why the Bootstrap Works; Slutsky's theorem, the central limit theorem and the studentized mean's limit from the CLT article; the delta method; the skewness of a mean and the one-term Edgeworth expansion quoted in The Central Limit Theorem.
Cited, not proved: the BCa model, its endpoint formula and the jackknife estimate of $a$ (Efron and Hastie, 2016, eqs. 11.38 to 11.40), whose properties under the model are then proved here; the first- and second-order accuracy of the intervals (Efron and Hastie, 2016, Section 11.4; Hall, 1988, abstract); the advice on $B$ and on small $\lvert z_0\rvert$ and $\lvert a\rvert$.
What the experiment shows, for this population only: that no interval reached 95% coverage for the CRP mean at any $n$ from 10 to 400; that the studentized interval came closest at every $n$, BCa second at every $n$ (by a small margin over the $t$ interval at $n = 10$), the percentile interval close to the $t$ interval, and the basic interval last; that nearly all misses fell on one side; and that from $n = 100$ to 400 the excess of the studentized and BCa intervals' upper misses roughly halved with each doubling of $n$, while that of the $t$, percentile and basic intervals shrank by about $1/\sqrt 2$, whereas below $n = 100$ the studentized interval's excess did not shrink at all. For the HbA1c AUC, with 115 cases, all the intervals agreed to within about 0.01.
Worked problems
The problems are posed on the unit's running sample and on a small exact case, one for each construction that has a formula to apply by hand. Every number is recomputed by the article's script, tools/experiments/statistics/bootstrap-confidence-intervals.py.
Problem 1 (the BCa limits by hand). For the running sample of thirty CRP values, a share of 0.5412 of 200,000 bootstrap means fell below $\bar x = 5.678$ (ties counted as one half), and the plug-in skewness is 3.2226. Compute $\hat z_0$, $\hat a$ and the two adjusted levels that give the 95% BCa interval, and say which order statistics they would pick out of $B = 2{,}000$ replicates.
Solution. The levels are 0.0712 and 0.9964, so the BCa interval runs from the 7.12% point to the 99.64% point of the bootstrap distribution; with $B = 2{,}000$ that is the 143rd and the 1,993rd smallest replicate. First, $\hat z_0 = \Phi^{-1}(0.5412) = 0.1034$. By Theorem 4, $\hat a = 3.2226/(6\sqrt{30}) = 0.0981$. For the lower limit, $w = \hat z_0 + z_{0.025} = 0.10339 - 1.95996 = -1.85658$ (five places are kept here and below, so that the rounding does not show), so
$$\hat z_0 + \frac{w}{1 - \hat a w} = 0.10339 + \frac{-1.85658}{1 + 0.09806 \times 1.85658} = 0.10339 - \frac{1.85658}{1.18206} = -1.46724,$$
and the level is $\Phi(-1.46724) = 0.0712$. For the upper limit, $w = 0.10339 + 1.95996 = 2.06335$, so $0.10339 + 2.06335/(1 - 0.09806 \times 2.06335) = 0.10339 + 2.06335/0.79766 = 2.69013$, and the level is $\Phi(2.69013) = 0.9964$. With $B = 2{,}000$ the order statistics are $\lceil 0.0712 \times 2{,}000\rceil = 143$ and $\lceil 0.9964 \times 2{,}000\rceil = 1{,}993$. Read from the 200,000 replicates, the limits are 3.58 and 10.76. The positive $\hat z_0$ moves both levels up a little (the BC interval, with $\hat a = 0$, would use $\Phi(2\hat z_0 \pm 1.96)$, that is 0.0398 and 0.9849); the positive $\hat a$ stretches the upper level much further into the tail than the lower.
Problem 2 (the jackknife acceleration of a mean, exactly). For the three values 4.48, 0.87 and 13.28, compute $\hat a$ from the jackknife formula and check Theorem 4.
Solution. Both routes give $\hat a = 0.0444$. Leaving out each value in turn gives the means $(0.87 + 13.28)/2 = 7.075$, $(4.48 + 13.28)/2 = 8.88$ and $(4.48 + 0.87)/2 = 2.675$, whose average is $\hat\theta_{(\cdot)} = 6.21 = \bar x$. The differences $\hat\theta_{(\cdot)} - \hat\theta_{(i)}$ are $-0.865$, $-2.67$ and $3.535$. Their cubes sum to 24.4928 and their squares to 20.3734, so
$$\hat a = \frac{24.4928}{6 \times 20.3734^{3/2}} = 0.0444.$$
For Theorem 4, the plug-in skewness of the three values is 0.4613, and $0.4613/(6\sqrt 3) = 0.0444$. The differences are the deviations $x_i - \bar x = -1.73, -5.34, 7.07$ divided by $n - 1 = 2$, as the proof of Theorem 4 says.
Problem 3 (the Monte Carlo error of the bias correction). With $B = 2{,}000$, how precisely is $\hat z_0$ known, and how much does that uncertainty move the BCa interval of Problem 1?
Solution. The standard error of $\hat z_0$ is about 0.028, and moving $\hat z_0$ by that much moves the interval's ends by about 0.05 and 0.17 mg/L. The share $\hat p_0$ of replicates below $\hat\theta$ is a binomial proportion out of $B$, with standard error $\sqrt{p_0(1 - p_0)/B}$. The function $\Phi^{-1}$ has derivative $1/\phi(\Phi^{-1}(p))$, so by the delta method (The Delta Method, Theorem 1) the standard error of $\hat z_0$ is about $\sqrt{p_0(1 - p_0)/B}/\phi(z_0)$. Near $p_0 = \frac12$ this is $\sqrt{1/(4 \times 2{,}000)}/\phi(0) = 0.0280$, that is, $\sqrt{\pi/2}/\sqrt B = 1.2533/\sqrt B$, the value Efron and Hastie (2016, Section 11.7, note 3) give. Recomputing Problem 1 with $\hat z_0 = 0.1034 \mp 0.0280$ gives levels 0.0649 and 0.9956 (interval 3.53 to 10.59, read from the 200,000 replicates) and 0.0779 and 0.9971 (interval 3.63 to 10.94). The upper end moves more, because the bootstrap means thin out in the right tail. This is only the part of the Monte Carlo error that comes through $\hat z_0$. The endpoint itself is also estimated from few replicates in the tail, and that part is larger: over 200 sets of 2,000 resamples (seeds 700 to 899), the BCa upper limit varied with standard deviation 0.27 even with $z_0$ held at its near-ideal value, against 0.20 for the part due to $\hat z_0$ alone (five sets gave upper limits from 10.16 to 11.21, which combines both parts). That is the practical reason for $B$ in the thousands, and for more when a limit falls far in the tail.
Problem 4 (transformation invariance on the log scale). Compute the percentile and studentized intervals for $\ln\mu$ from the running sample's 200,000 resamples, transform them back, and compare them with the intervals for $\mu$.
Solution. The percentile interval for $\ln\mu$ is 1.1336 to 2.2135, whose exponentials are exactly the percentile limits for $\mu$, 3.107 and 9.148: taking logarithms does not change which replicate is the 5,000th or the 195,000th smallest, so the 2.5% point of $\ln\bar X^*$ is the logarithm of the 2.5% point of $\bar X^*$. For the studentized interval the standard error of $\ln\bar X$ is, by the delta method, $s/(\sqrt n\,\bar x)$, and in each resample $s^*/(\sqrt n\,\bar X^*)$. The studentized interval on the log scale is 1.2148 to 2.5749, which transforms back to 3.37 to 13.13, against 3.31 to 12.17 computed on the original scale. The difference is larger than the Monte Carlo error at this number of resamples: a second set of 200,000 resamples (seed 2027) gave upper limits of 12.10 on the original scale and 13.03 from the log scale. The same holds with 2,000 resamples, because both limits come from the same resamples and their errors move together: over 100 sets of 2,000 resamples (seeds 500 to 599), the log-scale upper limit exceeded the original-scale one by 0.94 on average, with standard deviation 0.12 and never by less than 0.70, although the original-scale limit alone varied with standard deviation 0.26. The two answers differ because the studentized interval's endpoints are $\hat\theta$ minus quantiles times a standard error, and a nonlinear transformation changes both the estimate's scale and the standard error's. Which scale is better for the studentized interval is not settled by one sample; for the percentile and BCa intervals the question does not arise.
Summary
- An interval is built by solving $k_{\alpha/2} \le \text{root} \le k_{1-\alpha/2}$ for $\theta$; the bootstrap supplies the quantiles $k_p$. Coverage should be judged on each side separately.
- Basic: $[2\hat\theta - \hat\theta^*_{(1-\alpha/2)},\ 2\hat\theta - \hat\theta^*_{(\alpha/2)}]$, from the bootstrap distribution of $\hat\theta^* - \hat\theta$. Not invariant; on skewed data it leans the wrong way.
- Percentile: $[\hat\theta^*_{(\alpha/2)},\ \hat\theta^*_{(1-\alpha/2)}]$. Exact when some increasing transformation makes the error symmetric with a fixed distribution; transformation invariant.
- Studentized: $[\hat\theta - t^*_{(1-\alpha/2)}\widehat{\operatorname{se}},\ \hat\theta - t^*_{(\alpha/2)}\widehat{\operatorname{se}}]$, from the bootstrap distribution of $(\hat\theta^* - \hat\theta)/\widehat{\operatorname{se}}^*$. Needs a standard error formula; not invariant.
- BCa: the percentile interval read at levels $\Phi\big(\hat z_0 + (\hat z_0 + z_p)/(1 - \hat a(\hat z_0 + z_p))\big)$, exact under Efron's model $\hat\phi = \phi + (1 + a\phi)(Z - z_0)$, with $\hat z_0$ from the share of replicates below $\hat\theta$ and $\hat a$ from the jackknife; for a mean $\hat a = \hat\gamma/(6\sqrt n)$. Invariant.
- For a mean, the basic, percentile and studentized intervals have one-sided misses tending to $\alpha/2$ under a finite variance. The BCa interval is second-order accurate and the percentile and BC intervals are not (cited). On CRP, from $n = 100$ to 400, the studentized and BCa intervals' excess misses halved with each doubling and the others' shrank by about 0.7; below 100 the studentized interval's did not shrink.
- On CRP, the studentized interval covered 88.9% at $n = 10$ and 94.0% at $n = 400$, the BCa 77.4% and 92.9%, the $t$ interval 77.0% and 91.9%; almost all misses fell above.
What we have learned
A bootstrap interval is only as good as the root it resamples. The basic and percentile intervals resample the estimate itself, whose bootstrap distribution gets the spread wrong by as much as the sample standard deviation is wrong, and on CRP, from $n = 100$ to 400, their errors shrank only at the first-order rate. The studentized interval resamples a near-pivot, and the BCa interval corrects the percentile interval for bias and for a standard error that changes with the parameter. The BCa interval is second-order accurate (cited), and the Edgeworth heuristic above suggests the same gain for the studentized interval; on C-reactive protein both showed an error that halved with each doubling of the sample from 100 to 400, though not below 100. The theory explains the ranking the experiment found, and the experiment shows what the theory's orders of magnitude mean at sample sizes people use: on a population as skewed as CRP, even the best interval needs hundreds of observations to come near its nominal level.
The bootstrap estimates a sampling distribution by resampling, under the assumption that the observations are exchangeable draws from one population. The same assumption, used as a null hypothesis instead of an approximation, gives tests that are exact at every sample size: rearranging the data, instead of resampling it, is the subject of Permutation Tests.
References
- DeLong, E. R., DeLong, D. M., and Clarke-Pearson, D. L. (1988). Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach. Biometrics, 44(3), 837–845. Cited through ROC Curves for Diagnostic Tests. record · 2 pages
- Efron, B. (1987). Better bootstrap confidence intervals. Journal of the American Statistical Association, 82(397), 171–185. https://doi.org/10.1080/01621459.1987.10478410 (abstract read). record
- Efron, B., and Hastie, T. (2016). Computer Age Statistical Inference: Algorithms, Evidence, and Data Science. Cambridge University Press. Chapter 11, "Bootstrap Confidence Intervals": Sections 11.2 to 11.5 (eqs. 11.16 to 11.18, 11.31 to 11.33, 11.36 to 11.40, 11.53) and Section 11.7, notes 3 to 5. record · 3 pages
- Hall, P. (1988). Theoretical comparison of bootstrap confidence intervals. The Annals of Statistics, 16(3), 927–953. https://doi.org/10.1214/aos/1176350933 (abstract read). record
- National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018 Data Documentation: High-Sensitivity C-Reactive Protein (HSCRP_J). Centers for Disease Control and Prevention. https://wwwn.cdc.gov/Nchs/Data/Nhanes/Public/2017/DataFiles/HSCRP_J.htm record · 5 pages
- SciPy developers.
scipy.stats.bootstrap, SciPy 1.15.2 documentation (the docstring of the installed version was read): themethodargument, "percentile", "basic" (also called "reverse") or "BCa". https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html record