Among the twelve NHANES adults of the Linear Models unit, the correlation between age and systolic blood pressure is $r = 0.584$. Twelve people are few, and two questions follow at once. Is $r$ biased, so that correlations computed this way from twelve people tend to come out too high or too low? And how far from the population's correlation is it likely to be? Once the normal model is dropped, the correlation has no simple exact standard error and its bias no simple formula. Can the twelve observations answer both questions themselves, without a model and without random resampling?
The jackknife answers them by recomputing the statistic twelve times, each time with one observation left out. The spread of the twelve recomputed values estimates the statistic's variance, and the gap between their average and the original value estimates its bias. The method is older than the bootstrap: Quenouille and Tukey introduced it (Miller, 1974, reviews the work since "its introduction by Quenouille and Tukey"). Quenouille's (1956) "Notes on bias in estimation" treats the bias side, and Efron and Stein (1981) call the variance estimate "Tukey's". Efron (1979) described the jackknife as a linear approximation to the bootstrap, and that relation runs through this article.
Here is the route. First we define the leave-one-out values, the bias estimate, the variance estimate and the pseudo-values that connect them. Then we prove that the bias correction removes the $1/n$ term of a bias, exactly so for the plug-in variance. Next we show what the variance estimate measures by reading "leave one out" as a derivative: the jackknife is a finite-difference version of the influence function, and its infinitesimal limit is the nonparametric delta method. For least-squares coefficients the jackknife has a closed form, MacKinnon and White's HC3, which we derive from the deletion identity of Leverage, Influence and Residual Diagnostics. Then we prove the Efron–Stein identity, which says the jackknife sum of squares overestimates, on average, the variance of the statistic computed from $n - 1$ observations, and finally we prove that it fails for the median. The article assumes The Plug-In Principle and Why the Bootstrap Works for the empirical distribution and the idea of a statistic as a function of it, and The Bootstrap Method for the bootstrap standard error it is compared with.
The data and the notation
The twelve adults are drawn at random, with a fixed seed, from the 5,176 adults examined in NHANES 2017–2018 who have a systolic reading and a body mass index (BMI). Leverage, Influence and Residual Diagnostics introduces them. Their systolic pressures (the mean of up to three readings, mm Hg) have mean 121.056. As in the rest of the unit, the 5,176 adults serve as a finite population to sample from, and the survey's weights are ignored. Every number below comes from tools/experiments/statistics/jackknife.py.
The notation is that of The Bootstrap Method. The data are $x_1, \dots, x_n$, independent draws from a distribution $F$; each $x_i$ may be a pair, as (age, pressure) is here. The statistic is $\hat\theta = T(x_1, \dots, x_n)$, a symmetric function: it does not depend on the order of its arguments. Then:
- $\hat\theta_{(i)}$ is the statistic computed from the $n - 1$ observations other than $x_i$;
- $\hat\theta_{(\cdot)} = \frac1n\sum_{i=1}^n \hat\theta_{(i)}$ is the average of the $n$ leave-one-out values;
- the jackknife estimate of bias is $b_J = (n - 1)\big(\hat\theta_{(\cdot)} - \hat\theta\big)$, and the bias-corrected estimate is $\hat\theta_J = \hat\theta - b_J$;
- the jackknife estimate of variance is $$v_J = \frac{n - 1}{n}\sum_{i=1}^n \big(\hat\theta_{(i)} - \hat\theta_{(\cdot)}\big)^2;$$
- the pseudo-values are $\tilde\theta_i = n\hat\theta - (n - 1)\hat\theta_{(i)}$.
The two multipliers $n - 1$ look arbitrary at first. They are chosen so that both estimates are exactly right for the sample mean, as the next section and the section on variance prove. Efron and Stein (1981) call $v_J$ "Tukey's jackknife estimate of variance".
Proposition 1 (pseudo-values). The bias-corrected estimate is the mean of the pseudo-values, and the jackknife variance is their sample variance divided by $n$: $$\hat\theta_J = \frac1n\sum_{i=1}^n\tilde\theta_i, \qquad v_J = \frac{1}{n}\cdot\frac{1}{n - 1}\sum_{i=1}^n\big(\tilde\theta_i - \hat\theta_J\big)^2.$$
Proof. Averaging, $\frac1n\sum_i\tilde\theta_i = n\hat\theta - (n - 1)\hat\theta_{(\cdot)} = \hat\theta - (n - 1)(\hat\theta_{(\cdot)} - \hat\theta) = \hat\theta - b_J$. Subtracting the average from each pseudo-value, $\tilde\theta_i - \hat\theta_J = -(n - 1)\big(\hat\theta_{(i)} - \hat\theta_{(\cdot)}\big)$, so $\sum_i(\tilde\theta_i - \hat\theta_J)^2 = (n - 1)^2\sum_i(\hat\theta_{(i)} - \hat\theta_{(\cdot)})^2$. Dividing by $n(n - 1)$ gives $\frac{n - 1}{n}\sum_i(\hat\theta_{(i)} - \hat\theta_{(\cdot)})^2 = v_J$. $\square$
Tukey's reading of the proposition is that the pseudo-values behave like $n$ independent observations whose mean is $\hat\theta_J$, so that the usual standard error of a mean applies to them. The reading is exact for the sample mean, whose pseudo-values are the observations themselves (Worked problem 4), and approximate otherwise; the section on influence says how good the approximation is.
Why the bias estimate works
Here is the idea in one sentence. If the bias of a statistic computed from $n$ observations is $a/n$, the bias from $n - 1$ observations is $a/(n - 1)$, and a combination of the two with weights $n$ and $-(n - 1)$ cancels the $a$.
Theorem 1 (bias reduction). Let $\hat\theta_n$ denote the symmetric statistic computed from $n$ independent observations from $F$, and suppose that $$E(\hat\theta_n) = \theta + \frac{a_1}{n} + \frac{a_2}{n^2} + r_n, \qquad \lvert r_n\rvert \le \frac{C}{n^3},$$ for every $n \ge 2$, with constants $a_1$, $a_2$, $C$ that do not depend on $n$. Then, for $n \ge 3$ (so that the expansion also holds at $n - 1$), $$E(\hat\theta_J) = \theta - \frac{a_2}{n(n - 1)} + n\,r_n - (n - 1)\,r_{n-1},$$ so the bias of $\hat\theta_J$ is of order $1/n^2$, while that of $\hat\theta_n$ is of order $1/n$ when $a_1 \ne 0$. If $a_2 = 0$ and $r_n = 0$ for all $n$, $\hat\theta_J$ is exactly unbiased.
Proof. Each $\hat\theta_{(i)}$ is the statistic computed from $n - 1$ independent observations from $F$, so $E(\hat\theta_{(i)}) = E(\hat\theta_{n-1})$ and therefore $E(\hat\theta_{(\cdot)}) = E(\hat\theta_{n-1})$. Since $\hat\theta_J = \hat\theta - (n - 1)(\hat\theta_{(\cdot)} - \hat\theta) = n\hat\theta - (n - 1)\hat\theta_{(\cdot)}$, $$E(\hat\theta_J) = n\Big(\theta + \frac{a_1}{n} + \frac{a_2}{n^2} + r_n\Big) - (n - 1)\Big(\theta + \frac{a_1}{n - 1} + \frac{a_2}{(n - 1)^2} + r_{n-1}\Big).$$ The $\theta$ terms give $n\theta - (n - 1)\theta = \theta$; the $a_1$ terms give $a_1 - a_1 = 0$; the $a_2$ terms give $a_2\big(\frac1n - \frac1{n - 1}\big) = -\frac{a_2}{n(n - 1)}$. The remainders are at most $C/n^2 + C/(n - 1)^2$ in absolute value. $\square$
Two statistics have biases with exactly this form and no remainder, and for them the correction is exact.
Corollary 1 (the plug-in variance). Let $\hat\theta = \hat\sigma^2 = \frac1n\sum_i(x_i - \bar x)^2$, the variance of the empirical distribution. Then $\hat\theta_J = s^2 = \frac{1}{n - 1}\sum_i(x_i - \bar x)^2$, for every data set.
Proof. The plug-in variance has $E(\hat\sigma^2) = \frac{n - 1}{n}\sigma^2 = \sigma^2 - \frac{\sigma^2}{n}$, so $a_1 = -\sigma^2$, $a_2 = 0$ and no remainder, and Theorem 1 makes $\hat\theta_J$ unbiased. That is a statement about expectations; the corollary claims an identity for every data set, which needs its own proof. Write $Q = \sum_j(x_j - \bar x)^2$. Removing $x_i$ moves the mean to $\bar x_{(i)} = \bar x - (x_i - \bar x)/(n - 1)$, and the sum of squares about the new mean is $Q_{(i)} = Q - \frac{n}{n - 1}(x_i - \bar x)^2$. (Expand $\sum_{j \ne i}(x_j - \bar x_{(i)})^2$ with $x_j - \bar x_{(i)} = (x_j - \bar x) + (x_i - \bar x)/(n - 1)$: the cross term is $\frac{2(x_i - \bar x)}{n - 1}\sum_{j \ne i}(x_j - \bar x) = -\frac{2(x_i - \bar x)^2}{n - 1}$, and the square term adds $(n - 1)\frac{(x_i - \bar x)^2}{(n - 1)^2}$; with the removed $(x_i - \bar x)^2$ the total change is $-(x_i - \bar x)^2\big(1 + \frac{2}{n - 1} - \frac{1}{n - 1}\big) = -\frac{n}{n - 1}(x_i - \bar x)^2$.) So $\hat\theta_{(i)} = Q_{(i)}/(n - 1)$ and $$\hat\theta_{(\cdot)} = \frac{1}{n(n - 1)}\sum_i\Big(Q - \frac{n}{n - 1}(x_i - \bar x)^2\Big) = \frac{Q}{n - 1} - \frac{Q}{(n - 1)^2} = \frac{(n - 2)\,Q}{(n - 1)^2}.$$ Then $\hat\theta_J = n\frac{Q}{n} - (n - 1)\frac{(n - 2)Q}{(n - 1)^2} = Q\Big(1 - \frac{n - 2}{n - 1}\Big) = \frac{Q}{n - 1} = s^2$. $\square$
For the twelve pressures the plug-in variance is 192.55 and $s^2 = 210.06$; the jackknife estimates the bias of the plug-in value as $-17.50$, and subtracting it gives 210.06 to every digit the script carries.
Corollary 2 (the square of a mean). Let $\hat\theta = \bar x^2$, an estimate of $\mu^2$. Then $\hat\theta_J = \bar x^2 - s^2/n$, which is unbiased for $\mu^2$.
Proof. $E(\bar x^2) = \operatorname{Var}(\bar x) + \mu^2 = \mu^2 + \sigma^2/n$, so $a_1 = \sigma^2$, $a_2 = 0$, no remainder, and Theorem 1 gives unbiasedness. For the identity on any data set, write $d_i = x_i - \bar x$, so $\bar x_{(i)} = \bar x - d_i/(n - 1)$ and $\bar x_{(i)}^2 = \bar x^2 - \frac{2\bar x d_i}{n - 1} + \frac{d_i^2}{(n - 1)^2}$. Averaging over $i$ (the $d_i$ sum to zero), $\hat\theta_{(\cdot)} = \bar x^2 + \frac{Q}{n(n - 1)^2}$, so $b_J = (n - 1)\frac{Q}{n(n - 1)^2} = \frac{Q}{n(n - 1)} = \frac{s^2}{n}$. $\square$
For the twelve adults, $\bar x^2 = 14{,}654.45$ and the jackknife bias is $s^2/n = 17.50$, the same number as in Corollary 1 with the sign reversed. That is no coincidence: $\bar x^2 = \overline{x^2} - \hat\sigma^2$, and the mean of the squares $\overline{x^2}$ is a mean, which the jackknife leaves alone.
Example (a correction that is mostly noise). For the correlation of age and pressure, the twelve leave-one-out correlations average 0.5851, slightly above $r = 0.5839$. The jackknife bias is $b_J = 11 \times 0.0012 = 0.014$ (0.0137 before rounding), and the corrected estimate is $\hat\theta_J = 0.570$. It is tempting to read the corrected value as better. It is not, on this evidence: the jackknife standard error of $r$ is 0.197, found below, so the correction moves the estimate by about a fourteenth of its own uncertainty. A bias estimate is itself a statistic, and Theorem 1 guarantees only that it removes the $1/n$ term on average. The same advice appears in The Bootstrap Method for a bootstrap bias estimate that is small beside the standard error. (The correlation among all 5,176 adults is 0.486; one sample of twelve cannot say whether the correction moved toward it.)
The variance estimate
The variance estimate rests on a similar calibration. For the sample mean $\bar x_{(i)} - \bar x_{(\cdot)} = -(x_i - \bar x)/(n - 1)$, because $\bar x_{(\cdot)} = \bar x$. Hence $$v_J = \frac{n - 1}{n}\sum_i\frac{(x_i - \bar x)^2}{(n - 1)^2} = \frac{1}{n}\cdot\frac{1}{n - 1}\sum_i(x_i - \bar x)^2 = \frac{s^2}{n},$$ the usual unbiased estimate of $\operatorname{Var}(\bar x)$. The factor $(n - 1)/n$ in $v_J$ is what makes this come out exactly. The same algebra applies to any statistic of the form $\hat\theta = \frac1n\sum_i g(x_i)$, with $g(x_i)$ in place of $x_i$: for such linear statistics $v_J = s_g^2/n$, where $s_g^2$ is the sample variance of the $g(x_i)$, and the jackknife is exact in the sense that $E(v_J) = \operatorname{Var}(\hat\theta)$.
For a statistic that is not linear, the leave-one-out values carry information about how sensitive $\hat\theta$ is to each observation. The table gives them for the correlation of age and pressure, with the pseudo-values and the influence values $\mathrm{IF}_i$ defined in the next section.
| Case | Age | Systolic | $r_{(i)}$ | Pseudo-value $\tilde\theta_i$ | $\mathrm{IF}_i$ |
|---|---|---|---|---|---|
| 1 | 54 | 129.3 | 0.5778 | 0.651 | 0.066 |
| 2 | 39 | 116.0 | 0.5780 | 0.648 | 0.064 |
| 3 | 43 | 119.3 | 0.5831 | 0.593 | 0.009 |
| 4 | 61 | 148.0 | 0.5811 | 0.615 | 0.114 |
| 5 | 63 | 144.0 | 0.5509 | 0.947 | 0.340 |
| 6 | 69 | 125.3 | 0.5891 | 0.527 | −0.045 |
| 7 | 19 | 106.7 | 0.5235 | 1.248 | 0.576 |
| 8 | 21 | 111.3 | 0.5577 | 0.872 | 0.271 |
| 9 | 28 | 119.3 | 0.6005 | 0.402 | −0.167 |
| 10 | 27 | 97.3 | 0.5189 | 1.299 | 0.609 |
| 11 | 77 | 112.7 | 0.7633 | −1.389 | −1.697 |
| 12 | 67 | 123.3 | 0.5978 | 0.432 | −0.138 |
The pseudo-values average $\hat\theta_J = 0.570$, and their spread gives the jackknife standard error $\sqrt{v_J} = 0.197$. One case dominates. Case 11, a 77-year-old with a pressure of 112.7 mm Hg, is old and has a low pressure, against the trend of the other eleven; without that case the correlation rises to 0.763. Its pseudo-value is $-1.39$, a value no correlation can take, which shows that pseudo-values are not estimates of $\theta$ in their own right, only devices for the two summaries of Proposition 1.
How does 0.197 compare with other estimates of the same standard error? The normal-theory value $(1 - r^2)/\sqrt{n - 1}$ is 0.199; it assumes a bivariate normal population, and is the large-sample formula, not an exact one. The bootstrap standard error with 2,000 resamples has a median of 0.189 over 30 seeds (interquartile range 0.187 to 0.192). The infinitesimal jackknife, defined next, gives 0.163. Four methods, four numbers between 0.16 and 0.20, from twelve observations; the next two sections explain why they differ in the directions they do.
Leaving one out as a derivative
Here is the idea. A statistic that depends on the data only through their empirical distribution $\hat F$ is a function $T(\hat F)$. Leaving out $x_i$ moves $\hat F$ a small step in a particular direction, and the change in $T$ is approximately the step size times the derivative of $T$ in that direction. That derivative is the influence function.
The influence function. For a functional $T$ and a distribution $G$, the influence function of $T$ at $G$ is $$\mathrm{IF}(x; T, G) = \lim_{\varepsilon \to 0}\frac{T\big((1 - \varepsilon)G + \varepsilon\delta_x\big) - T(G)}{\varepsilon},$$ where $\delta_x$ is the distribution that puts all its mass at $x$. It measures the effect on $T$ of adding an infinitesimal amount of contamination at $x$. Hampel (1974) treated it as the first derivative of an estimator viewed as a functional and used it to study robustness; his abstract notes relations to the jackknife. Its empirical version is $\mathrm{IF}_i = \mathrm{IF}(x_i; T, \hat F)$.
Leaving out $x_i$ is a step of size $-1/(n - 1)$. Let $\hat F_{(i)}$ be the empirical distribution of the data without $x_i$, which puts mass $1/(n - 1)$ on each $x_j$, $j \ne i$. Then $$\hat F_{(i)} = (1 - \varepsilon)\hat F + \varepsilon\,\delta_{x_i}\qquad\text{with } \varepsilon = -\frac{1}{n - 1}.$$ To check it, compute the mass at each point. At $x_i$ the right side gives $(1 - \varepsilon)\frac1n + \varepsilon = \frac1n\cdot\frac{n}{n - 1} - \frac{1}{n - 1} = 0$. At $x_j$, $j \ne i$, it gives $(1 - \varepsilon)\frac1n = \frac1n\cdot\frac{n}{n - 1} = \frac{1}{n - 1}$. (With repeated values, add the masses.) So $\hat\theta_{(i)} = T\big((1 - \varepsilon)\hat F + \varepsilon\delta_{x_i}\big)$ at $\varepsilon = -1/(n - 1)$, and if $T$ is differentiable in this direction, $$\hat\theta_{(i)} - \hat\theta \approx -\frac{\mathrm{IF}_i}{n - 1}, \qquad\text{that is,}\qquad \tilde\theta_i = \hat\theta + (n - 1)\big(\hat\theta - \hat\theta_{(i)}\big) \approx \hat\theta + \mathrm{IF}_i.$$ The pseudo-value is the estimate plus a finite-difference influence value.
The infinitesimal jackknife. Replacing the finite difference by the derivative in Proposition 1 gives $$v_{IJ} = \frac{1}{n^2}\sum_{i=1}^n \mathrm{IF}_i^2,$$ the infinitesimal jackknife estimate of variance. (The empirical influence values sum to zero for the statistics met here, as for the correlation below, so $\sum_i\mathrm{IF}_i^2$ is $n - 1$ times their sample variance, and the divisor $n^2$ rather than $n(n - 1)$ is the convention.) Wager, Hastie and Efron (2014) attribute it to Jaeckel (1972), who introduced it as a practical approximation to the first-order variance of an estimator, and note that it is also called the nonparametric delta method. The name fits: for a linear statistic $\frac1n\sum g(x_i)$, $\mathrm{IF}_i = g(x_i) - \bar g$ and $v_{IJ} = \frac{1}{n^2}\sum(g(x_i) - \bar g)^2$, the plug-in variance of the mean of the $g$ values, and for a smooth function of several means it is what The Delta Method gives with the population moments replaced by sample ones.
The influence function of the correlation. Write the population correlation as $T(G) = \sigma_{xy}/(\sigma_x\sigma_y)$, a function of three moments of $G$. The derivatives of the moments are found first. Under $G_\varepsilon = (1 - \varepsilon)G + \varepsilon\delta_{(x_0, y_0)}$ the mean of $x$ is $\mu_x + \varepsilon(x_0 - \mu_x)$, and $$\sigma_{xy}(G_\varepsilon) = (1 - \varepsilon)\big[\sigma_{xy} + \varepsilon^2(x_0 - \mu_x)(y_0 - \mu_y)\big] + \varepsilon(1 - \varepsilon)^2(x_0 - \mu_x)(y_0 - \mu_y),$$ by computing the covariance about the moved means. Its derivative at $\varepsilon = 0$ is $(x_0 - \mu_x)(y_0 - \mu_y) - \sigma_{xy}$; setting $y = x$ gives $(x_0 - \mu_x)^2 - \sigma_x^2$ for the variance. Write $u = (x_0 - \mu_x)/\sigma_x$ and $v = (y_0 - \mu_y)/\sigma_y$ and let $\rho = T(G)$. The chain rule applied to $\log T = \log\sigma_{xy} - \tfrac12\log\sigma_x^2 - \tfrac12\log\sigma_y^2$ gives $$\frac{\mathrm{IF}}{\rho} = \frac{\sigma_x\sigma_y\,uv - \sigma_{xy}}{\sigma_{xy}} - \frac12(u^2 - 1) - \frac12(v^2 - 1),$$ and multiplying by $\rho = \sigma_{xy}/(\sigma_x\sigma_y)$, $$\mathrm{IF}\big((x_0, y_0); T, G\big) = uv - \rho - \frac{\rho}{2}(u^2 - 1) - \frac{\rho}{2}(v^2 - 1) = uv - \frac{\rho}{2}\big(u^2 + v^2\big).$$ (The logarithm needs $\sigma_{xy} \ne 0$; multiplying through shows the final formula also holds when it is 0.) At $\hat F$ the $u$ and $v$ are the observations standardized with the plug-in means and standard deviations, so $\sum u_i v_i = n r$ and $\sum u_i^2 = \sum v_i^2 = n$, and the empirical influence values sum to $nr - \frac r2(2n) = 0$. The script checks the formula for case 1 against a numerical derivative with $\varepsilon = 10^{-6}$.
The last column of the table lists these values. They track the pseudo-values minus $\hat\theta$ closely for most cases, and less well for some. In relative terms the worst is case 4 (0.114 against 0.031), and in absolute terms case 11 ($-1.697$ against $-1.973$): the finite step $-1/11$ is large enough for the curvature of $T$ to matter. The infinitesimal jackknife standard error is $\sqrt{v_{IJ}} = 0.163$, smaller than the jackknife's 0.197.
Influence functions in machine learning. The same first-order step appears in the study of trained models. Koh and Liang (2017) compute the influence of a training point $z$ on the parameters of a model fitted by minimizing an average loss as $\mathcal{I}(z) = -H^{-1}\nabla_\theta L(z, \hat\theta)$, where $H$ is the Hessian of the average loss, and approximate the effect of removing $z$, "the same as upweighting it by $\epsilon = -\frac1n$", by $\hat\theta_{-z} - \hat\theta \approx -\frac1n\mathcal{I}(z)$. That is the jackknife's step, with $1/n$ in place of $1/(n - 1)$ because the loss is averaged over the original $n$ points. They name Hampel (1974) and Jaeckel (1972) as the origins of these diagnostics. Studying Large Language Model Generalization with Influence Functions describes how the method was scaled to language models of up to 52 billion parameters, where retraining without a sequence is impossible and only the derivative is available.
The derivative view also explains Efron's (1979) description of the jackknife as a linear approximation to the bootstrap. The bootstrap reweights all the observations at random at once; the jackknife and the infinitesimal jackknife look only at the first-order effect of each observation separately. For a statistic that is close to linear in $\hat F$ the two agree; for a statistic with strong curvature, or with a derivative that does not exist, they part, which is what the last section shows.
Regression coefficients: the jackknife is HC3
For least-squares coefficients the leave-one-out values have a closed form, so the jackknife needs no refitting. Leverage, Influence and Residual Diagnostics proves, as part 1 of its Theorem 1, that for a full-rank $n \times p$ design $X$ with rows $\mathbf{x}_i^{\mathsf T}$, $M = X^{\mathsf T}X$, residuals $e_i$ and leverages $h_{ii} < 1$, $$\hat{\boldsymbol\beta}_{(i)} = \hat{\boldsymbol\beta} - M^{-1}\mathbf{x}_i\,\tilde e_i, \qquad \tilde e_i = \frac{e_i}{1 - h_{ii}},$$ where $\tilde e_i$ is the deleted residual. The jackknife covariance matrix replaces the square in $v_J$ by an outer product.
Theorem 2 (MacKinnon and White's HC3). The jackknife covariance matrix of $\hat{\boldsymbol\beta}$, $$V_J = \frac{n - 1}{n}\sum_{i=1}^n\big(\hat{\boldsymbol\beta}_{(i)} - \hat{\boldsymbol\beta}_{(\cdot)}\big)\big(\hat{\boldsymbol\beta}_{(i)} - \hat{\boldsymbol\beta}_{(\cdot)}\big)^{\mathsf T},$$ equals $$V_J = \frac{n - 1}{n}\,M^{-1}\Big(X^{\mathsf T}\tilde\Omega X - \frac1n X^{\mathsf T}\tilde{\mathbf e}\,\tilde{\mathbf e}^{\mathsf T}X\Big)M^{-1},$$ where $\tilde\Omega$ is the diagonal matrix with entries $\tilde e_i^2$ and $\tilde{\mathbf e}$ the vector of deleted residuals.
Proof. Write $\mathbf a_i = \mathbf x_i\tilde e_i$, so $\hat{\boldsymbol\beta}_{(i)} = \hat{\boldsymbol\beta} - M^{-1}\mathbf a_i$ and $\hat{\boldsymbol\beta}_{(\cdot)} = \hat{\boldsymbol\beta} - M^{-1}\bar{\mathbf a}$, with $\bar{\mathbf a} = \frac1n\sum_i\mathbf a_i = \frac1n X^{\mathsf T}\tilde{\mathbf e}$. Then $\hat{\boldsymbol\beta}_{(i)} - \hat{\boldsymbol\beta}_{(\cdot)} = -M^{-1}(\mathbf a_i - \bar{\mathbf a})$, and since $M^{-1}$ is symmetric, $$\sum_i\big(\hat{\boldsymbol\beta}_{(i)} - \hat{\boldsymbol\beta}_{(\cdot)}\big)\big(\cdot\big)^{\mathsf T} = M^{-1}\Big(\sum_i(\mathbf a_i - \bar{\mathbf a})(\mathbf a_i - \bar{\mathbf a})^{\mathsf T}\Big)M^{-1} = M^{-1}\Big(\sum_i\mathbf a_i\mathbf a_i^{\mathsf T} - n\bar{\mathbf a}\bar{\mathbf a}^{\mathsf T}\Big)M^{-1}.$$ Now $\sum_i\mathbf a_i\mathbf a_i^{\mathsf T} = \sum_i\tilde e_i^2\,\mathbf x_i\mathbf x_i^{\mathsf T} = X^{\mathsf T}\tilde\Omega X$ and $n\bar{\mathbf a}\bar{\mathbf a}^{\mathsf T} = \frac1n X^{\mathsf T}\tilde{\mathbf e}\tilde{\mathbf e}^{\mathsf T}X$. Multiplying by $(n - 1)/n$ gives the formula. $\square$
MacKinnon and White (1985) derived this matrix as the jackknife covariance, from the same deletion formula, and named it HC3. (The version read for this article is their 1983 Queen's University working paper, equations 10 to 12.) It belongs to a family of heteroskedasticity-consistent estimators that do not assume a common error variance. The simplest, White's (1980), which MacKinnon and White call HC and The Bootstrap Method calls HC0 (the name used here), is $M^{-1}X^{\mathsf T}\Omega X M^{-1}$ with the squared ordinary residuals $e_i^2$ on the diagonal of $\Omega$; that page compares it with the pairs bootstrap. HC3 differs in two ways: it inflates each squared residual by $(1 - h_{ii})^{-2}$, and it subtracts the small centering term. Since $\mathbf x_i\tilde e_i$ has mean near zero, the centering term is of lower order, and MacKinnon and White note that omitting the factor $(n - 1)/n$ usually changes little. Their sampling experiments led them to conclude that "among the heteroskedasticity-consistent estimators, HC3 is clearly the procedure of choice". The estimator now usually called HC3, and the one used in Linear Regression Assumptions (which calls it a simplified form of MacKinnon and White's jackknife estimator), drops the centering term and the factor $(n - 1)/n$, leaving $M^{-1}X^{\mathsf T}\tilde\Omega XM^{-1}$. This article keeps the full jackknife of Theorem 2.
For the twelve adults, with pressure regressed on age, sex and BMI, the script refits the model twelve times. The refitted covariance equals the closed form of Theorem 2 (the last column):
| Coefficient | Estimate | Classical SE | HC0 SE | Jackknife SE |
|---|---|---|---|---|
| age | 0.187 | 0.219 | 0.204 | 0.388 |
| BMI | 1.379 | 0.877 | 0.853 | 1.729 |
The jackknife standard errors are about twice the others, and one observation accounts for most of the difference. Case 2 is the 39-year-old woman with a BMI of 45.1 whose deletion turns the age slope negative (Leverage, Influence and Residual Diagnostics opens with her). Her leverage is 0.573, so her deleted residual is her residual divided by 0.427, and her leave-one-out age slope is $-0.155$. Her share of the sum of squared deviations behind the jackknife variance of the age slope is 0.68. HC0 sees only her ordinary residual, which a high-leverage point keeps small; the jackknife sees what happens when she is removed. Which is closer to the truth for twelve observations is not something one sample can settle. The simplified HC3 gives 0.406 for age and 1.812 for BMI, about 5% above the full jackknife: here the factor $(n - 1)/n = 11/12$ and the small centering term both pull the jackknife down. Linear Regression Assumptions measured, by simulation, the coverage of intervals built on the simplified HC3 for a slope when the error variance grows with the input, and found that they recovered part of the coverage the textbook interval lost.
Is the jackknife variance too large?
The comparison of the four standard errors of the correlation put the jackknife at the top. One sample's ordering shows nothing general, but Theorem 3 shows that, measured against the variance it naturally targets, the jackknife sum of squares is too large on average for every statistic that is not linear.
Here is the plan. Any square-integrable symmetric statistic of independent observations splits into uncorrelated pieces: a part that depends on the observations one at a time, a part that depends on them two at a time, and so on. The variance of the statistic is the sum of the variances of these pieces. The jackknife counts each piece's variance once for the one-at-a-time part, twice for the two-at-a-time part, and so on. So it is exact for linear statistics and too large for all others.
Lemma (the ANOVA decomposition). Let $X_1, \dots, X_m$ be independent with a common distribution, and $S = S(X_1, \dots, X_m)$ a symmetric function with $E(S^2) < \infty$. For a subset $A \subseteq \{1, \dots, m\}$ write $X_A$ for the observations indexed by $A$, $E_A = E(S \mid X_A)$ (with $E_\emptyset = E(S)$), and $$\gamma_A = \sum_{B \subseteq A}(-1)^{\lvert A\rvert - \lvert B\rvert}\,E_B.$$ Then (1) $S = \sum_{A \subseteq \{1, \dots, m\}}\gamma_A$; (2) $E(\gamma_A \mid X_C) = 0$ whenever $A \not\subseteq C$; (3) $E(\gamma_A\gamma_B) = 0$ for $A \ne B$; (4) $E(\gamma_A^2)$ depends on $A$ only through $k = \lvert A\rvert$; call it $\sigma_k^2$. Consequently $$\operatorname{Var}(S) = \sum_{k=1}^m V_k, \qquad V_k = \binom{m}{k}\sigma_k^2.$$
Proof. (1) By Möbius inversion on subsets: $\sum_{A \subseteq [m]}\gamma_A = \sum_{B}E_B\sum_{A \supseteq B}(-1)^{\lvert A\rvert - \lvert B\rvert}$, and the inner sum runs over the subsets of $[m] \setminus B$ with alternating signs, which is $(1 - 1)^{m - \lvert B\rvert} = 0$ unless $B = [m]$, where it is 1. So the total is $E_{[m]} = S$. (2) By independence, conditioning $E_B$ on $X_C$ averages out the observations in $B$ outside $C$: $E(E_B \mid X_C) = E_{B \cap C}$. Pick $j \in A \setminus C$. Pair each $B \subseteq A$ not containing $j$ with $B \cup \{j\}$. The two have the same intersection with $C$, because $j \notin C$, and opposite signs in the sum defining $\gamma_A$, so the conditional expectations cancel in pairs. (3) If $A \ne B$, one of them is not contained in the other; say $A \not\subseteq B$. Then $E(\gamma_A\gamma_B) = E\big(\gamma_B\,E(\gamma_A \mid X_B)\big) = 0$ by (2), since $\gamma_B$ is a function of $X_B$. (4) Symmetry of $S$ and the common distribution make the joint law of $(X_A, S)$ the same for all $A$ of one size, after relabeling. Finally $\gamma_\emptyset = E(S)$, and by (1) and (3) $\operatorname{Var}(S) = \sum_{A \ne \emptyset}E(\gamma_A^2) = \sum_k\binom mk\sigma_k^2$. $\square$
Theorem 3 (Efron and Stein). Let $S$ be a symmetric statistic of $m = n - 1$ observations with $E(S^2) < \infty$, let $X_1, \dots, X_n$ be independent from the common distribution, $S_{(i)}$ the statistic computed without $X_i$, and $D = \sum_{i=1}^n(S_{(i)} - S_{(\cdot)})^2$. Then $$E(D) = \sum_{k=1}^{n-1}k\,V_k \;\ge\; \sum_{k=1}^{n-1}V_k = \operatorname{Var}(S),$$ with equality only when $V_k = 0$ for every $k \ge 2$.
Two remarks about the target before the proof. $D$ is computed from $n$ observations but estimates the variance of the statistic computed from $n - 1$, which is the natural object, since the leave-one-out values are statistics of $n - 1$ observations. Efron and Stein's abstract describes their theorem as proved "for the natural jackknife estimate of $\operatorname{Var} S(X_1, X_2, \cdots, X_{n-1})$ based on $X_1, X_2, \cdots, X_n$", and Wager, Hastie and Efron (2014, eq. 23) state the identity $E(V_J) = V_1 + 2V_2 + \dots + nV_n$ for the jackknife computed on $n + 1$ points. Tukey's $v_J = \frac{n - 1}{n}D$ is used instead as an estimate of the variance of the statistic computed from all $n$ observations. For a linear statistic in this article's sense, an average $\frac1n\sum_i g(x_i)$ (Worked problem 4), that variance is exactly $\frac{n - 1}{n}$ times the variance from $n - 1$ observations, which is what the factor corrects; for other statistics Theorem 3 says nothing exact about $v_J$ as an estimate of $\operatorname{Var}(S_n)$, because the ratio of $\operatorname{Var}(S_n)$ to $\operatorname{Var}(S_{n-1})$ is then not exactly $(n - 1)/n$. In the exact check below, Tukey's $v_J$ for the plug-in variance from four observations has expectation 0.672 against $\operatorname{Var}(S_4) = 0.397$, an excess of 0.275, close to $\frac34$ of Theorem 3's excess, $\frac34(V_2 + 2V_3) = 0.269$. The agreement is not a rule. For the sample total $S_n = \sum_i x_i$ of the same values the excess is negative: $E(v_J) = 7/2$ against $\operatorname{Var}(S_4) = 56/9$. A total has $V_k = 0$ for $k \ge 2$, but its variance grows with $n$, so $\operatorname{Var}(S_n)/\operatorname{Var}(S_{n-1}) = n/(n - 1)$, not $(n - 1)/n$. The simulation below measures the net effect for the correlation.
Proof. Apply the lemma to $S$ on $m = n - 1$ arguments, and extend the components to subsets of $\{1, \dots, n\}$: for $A \subseteq \{1, \dots, n\}$ with $\lvert A\rvert = k \le n - 1$, let $\gamma_A$ be the component of order $k$ evaluated at $X_A$ (by symmetry, the component depends on $A$ only through the values $X_A$). Then $S_{(i)} = \sum_{A \not\ni i}\gamma_A$, the sum over the subsets of $\{1, \dots, n\} \setminus \{i\}$. Each $A$ of size $k$ is missing from exactly $n - k$ of the $n$ sets, so $$S_{(\cdot)} = \sum_A\frac{n - k}{n}\gamma_A, \qquad S_{(i)} - S_{(\cdot)} = \sum_A\Big(\mathbf 1[i \notin A] - \frac{n - k}{n}\Big)\gamma_A.$$ The components are uncorrelated (part 3 of the lemma applies to any two distinct subsets of $\{1, \dots, n\}$, by the same argument), so $$E\sum_i\big(S_{(i)} - S_{(\cdot)}\big)^2 = \sum_A\sigma_{\lvert A\rvert}^2\sum_{i=1}^n\Big(\mathbf 1[i \notin A] - \frac{n - k}{n}\Big)^2.$$ For $\lvert A\rvert = k$, the $k$ indices in $A$ contribute $\big(\frac{n - k}{n}\big)^2$ each and the $n - k$ others $\big(\frac kn\big)^2$ each, a total of $\frac{k(n - k)^2 + (n - k)k^2}{n^2} = \frac{k(n - k)}{n}$. There are $\binom nk$ subsets of size $k$, and $\binom nk\frac{n - k}{n} = \binom{n - 1}{k}$. So the expectation is $\sum_k k\binom{n - 1}{k}\sigma_k^2 = \sum_k kV_k$, where $V_k = \binom{n-1}{k}\sigma_k^2$ is the order-$k$ variance of $S$ on $n - 1$ arguments. $\square$
The proof shows where the excess comes from: a component that involves $k$ observations is disturbed by deleting any of its $k$ members, and the jackknife counts it $k$ times. For a linear statistic only $V_1$ is nonzero and the jackknife is unbiased, as found above.
An exact check. Take three equally likely values 0, 1 and 3, and $S$ the plug-in variance of three observations. Enumerating all 27 samples in rational arithmetic gives $\operatorname{Var}(S) = 392/729 = 0.5377$, with components $V_1 = 392/2187$, $V_2 = 784/2187$ and $V_3$ = 0 (the plug-in variance involves pairs of observations at most). Enumerating all 81 samples of four observations gives $E\sum_i(S_{(i)} - S_{(\cdot)})^2 = 1960/2187 = 0.8962$, which is $V_1 + 2V_2$ exactly, two-thirds larger than the variance.
The correlation of the NHANES adults. The script draws samples of 12 from the 5,176 adults and computes the leave-one-out correlations of age and pressure in each (30 seeds of 20,000 samples; medians over seeds). Against the target of Theorem 3, the actual variance of $r$ in samples of 11 adults (standard deviation 0.236), the average of $D$ is larger by a ratio of 1.232 (interquartile range 1.225 to 1.241). That 23% excess is $V_2 + 2V_3 + \cdots$ for this statistic and population, which Theorem 3 says cannot be negative. In practice the question is how Tukey's $v_J$ compares with the variance of $r$ from the 12 adults it was computed from. That standard deviation is 0.223, while the root of the average $v_J$ is 0.250: for samples of 12 adults the jackknife variance is too large by a ratio of 1.256 on average. It fits the jackknife's place at the top of the four estimates computed earlier from the twelve.
The infinitesimal jackknife errs the other way. Wager, Hastie and Efron (2014, eq. 24) note that its expectation is approximately $V_1$, so it drops the higher-order terms rather than inflating them, and the two biases partially cancel. Its value for the twelve adults, 0.163, below the other three, fits that description, though one sample cannot confirm it.
Where the jackknife fails: the median
The derivative view predicts trouble for a statistic that is not smooth in $\hat F$. The median is the standard example, and the twelve adults show it in the most extreme form.
Example (a standard error of zero). The twelve pressures, sorted, have sixth and seventh values 119.333 and 119.333: two adults share the reading. The median is their average, 119.333. Leaving out any one adult leaves eleven values, whose median is the sixth of the remaining ones, and that is 119.333 whichever adult is removed. All twelve leave-one-out medians are equal, so the jackknife standard error is 0.000. The bootstrap standard error from 2,000 resamples has a median over 30 seeds of 4.070 mm Hg.
The zero is an extreme case of a general defect. For an even sample size $n = 2m$, write $x_{(1)} \le \dots \le x_{(n)}$ for the ordered values. Deleting any of the $m$ observations at or below $x_{(m)}$ leaves $x_{(m+1)}$ as the middle of the remaining $2m - 1$ values, and deleting any of the others leaves $x_{(m)}$. So the leave-one-out medians take the value $x_{(m+1)}$ $m$ times and $x_{(m)}$ $m$ times, their average is $\frac12(x_{(m)} + x_{(m+1)})$, and $$v_J = \frac{n - 1}{n}\cdot n\cdot\frac14\big(x_{(m+1)} - x_{(m)}\big)^2 = \frac{n - 1}{4}\big(x_{(m+1)} - x_{(m)}\big)^2.$$ The jackknife variance depends on the data only through a single gap between two adjacent order statistics. One gap is a noisy measure of the density near the median, however large $n$ is.
Theorem 4 (inconsistency of the jackknife for the median). Let $X_1, \dots, X_n$ be independent from a distribution with distribution function $F$, median $\theta$, and a density $f$ that is continuous and positive in a neighborhood of $\theta$. For $n = 2m$ even, $$4f(\theta)^2\,n\,v_J \;\to\; E^2 \quad\text{in distribution as } m \to \infty,$$ where $E$ is a standard exponential variable. Since $1/(4nf(\theta)^2)$ is the variance of the large-sample normal distribution of the sample median, the ratio of the jackknife variance to it converges to $E^2$, not to 1: the jackknife variance is not consistent.
Proof. Step 1: the gap for uniform data. Let $U_1, \dots, U_n$ be independent uniform on $(0, 1)$ with order statistics $U_{(k)}$. The joint density of $(U_{(m)}, U_{(m+1)})$ at $u < v$ is $$c\,u^{m-1}(1 - v)^{n - m - 1}, \qquad c = \frac{n!}{(m - 1)!\,(n - m - 1)!},$$ by counting: the event that one observation falls in $[u, u + du]$, one in $[v, v + dv]$, $m - 1$ below $u$ and $n - m - 1$ above $v$ has probability $c\,u^{m-1}(1 - v)^{n-m-1}du\,dv$, the multinomial coefficient $c$ counting the ways to assign the $n$ labeled observations to the four groups. The gap $D = U_{(m+1)} - U_{(m)}$ therefore has density, at $0 < d < 1$, $$\int_0^{1-d}c\,u^{m-1}(1 - u - d)^{n-m-1}\,du = c\,(1 - d)^{n-1}\int_0^1 t^{m-1}(1 - t)^{n-m-1}\,dt = c\,(1 - d)^{n-1}\frac{(m - 1)!\,(n - m - 1)!}{(n - 1)!} = n(1 - d)^{n-1},$$ substituting $u = (1 - d)t$ and using the beta integral. So $P(D > d) = (1 - d)^n$, and for $s > 0$, $P(nD > s) = (1 - s/n)^n \to e^{-s}$: $nD \to E$ in distribution. Step 2: from uniform to $F$. Write $X_{(k)} = F^{-1}(U_{(k)})$, which has the distribution of the order statistics of the $X$ sample. Near $\frac12$ the inverse $F^{-1}$ is differentiable with derivative $1/f(F^{-1}(u))$, so by the mean value theorem $X_{(m+1)} - X_{(m)} = D/f(\xi)$ for some $\xi$ between $X_{(m)}$ and $X_{(m+1)}$, whenever both lie in the neighborhood. Both converge in probability to $\theta$ (the sample median is consistent, as The Sampling Distribution of the Mean shows in its section on the median), so $f(\xi) \to f(\theta)$ in probability, and $n f(\theta)(X_{(m+1)} - X_{(m)}) \to E$ in distribution by Slutsky's theorem (Convergence in Distribution and the Proof of the Central Limit Theorem). Step 3. $4f(\theta)^2 n v_J = \frac{n - 1}{n}\big(nf(\theta)(X_{(m+1)} - X_{(m)})\big)^2 \to E^2$ by the continuous mapping theorem. The normal limit of the median with standard deviation $1/(2f(\theta)\sqrt n)$ is derived, in outline, in the same section of The Sampling Distribution of the Mean. $\square$
The limit is a whole distribution, $E^2$, whose median is $(\ln 2)^2 = 0.480$ and whose mean is $E(E^2) = 2$. Half the time the jackknife variance is less than half the right value, while on average it is twice the right value. The probability that it is off by more than a factor of 2 in either direction is $P(E^2 < \tfrac12) + P(E^2 > 2) = (1 - e^{-\sqrt{1/2}}) + e^{-\sqrt 2} = 0.750$. The script draws normal samples and compares $v_J$ with the actual variance of the median across the same 10,000 samples (30 seeds; medians over seeds):
| $n$ | Median of $v_J/\operatorname{Var}$ | Mean of $v_J/\operatorname{Var}$ | Share off by more than a factor of 2 |
|---|---|---|---|
| 20 | 0.505 | 1.89 | 0.743 |
| 100 | 0.487 | 1.98 | 0.748 |
| 1,000 | 0.479 | 1.99 | 0.750 |
The ratio does not settle as $n$ grows; it converges to the random limit, as the theorem says. This failure is one reason the jackknife cannot replace the bootstrap. Informally, and without a proof here, the bootstrap reweights many observations at once and so looks at the region around the median rather than a single gap. What is established: Singh (1981) obtained the exact rate at which the bootstrap distribution of a sample quantile converges, and The Bootstrap Method found, by simulation from an exponential population, bootstrap standard errors of the median of 25 values centered near the true value (mean 0.217 against 0.201) and percentile intervals for the median covering 0.952 at $n = 25$ and 0.946 at $n = 80$.
The jackknife can be repaired by deleting more. The delete-$d$ jackknife recomputes the statistic on every subset of size $n - d$. Shao and Wu (1989) give a general theory of when it works, and state that for sample quantiles it is consistent and asymptotically unbiased when $d$ satisfies $n^{1/2}/d \to 0$ and $n - d \to \infty$. Deleting many observations at once moves the empirical distribution by more than a single gap, which is what the median needs.
What is proved here, what is cited, what the experiment shows
Proved here. The pseudo-value identities (Proposition 1). Bias reduction under an expansion of the bias in powers of $1/n$ (Theorem 1), with the exact identities for the plug-in variance and the square of a mean (Corollaries 1 and 2). The exactness of $v_J$ for linear statistics. That deleting one observation is a step of $-1/(n - 1)$ toward a point mass, and the influence function of the correlation. The equality of the jackknife covariance of least-squares coefficients with HC3 (Theorem 2), given the deletion identity proved in Leverage, Influence and Residual Diagnostics. The ANOVA decomposition and the Efron–Stein identity (Lemma and Theorem 3). The limit of the jackknife variance for the median (Theorem 4), given the consistency and the normal limit of the median from The Sampling Distribution of the Mean and Slutsky's theorem.
Cited. The history (Miller, 1974; Efron and Stein, 1981; Wager, Hastie and Efron, 2014, for Jaeckel's infinitesimal jackknife). Efron's (1979) description of the jackknife as a linear approximation to the bootstrap, MacKinnon and White's (1985) naming of HC3 and their simulation findings, the consistency of the delete-$d$ jackknife for quantiles (Shao and Wu, 1989), Singh's (1981) result for the bootstrap of quantiles, and Koh and Liang's (2017) influence approximation.
What the experiments show. On the twelve adults: the numerical identities (Corollaries 1 and 2, Theorem 2 against explicit refits), the four standard errors of the correlation, the dominance of one case in the regression jackknife, and the zero standard error of the median. In simulation: the upward bias of the jackknife variance of the correlation for samples of 12 NHANES adults (a ratio of 1.256 to the actual variance), and the distribution of the jackknife variance of the normal median at three sample sizes. These are observations of these populations and sample sizes.
Worked problems
The four problems are posed here, on small exact cases and on the twelve adults; the article's script checks each number.
Problem 1 (the median of four numbers). Compute the leave-one-out medians, the pseudo-values, the jackknife bias and the jackknife variance of the median of 3, 5, 6, 10.
Solution. The bias estimate is 0 and the jackknife variance is 0.75. With $n = 4$ and $m = 2$, deleting 3 or 5 leaves a median of 6, and deleting 6 or 10 leaves 5. The leave-one-out medians are $6, 6, 5, 5$, with average 5.5, equal to the full-sample median $(5 + 6)/2 = 5.5$, so $b_J = 3(5.5 - 5.5) = 0$. The pseudo-values are $4 \times 5.5 - 3 \times 6 = 4$ (twice) and $4 \times 5.5 - 3 \times 5 = 7$ (twice). By the definition, $v_J = \frac34\big[4 \times (0.5)^2\big] = 0.75$, which agrees with the formula $\frac{n - 1}{4}(x_{(3)} - x_{(2)})^2 = \frac34(6 - 5)^2$. The value 10 could be replaced by any number above 6, and 3 by any number below 5, without changing anything: the jackknife variance of a median sees only the two middle values.
Problem 2 (the square of a mean). For the data 1, 2, 6, compute the jackknife bias-corrected estimate of $\mu^2$ from $\bar x^2$ and check Corollary 2.
Solution. The corrected estimate is $20/3 \approx 6.667$, which is $\bar x^2 - s^2/n$. Here $\bar x = 3$ and $\bar x^2 = 9$. The leave-one-out means are $(2 + 6)/2 = 4$, $(1 + 6)/2 = 3.5$ and $(1 + 2)/2 = 1.5$, with squares 16, 12.25 and 2.25, averaging $30.5/3 = 61/6$. The bias estimate is $b_J = 2(61/6 - 9) = 7/3$, and $\hat\theta_J = 9 - 7/3 = 20/3$. On the other side, $s^2 = \frac{(1 - 3)^2 + (2 - 3)^2 + (6 - 3)^2}{2} = \frac{14}{2} = 7$, so $s^2/n = 7/3$, the same bias estimate, and $\bar x^2 - s^2/n = 20/3$.
Problem 3 (the coefficient of variation). The coefficient of variation $s/\bar x$ of the twelve pressures measures their spread relative to their level. Find its jackknife bias, bias-corrected value and standard error.
Solution. The coefficient of variation is 0.1197; the jackknife bias is $-0.0032$, the corrected value 0.1229 and the standard error 0.0256. The computation follows the definitions. Recompute $s/\bar x$ twelve times with one adult left out and average the twelve values. Multiply the difference between that average and 0.1197 by 11 to get $b_J$, and take $\frac{11}{12}$ times the sum of squared deviations of the twelve values for $v_J$. The correction is about an eighth of a standard error, so as with the correlation it is not worth making. Theorem 1 justifies the correction only if the bias of $s/\bar x$ has an expansion of the form it assumes. That is not proved here; it needs at least that $\bar x$ stay away from zero (for example, positive observations bounded away from zero and bounded above, as blood pressures are), since for normal data $E(1/\bar x)$ does not even exist.
Problem 4 (linear statistics). Show that for $\hat\theta = \frac1n\sum_i g(x_i)$ the pseudo-values are $g(x_1), \dots, g(x_n)$, so that $b_J = 0$ and $v_J = s_g^2/n$.
Solution. Write $g_i = g(x_i)$ and $\bar g = \frac1n\sum_j g_j$. Then $\hat\theta_{(i)} = \frac{n\bar g - g_i}{n - 1}$, and $$\tilde\theta_i = n\bar g - (n - 1)\frac{n\bar g - g_i}{n - 1} = n\bar g - n\bar g + g_i = g_i.$$ By Proposition 1, $\hat\theta_J$ is the mean of the $g_i$, which is $\hat\theta$, so $b_J = 0$; and $v_J$ is the sample variance of the $g_i$ divided by $n$, which is $s_g^2/n$. Since $E(s_g^2) = \operatorname{Var}(g(X))$, $E(v_J) = \operatorname{Var}(\hat\theta)$ exactly, the case $V_k = 0$ for $k \ge 2$ of Theorem 3.
Summary
- The jackknife recomputes a statistic with each observation left out. $b_J = (n - 1)(\hat\theta_{(\cdot)} - \hat\theta)$ estimates the bias and $v_J = \frac{n - 1}{n}\sum(\hat\theta_{(i)} - \hat\theta_{(\cdot)})^2$ the variance; both are summaries of the pseudo-values $n\hat\theta - (n - 1)\hat\theta_{(i)}$.
- The bias correction removes the $1/n$ term of a bias. It turns the plug-in variance into $s^2$ and $\bar x^2$ into $\bar x^2 - s^2/n$, exactly.
- Leaving one out is a step of $-1/(n - 1)$ toward a point mass, so pseudo-values are finite-difference influence values. The infinitesimal jackknife $\frac{1}{n^2}\sum\mathrm{IF}_i^2$ is the nonparametric delta method, and the influence functions used to study trained models take the same step.
- For least-squares coefficients the jackknife covariance is MacKinnon and White's HC3, which weights each squared residual by $(1 - h_{ii})^{-2}$.
- On average the jackknife sum of squares is $\sum kV_k$ against the true $\sum V_k$: exact for linear statistics, too large otherwise. For the correlation of twelve NHANES adults, Tukey's $v_J$ averaged 1.256 times the actual variance.
- For the median the jackknife variance depends on one gap between order statistics and does not converge; its ratio to the truth tends to $E^2$. Deleting $d$ observations at a time, with $d$ growing faster than $\sqrt n$, repairs it.
What we have learned
Twelve recomputations of a statistic, each without one observation, estimate its bias and its variance. The estimates are exact for means and other linear statistics, accurate to first order for smooth statistics, and, as a sum of squares measured against the variance from $n - 1$ observations, too large on average by an amount the ANOVA decomposition makes exact. Their accuracy depends on the statistic being differentiable as a function of the empirical distribution, and the median shows what happens when it is not. The bootstrap, which The Plug-In Principle and Why the Bootstrap Works justifies and Bootstrap Confidence Intervals: Percentile, Basic, Studentized and BCa turns into intervals, does not depend on a single gap in this way; Singh (1981) obtained the rate at which its distribution for a sample quantile converges. Both methods, as used here, assume independent observations. Resampling Dependent and Clustered Data shows what goes wrong when observations come in sequences or groups, and how deleting or resampling whole blocks, the idea Künsch (1989) applied to both methods, puts it right.
References
- Efron, B. (1979). Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7(1), 1–26. https://doi.org/10.1214/aos/1176344552 record · 3 pages
- Efron, B., and Stein, C. (1981). The jackknife estimate of variance. The Annals of Statistics, 9(3), 586–596. https://doi.org/10.1214/aos/1176345462 record
- Hampel, F. R. (1974). The influence curve and its role in robust estimation. Journal of the American Statistical Association, 69(346), 383–393. https://doi.org/10.1080/01621459.1974.10482962 record · 2 pages
- Jaeckel, L. A. (1972). The infinitesimal jackknife. Unpublished memorandum, Bell Telephone Laboratories, Murray Hill, NJ (as cited by Wager, Hastie and Efron, 2014, and Koh and Liang, 2017). record
- Koh, P. W., and Liang, P. (2017). Understanding black-box predictions via influence functions. Proceedings of the 34th International Conference on Machine Learning, PMLR 70, 1885–1894. arXiv:1703.04730. record · 2 pages
- Künsch, H. R. (1989). The jackknife and the bootstrap for general stationary observations. The Annals of Statistics, 17(3), 1217–1241. https://doi.org/10.1214/aos/1176347265 record · 2 pages
- MacKinnon, J. G., and White, H. (1985). Some heteroskedasticity-consistent covariance matrix estimators with improved finite sample properties. Journal of Econometrics, 29(3), 305–325. https://doi.org/10.1016/0304-4076(85)90158-7 (Read as Queen's Economics Department Working Paper No. 537, 1983.) record · 2 pages
- Miller, R. G. (1974). The jackknife: A review. Biometrika, 61(1), 1–15. https://doi.org/10.1093/biomet/61.1.1 record
- Quenouille, M. H. (1956). Notes on bias in estimation. Biometrika, 43(3–4), 353–360. https://doi.org/10.1093/biomet/43.3-4.353 record
- Shao, J., and Wu, C. F. J. (1989). A general theory for jackknife variance estimation. The Annals of Statistics, 17(3), 1176–1197. https://doi.org/10.1214/aos/1176347263 record
- Singh, K. (1981). On the asymptotic accuracy of Efron's bootstrap. The Annals of Statistics, 9(6), 1187–1195. https://doi.org/10.1214/aos/1176345636 record · 2 pages
- Wager, S., Hastie, T., and Efron, B. (2014). Confidence intervals for random forests: The jackknife and the infinitesimal jackknife. Journal of Machine Learning Research, 15, 1625–1651. arXiv:1311.4555. record
- White, H. (1980). A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica, 48(4), 817–838. https://doi.org/10.2307/1912934 record · 3 pages