Resampling Dependent and Clustered Data

Why resampling single observations fails when observations are dependent, and what to resample instead. The variance of the mean of a stationary series derived, with the long-run variance as its limit; the proof that the ordinary bootstrap estimates only the lag-0 part; the circular block bootstrap's variance of the mean derived exactly as a Bartlett-weighted sum of autocovariances, with its bias, its exact centering bias for uncorrelated data and the two limits of the block length (one observation per block, one block); the moving block, non-overlapping block and stationary bootstraps compared; for clustered data, the exact bootstrap variances of resampling observations, clusters and both stages, the cluster bootstrap's equality with the cluster-robust variance of a mean and its downward bias with few clusters. Worked on Iowa's weekly deaths, where a season and a trend make the answer depend on the block length and on what is treated as fixed, on a simulated autoregressive series and on three blood pressure readings per NHANES adult. Worked problems on a block bootstrap by hand, effective sample size, three clusters and the single block.

Deaths occurring in Iowa from all causes averaged 570.3 deaths a week over six years beginning in 2014. The bootstrap of The Bootstrap Method, applied to the 312 weekly counts, gives that average a standard error of 2.65 deaths. But a week of many deaths tends to be followed by another: the correlation between consecutive weeks is 0.66, because winters bring more deaths for weeks at a time. The ordinary bootstrap draws weeks independently and scatters each winter across the resampled series. Is 2.65 the right standard error, and if not, what should be resampled instead?

If the series is treated as stationary, it is not: resampling blocks of a month or longer instead of single weeks gives standard errors between about 1.6 and 2.8 times as large, depending on the block length. The series also has a seasonal cycle and a trend, and once those are treated as fixed the answer changes again; the section on the Iowa series measures both. The ordinary bootstrap rests on the observations being independent draws from one distribution, which is what The Plug-In Principle and Why the Bootstrap Works assumes when it replaces $F$ by the empirical distribution. When observations are dependent, in time or in groups, the resampling must keep the dependence intact. For a series that means resampling blocks of consecutive observations; for clustered data it means resampling whole clusters.

Here is the route. First we derive the variance of the mean of a dependent series and prove that the ordinary bootstrap misses every term but one. Then we derive the circular block bootstrap's variance exactly, show what it estimates and why both short and long blocks fail, and compare the other block schemes on the Iowa data. A simulated series with known variance measures the coverage the block bootstrap achieves. Iowa's series, which has a season and a trend, shows the limits of all of them. The second half turns to clusters: three blood pressure readings for each of several thousand NHANES adults, where resampling readings, adults or both gives three different answers, and only one is right. The article assumes The Plug-In Principle and Why the Bootstrap Works (its Theorem 4, the exact bootstrap variance of a mean) and the design effect of The Binomial Distribution.

The data and the notation

The weekly deaths. The National Center for Health Statistics publishes weekly counts of deaths by state, by the week in which they occurred and the state of occurrence (NCHS, 2020, updated 2021); the course pages loaded Iowa's counts for 2014 to 2019, 313 weeks, from all causes and from cancer. This article uses the first 312 weeks, exactly six blocks of 52, so that block lengths that divide 312 are available. MMWR year 2014 has 53 weeks, so the 312 weeks run from the first week of 2014 to the 51st week of 2019, and each 52-week block after the first starts a week before the matching MMWR year. Every number below comes from tools/experiments/statistics/bootstrap-for-dependent-data.py.

Notation for a series. The series is $x_1, \dots, x_n$ with mean $\bar x$. It is modeled as part of a stationary process: the joint distribution of $(X_{t+1}, \dots, X_{t+k})$ does not depend on $t$, so the mean $\mu$ and the autocovariances $\gamma(h) = \operatorname{Cov}(X_t, X_{t+h})$ are the same at every time, with $\gamma(-h) = \gamma(h)$. The autocorrelation at lag $h$ is $\gamma(h)/\gamma(0)$. Resampling notation follows the unit: given the data, $P_*$, $E_*$ and $\operatorname{Var}_*$ are probability, expectation and variance over the random resampling, and $\bar x^*$ is the mean of a resampled series.

For the all-cause series the standard deviation is 46.7 deaths, and the sample autocorrelations are 0.66 at lag 1, 0.06 at lag 13, $-0.52$ at lag 26 and 0.51 at lag 52: high in consecutive weeks, negative half a year apart, positive a year apart. That is the shape of an annual cycle. The cancer series, mean 122.7 deaths a week, has a lag-1 autocorrelation of 0.01 and no visible cycle.

Why resampling weeks fails

Here is the idea. The variance of a mean is the sum of all the covariances among the observations, divided by $n^2$. Independent resampling destroys every covariance except each observation's covariance with itself, so it estimates only that part of the sum.

Theorem 1 (the variance of the mean of a stationary series). For a stationary process with autocovariances $\gamma(h)$, $$n\operatorname{Var}(\bar X) = \sum_{\lvert h\rvert < n}\Big(1 - \frac{\lvert h\rvert}{n}\Big)\gamma(h).$$ If $\sum_h\lvert\gamma(h)\rvert < \infty$, then as $n \to \infty$, $$n\operatorname{Var}(\bar X) \to \sigma_\infty^2 = \sum_{h=-\infty}^{\infty}\gamma(h) = \gamma(0) + 2\sum_{h=1}^\infty\gamma(h),$$ the long-run variance of the process.

Proof. $\operatorname{Var}(\sum_t X_t) = \sum_{s=1}^n\sum_{t=1}^n\operatorname{Cov}(X_s, X_t) = \sum_{s,t}\gamma(t - s)$. The number of pairs $(s, t)$ in $\{1, \dots, n\}^2$ with $t - s = h$ is $n - \lvert h\rvert$, so the double sum is $\sum_{\lvert h\rvert < n}(n - \lvert h\rvert)\gamma(h)$; dividing by $n$ gives the identity. For the limit, each term $(1 - \lvert h\rvert/n)\gamma(h)\mathbf 1[\lvert h\rvert < n]$ tends to $\gamma(h)$ as $n \to \infty$ and is bounded by $\lvert\gamma(h)\rvert$, which is summable, so the sum converges to $\sum_h\gamma(h)$ by dominated convergence for series. $\square$

For independent observations $\gamma(h) = 0$ for $h \ne 0$, and the formula reduces to $\operatorname{Var}(\bar X) = \gamma(0)/n$. Positive autocorrelations make the mean more variable than that; negative ones make it less so.

What the ordinary bootstrap estimates. Resampling the $n$ observations independently with replacement gives, by Theorem 4 of The Plug-In Principle and Why the Bootstrap Works, $$\operatorname{Var}_*(\bar x^*) = \frac{\hat\gamma(0)}{n}, \qquad \hat\gamma(0) = \frac1n\sum_{t=1}^n(x_t - \bar x)^2,$$ the plug-in variance divided by $n$, whatever the order of the data. The bootstrap cannot see the order: shuffling the series leaves its answer unchanged. If $\hat\gamma(0)$ converges to $\gamma(0)$, which holds for the processes met here, then $$\frac{\operatorname{Var}_*(\bar x^*)}{\operatorname{Var}(\bar X)} \to \frac{\gamma(0)}{\sigma_\infty^2},$$ and the ordinary bootstrap is consistent only when the autocovariances at nonzero lags sum to zero.

Example (an autoregressive series). In the autoregressive process $X_t = \phi X_{t-1} + \varepsilon_t$ with $\lvert\phi\rvert < 1$ and independent errors of variance 1, $\gamma(h) = \phi^{\lvert h\rvert}/(1 - \phi^2)$ (the stationary variance $\gamma(0)$ solves $\gamma(0) = \phi^2\gamma(0) + 1$, and multiplying $X_{t+h} = \phi X_{t+h-1} + \varepsilon_{t+h}$ by $X_t$ and taking expectations gives $\gamma(h) = \phi\gamma(h - 1)$ for $h \ge 1$). Summing the geometric series, $\sigma_\infty^2 = \gamma(0)\big(1 + \frac{2\phi}{1 - \phi}\big) = \gamma(0)\frac{1 + \phi}{1 - \phi}$, so $$\frac{\gamma(0)}{\sigma_\infty^2} = \frac{1 - \phi}{1 + \phi}.$$ With $\phi = 0.6$ the ratio is $0.4/1.6 = 0.25$, so in large samples the ordinary bootstrap's standard error is half the right one. For $n = 312$ the exact standard deviation of the mean, from Theorem 1, is 0.1411, while $\sqrt{\gamma(0)/n} = 0.0708$, the value the ordinary bootstrap estimates. Worked problem 2 turns this into an effective sample size.

Resampling blocks

Here is the idea. Cut the series into blocks of $\ell$ consecutive observations, and build a resampled series by stringing together blocks drawn at random. Dependence within a block is kept; dependence between the end of one block and the start of the next is lost. If $\ell$ is long compared with the range of the dependence, little is lost.

Several schemes differ in which blocks are available.

  • The moving block bootstrap draws from all $n - \ell + 1$ overlapping blocks $(x_s, \dots, x_{s+\ell-1})$. Künsch (1989) constructed bootstrap replicates "by selecting blocks of length $l$ randomly with replacement among the blocks of observations", and defined the matching jackknife, which deletes each block of $\ell$ consecutive observations in turn; his abstract states consistency of that jackknife's variance estimate under conditions with $\ell \to \infty$ and $\ell/n \to 0$, and treats general statistics by approximating them with an arithmetic mean.
  • The non-overlapping block bootstrap draws from the $n/\ell$ disjoint blocks; its variance of the mean is closely related to Carlstein's (1986) estimator, which computes a statistic on adjacent non-overlapping subseries and takes the variance of the results.
  • The circular block bootstrap wraps the series into a circle, $x_{n+t} = x_t$, so that a block may start at any of the $n$ positions. Sharipov and Wendler (2013) describe it as introduced by Politis and Romano (1992). Every observation then appears in exactly $\ell$ of the $n$ blocks.
  • The stationary bootstrap of Politis and Romano (1994) resamples "blocks of random length, where the length of each block has a geometric distribution", on the circle; the resampled series is then itself stationary given the data.

The circular scheme has an exact variance of the mean, which explains the behavior of all four.

Theorem 2 (the circular block bootstrap variance of the mean). Let $n = k\ell$. Draw $k$ starting points independently and uniformly from $\{1, \dots, n\}$, take the $\ell$ consecutive observations on the circle from each, and let $\bar x^*$ be the mean of the $n$ resampled values. Then $E_*(\bar x^*) = \bar x$ and $$n\operatorname{Var}_*(\bar x^*) = \sum_{\lvert h\rvert < \ell}\Big(1 - \frac{\lvert h\rvert}{\ell}\Big)\hat\gamma_c(h), \qquad \hat\gamma_c(h) = \frac1n\sum_{s=1}^n(x_s - \bar x)(x_{s+h} - \bar x),$$ where the indices in $\hat\gamma_c$ are taken around the circle, and $\hat\gamma_c(-h) = \hat\gamma_c(h)$.

Proof. The resampled mean is the average of $k$ independent block means, $\bar x^* = \frac1k\sum_{j=1}^kB_j$, where $B_j = \frac1\ell\sum_{u=0}^{\ell-1}x_{S_j+u}$ and $S_j$ is uniform. Each observation lies in exactly $\ell$ of the $n$ circular blocks, so $E_*(B_j) = \frac1n\sum_s\frac1\ell\sum_ux_{s+u} = \frac1n\cdot\frac1\ell\cdot\ell\sum_tx_t = \bar x$. For the variance, $\operatorname{Var}_*(\bar x^*) = \frac1k\operatorname{Var}_*(B_1)$, and $$\operatorname{Var}_*(B_1) = \frac1n\sum_{s=1}^n\Big(\frac1\ell\sum_{u=0}^{\ell-1}(x_{s+u} - \bar x)\Big)^2 = \frac{1}{\ell^2}\sum_{u=0}^{\ell-1}\sum_{v=0}^{\ell-1}\frac1n\sum_s(x_{s+u} - \bar x)(x_{s+v} - \bar x) = \frac{1}{\ell^2}\sum_{u,v}\hat\gamma_c(v - u),$$ because on the circle $\frac1n\sum_s(x_{s+u} - \bar x)(x_{s+v} - \bar x)$ depends only on $v - u$. The number of pairs $(u, v)$ in $\{0, \dots, \ell - 1\}^2$ with $v - u = h$ is $\ell - \lvert h\rvert$, so $\operatorname{Var}_*(B_1) = \frac1\ell\sum_{\lvert h\rvert < \ell}(1 - \lvert h\rvert/\ell)\hat\gamma_c(h)$. Dividing by $k$ and multiplying by $n = k\ell$ gives the formula. $\square$

The formula is Theorem 1 with the sample autocovariances in place of the true ones, truncated at lag $\ell$ and with the triangular weights $1 - \lvert h\rvert/\ell$ of a block of length $\ell$ in place of $1 - \lvert h\rvert/n$. In the language of time series it is an estimate of the long-run variance with Bartlett's triangular weights. Künsch (1989) found the same structure for the block jackknife of the mean, which his abstract describes as "equivalent to a weighted covariance estimate of the spectral density of the observations at zero". With $\ell = 1$ it is $\hat\gamma(0)$, the ordinary bootstrap.

Two limits of the block length.

  • Blocks too short. For a fixed $\ell$, if the sample autocovariances converge to the true ones, $n\operatorname{Var}_*(\bar x^*)$ converges to $\sigma_\ell^2 = \sum_{\lvert h\rvert < \ell}(1 - \lvert h\rvert/\ell)\gamma(h)$, and $$\sigma_\infty^2 - \sigma_\ell^2 = \sum_{\lvert h\rvert < \ell}\frac{\lvert h\rvert}{\ell}\gamma(h) + \sum_{\lvert h\rvert \ge \ell}\gamma(h).$$ Both sums tend to zero as $\ell \to \infty$ (the first because $\sum_h\lvert h\rvert\gamma(h)/\ell \to 0$ by dominated convergence, each term being at most $\lvert\gamma(h)\rvert$; the second because it is the tail of a convergent series). For a process with positive autocovariances, such as the autoregressive one with $\phi > 0$, every term is positive, and the block bootstrap underestimates the variance for every finite $\ell$.
  • Blocks too long. With $\ell = n$ there is one block, and every resampled series is a rotation of the original, whose mean is $\bar x$; the bootstrap variance is 0 (Worked problem 4). Short of that extreme, long blocks bias the estimate downward in finite samples, because the autocovariances are measured about $\bar x$ rather than $\mu$. Proposition 1 below makes this exact. Roughly speaking, and as a heuristic rather than a result proved here, the estimate is also built from about $n/\ell$ nearly independent blocks, so its own variability grows as $\ell$ grows.

Proposition 1 (the expected bootstrap variance and its centering bias). Let $X_1, \dots, X_n$ have a common mean and finite variances, let $n = k\ell$, and write $c(h) = \frac1n\sum_{s=1}^n\operatorname{Cov}(X_s, X_{s+h})$ with indices taken around the circle. Then $$E\big[n\operatorname{Var}_*(\bar x^*)\big] = \sum_{\lvert h\rvert < \ell}\Big(1 - \frac{\lvert h\rvert}{\ell}\Big)c(h) \;-\; \ell\operatorname{Var}(\bar X).$$ In particular, if the $X_s$ are uncorrelated with common mean and common variance $\sigma^2$, then $E[n\operatorname{Var}_*(\bar x^*)] = \sigma^2(1 - \ell/n)$.

Proof. With a common mean, $E[(X_s - \bar X)(X_t - \bar X)] = \operatorname{Cov}(X_s, X_t) - \operatorname{Cov}(X_s, \bar X) - \operatorname{Cov}(X_t, \bar X) + \operatorname{Var}(\bar X)$. Averaging over $s$ with $t = s + h$ on the circle, $\frac1n\sum_s\operatorname{Cov}(X_s, \bar X) = \operatorname{Cov}(\bar X, \bar X) = \operatorname{Var}(\bar X)$, and the same holds for $\frac1n\sum_s\operatorname{Cov}(X_{s+h}, \bar X)$, since $s + h$ also runs over every index once. So $E\hat\gamma_c(h) = c(h) - \operatorname{Var}(\bar X)$. The weights $1 - \lvert h\rvert/\ell$ of Theorem 2 sum over $\lvert h\rvert < \ell$ to $1 + 2\sum_{h=1}^{\ell-1}(1 - h/\ell) = 1 + (\ell - 1) = \ell$, so taking expectations in Theorem 2 gives the formula. For uncorrelated data $c(0) = \sigma^2$, $c(h) = 0$ for $0 < \lvert h\rvert < \ell \le n$ (the two indices differ around the circle) and $\operatorname{Var}(\bar X) = \sigma^2/n$, which gives $\sigma^2 - \ell\sigma^2/n$. $\square$

The term $-\ell\operatorname{Var}(\bar X) = -\frac{\ell}{n}\cdot n\operatorname{Var}(\bar X)$ is always downward and grows in proportion to $\ell$: it is the price of measuring the autocovariances about $\bar x$. Without a common mean the formula fails, since $E[(X_s - \bar X)(X_t - \bar X)]$ then gains the term $(\mu_s - \bar\mu)(\mu_t - \bar\mu)$; a trend is the example.

At $\ell = 1$ this is the familiar bias of the plug-in variance, and at $\ell = n$ it is 0, as Worked problem 4 finds for every data set.

The choice of $\ell$ trades the two errors. Hall, Horowitz and Jing (1995) showed that the optimal block size depends on the purpose, being of order $n^{1/3}$ for estimating a variance or bias and smaller powers of $n$ for distribution functions. They also gave an empirical rule for choosing it. Politis and White (2004) proposed estimators of the optimal block size based on spectral estimation with flat-top lag windows, adapting to the strength of the correlation in the correlogram.

The Iowa series

For Iowa's weekly deaths the exact formula of Theorem 2 gives the circular block bootstrap's standard error of the mean for any block length that divides 312. The ordinary bootstrap's value is 2.65 deaths for all causes and 0.61 for cancer.

Block length $\ell$ (weeks) Blocks SE, all causes Ratio to $\ell = 1$ SE, cancer
4 78 4.55 1.72 0.64
13 24 7.11 2.69 0.59
26 12 7.46 2.82 0.49
52 6 4.33 1.64 0.40
104 3 4.42 1.67 0.38

For the cancer series, which shows no autocorrelation, short blocks agree with the ordinary bootstrap, as they should: 0.64 at $\ell = 4$ against 0.61. Long blocks give smaller values, 0.40 at a year and 0.38 at two years. That is the second limit at work. Part of it is the exact centering bias of Proposition 1: for uncorrelated data with the ordinary bootstrap's 0.61 at $\ell = 1$, the identity would predict expected values of 0.56 at $\ell = 52$ and 0.50 at $\ell = 104$. The observed values are lower still. The rest is the chance variation of an estimate built from six or three blocks, which the script does not measure separately.

For all causes the standard error rises from 2.65 to about 7 as the blocks grow to a quarter or half a year, as the strong positive autocorrelation at short lags requires. Then it falls to 4.33 at a full year. A block of 52 weeks contains one whole seasonal cycle wherever it starts, so its mean averages the winter peak and the summer trough. The variation between block means then reflects only differences between years. The negative autocorrelation at lag 26 enters the Bartlett sum with large weight at $\ell = 52$ and cancels much of the positive autocorrelation at short lags.

Which number is the standard error? Theorem 1 assumes a stationary process, and two features of this series sit badly with that. One is the annual cycle. The other is a rise over the six years: the means of the six consecutive 52-week blocks (each within a week of a calendar year) are 553.7, 563.0, 558.5, 580.2, 580.0 and 586.6. To see which feature drives which part of the table, the script removes by least squares a linear trend, an annual cycle (three harmonics with period $365.25/7 = 52.18$ weeks), or both, and recomputes the exact circular block bootstrap standard errors from the residuals. The choice of model is ours, and another model would give somewhat different numbers.

Series $\ell = 1$ 4 13 26 52 104
raw 2.65 4.55 7.11 7.46 4.33 4.42
trend removed 2.60 4.43 6.86 6.99 2.70 1.79
season removed 1.79 2.43 3.36 4.07 4.32 4.43
both removed 1.68 2.07 2.54 2.89 2.72 1.73
both removed, allowing for the fit 1.70 2.17 2.82 3.29 3.22 2.42

Removing the cycle cuts the lag-1 autocorrelation from 0.66 to 0.28 and removes most of the rise at short and medium block lengths. Removing the trend changes little at short blocks but lowers the values at 52 and 104 weeks. In this measurement the cycle drives the rise and the trend drives the long-block values. With both removed, the remainder has a standard deviation of 29.6 deaths and a lag-1 autocorrelation of 0.18, and its block standard errors run from 1.7 to 2.9.

Both ends of that range are biased downward. At $\ell = 1$ the remaining positive correlation is ignored, as Theorem 1 shows. At long blocks the fitted trend and cycle absorb part of the variation, beyond the centering bias of Proposition 1. The script measures the second effect exactly: for white noise passed through the same least-squares fit (residual covariance $I - X X^{+}$, with $X$ the fitted columns), the expected value of $n\operatorname{Var}_*$ is 0.71 of the true $n\operatorname{Var}(\bar X)$ at $\ell = 52$ and 0.51 at $\ell = 104$, against 0.83 and 0.67 from Proposition 1 with no fit. The last row divides each standard error by the square root of that white-noise factor. The remainder is not white noise, so the correction is approximate. Allowing for the fit in this way, the remainder's standard error at blocks of 13 to 52 weeks is about 2.8 to 3.3 deaths, close to and slightly above the ordinary bootstrap's 2.65 for the raw series.

These are standard errors of different quantities. The block bootstrap of the raw series answers the question "how variable is the mean of a stationary series with these autocovariances?" Under that model the ordinary bootstrap's 2.65 is too small. If instead the cycle and the trend are treated as fixed (over six 52-week blocks the cycle almost exactly averages out of the mean), the uncertainty left is that of the weakly correlated remainder, which the last row, allowing for the bias of the fit, puts at about 3 deaths. Treating the six yearly block means as independent gives a standard error of 5.56, from five degrees of freedom, with the trend counted as error. Choosing among these is a modeling decision, and Stochastic Processes and Time Series is the place for such models. What the data show is that the block bootstrap's answer depends both on the block length and on what is treated as fixed.

The four block schemes compared. The script also ran each scheme by simulation, 2,000 resamples for each of 30 seeds (medians shown), with $\ell$ the block length or, for the stationary bootstrap, the mean block length:

$\ell$ Circular, exact Circular, simulated Moving Non-overlapping Stationary
13 7.11 7.06 7.10 7.25 6.56
52 4.33 4.34 4.31 5.12 5.28

The simulated circular bootstrap reproduces the exact formula, and the moving block bootstrap is close to it. The non-overlapping scheme at $\ell = 52$ uses only the six fixed 52-week blocks, and the stationary bootstrap's random block lengths do not respect the 52-week period, so both differ from the circular value there. On a series without a strong cycle the schemes agree more closely; here the cycle magnifies their differences.

How well the block bootstrap does when the answer is known

On the Iowa series the true standard error is unknown. For the autoregressive process with $\phi = 0.6$ and $n = 312$ it is known exactly (0.1411), so the coverage of the interval $\bar x \pm 1.96\sqrt{\operatorname{Var}_*(\bar x^*)}$ can be measured. The script simulates 2,000 series for each of 30 seeds, computes the circular block bootstrap variance of each by the exact formula, and records how often the interval contains the true mean 0.

$\ell$ Median SE Coverage IQR of coverage
1 0.070 0.663 0.659 to 0.672
2 0.088 0.775 0.771 to 0.782
4 0.107 0.858 0.852 to 0.864
8 0.121 0.899 0.896 to 0.902
13 0.126 0.910 0.906 to 0.913
26 0.126 0.909 0.904 to 0.912
52 0.120 0.882 0.876 to 0.889

The ordinary bootstrap ($\ell = 1$) covers 66% of the time, as a standard error half the right size predicts: $2\Phi(1.96 \times 0.0708/0.1411) - 1 = 0.67$. Coverage rises with $\ell$, peaks near 91% for blocks of 13 to 26, and falls again at 52. It never reaches 95%. The script computes the exact expectation of the bootstrap variance for this process (the trace of a quadratic form in the series' covariance matrix). Its square root is 0.127 at $\ell = 13$, 0.130 at $\ell = 26$ and 0.126 at $\ell = 52$, all below the true 0.1411, while the long-run variance is 6.25. Proposition 1 splits the expectation into two parts. The first, the Bartlett-weighted sum of the circular covariances, is close to $\sigma_\ell^2$, which is still below $\sigma_\infty^2$ (the bias term of the previous section, all of whose terms are positive here). The second, the centering term $\ell\operatorname{Var}(\bar X)$, is subtracted: 0.259 at $\ell = 13$, 0.518 at $\ell = 26$ and 1.035 at $\ell = 52$. The script confirms that the two parts reproduce the exact expectations. The centering term is why the fall from 26 to 52 is in the expected value itself, not only in the noise. The estimates from six blocks are also more variable, which the medians in the table do not show. These are results for this process and this length. The limit argument of the previous section says what a longer series would allow: with $\ell$ growing and $\ell/n$ shrinking, both the bias and the noise of the estimate shrink, so coverage can approach 95%, but not at $n = 312$ with $\phi = 0.6$.

A different route: the sieve bootstrap. When the series is well described by an autoregression, one can fit an autoregressive model, resample its residuals independently and rebuild series from the fitted model. Bühlmann (1997) studied this sieve bootstrap, letting the order of the autoregression grow with $n$, and proved its consistency for a class of estimators. He also found that its variance of the mean can converge faster than the block bootstrap's when the dependence decays quickly. It trades the block bootstrap's freedom from a model for the efficiency of one.

Clustered data

The second kind of dependence is grouping. NHANES 2017–2018 took up to three systolic blood pressure readings from each examined adult; 4,730 adults have all three. Readings from the same adult are far more alike than readings from different adults. The Statistics of Model Evaluation unit treats the same structure for benchmark questions that share a subject, and its symbols $G$, $g$ and $\rho$ are used here, with $x$ for a reading. There are $G$ clusters (here adults) $g = 1, \dots, G$, each with $m = 3$ observations $x_{g1}, x_{g2}, x_{g3}$, cluster means $\bar c_g$, overall mean $\bar x$, and intraclass correlation $\rho$, the correlation between two readings of the same adult. As a finite population the 4,730 adults have mean reading 125.367 mm Hg, an intraclass correlation of 0.965 (the one-way analysis-of-variance estimate) and a design effect $1 + (m - 1)\rho = 2.93$. The mean of three readings from one adult varies 2.93 times as much as the mean of three readings from three different adults would. The Binomial Distribution derives the design effect from the law of total variance, and Clustered Standard Errors derives the cluster-robust variance in general.

A sample of 200 adults (600 readings; seed 7) has mean reading 124.52 mm Hg. Three resampling schemes are available.

Theorem 3 (three bootstraps of a clustered mean). For $G$ clusters of equal size $m$, let $\hat\sigma^2 = \frac{1}{Gm}\sum_{g,j}(x_{gj} - \bar x)^2$ and $\hat\sigma_g^2 = \frac1m\sum_j(x_{gj} - \bar c_g)^2$. The bootstrap variance of the overall mean is:

  1. resampling the $Gm$ observations independently: $\operatorname{Var}_*(\bar x^*) = \hat\sigma^2/(Gm)$;
  2. resampling $G$ clusters with replacement and keeping each drawn cluster whole (the cluster bootstrap): $\operatorname{Var}_*(\bar x^*) = \frac{1}{G^2}\sum_g(\bar c_g - \bar x)^2$;
  3. resampling $G$ clusters and then $m$ observations with replacement within each drawn cluster (the two-stage bootstrap): $\operatorname{Var}_*(\bar x^*) = \frac{1}{G^2}\sum_g(\bar c_g - \bar x)^2 + \frac{1}{G^2m}\sum_g\hat\sigma_g^2$.

Proof. (1) is the plug-in variance of a mean of $Gm$ independent draws from the empirical distribution of the observations (Theorem 4 of The Plug-In Principle and Why the Bootstrap Works). (2) With equal sizes the overall mean is the mean of the cluster means, so $\bar x^*$ is the mean of $G$ independent draws from the empirical distribution of $\bar c_1, \dots, \bar c_G$, whose mean is $\bar x$; the same theorem gives the variance. (3) Let $I$ be the index of one drawn cluster and $Y$ the mean of $m$ observations drawn with replacement from it. Given $I = g$, $Y$ has mean $\bar c_g$ and variance $\hat\sigma_g^2/m$. By the law of total variance (Conditional Expectation and the Laws of Total Expectation and Variance), $\operatorname{Var}_*(Y) = \operatorname{Var}_*(\bar c_I) + E_*(\hat\sigma_I^2/m) = \frac1G\sum_g(\bar c_g - \bar x)^2 + \frac{1}{Gm}\sum_g\hat\sigma_g^2$. The $G$ drawn clusters are independent, so dividing by $G$ gives (3). $\square$

For the 200 adults the three standard errors are 0.801 for resampling readings, 1.370 for the cluster bootstrap and 1.376 for the two-stage bootstrap, and simulations with 4,000 resamples reproduce all three. The ratio of the cluster bootstrap's variance to the reading bootstrap's is 2.93, the design effect. Resampling readings treats 600 readings as 600 independent pieces of information when they carry about as much as 205 would.

The cluster bootstrap and the cluster-robust variance. The cluster-robust (sandwich) variance of a mean adds, over clusters, the squared sum of each cluster's deviations: $V_{\mathrm{CR}} = \frac{1}{(Gm)^2}\sum_g\big(\sum_j(x_{gj} - \bar x)\big)^2$, the form Clustered Standard Errors derives without a small-sample factor. Since $\sum_j(x_{gj} - \bar x) = m(\bar c_g - \bar x)$, this is $\frac{1}{G^2}\sum_g(\bar c_g - \bar x)^2$, exactly the cluster bootstrap's variance in Theorem 3. For a mean of equal-sized clusters, the cluster bootstrap and the cluster-robust formula are the same estimator.

Why the two-stage bootstrap overcounts. The variation of the cluster means already includes the variation of readings within clusters: each $\bar c_g$ is a mean of $m$ noisy readings. Resampling within clusters adds that within-cluster variation a second time, the term $\frac{1}{G^2m}\sum_g\hat\sigma_g^2$ of Theorem 3. For blood pressure readings the within-adult variation is small beside the between-adult variation (intraclass correlation 0.965), so the double count adds little, 1.376 against 1.370. For data with a small intraclass correlation it can add almost as much again. Field and Welsh (2007) compare these and other bootstraps for one-way clustered data and report that the cluster bootstrap gives consistent variance estimates under both of the models they consider. That supports the advice to resample clusters and leave them whole.

How many clusters? The cluster bootstrap estimates a variance from $G$ cluster means, and for $G$ independent clusters its expectation is $\frac{G - 1}{G}\operatorname{Var}(\bar x)$, by the same calculation as the bias of a plug-in variance; Clustered Standard Errors proves (its Theorem 3) that for equal clusters the factor $G/(G - 1)$ makes the cluster-robust variance unbiased. The script measures coverage by drawing samples of adults with replacement from the 4,730, 2,000 samples for each of 30 seeds, and checking whether $\bar x \pm 1.96\,\mathrm{SE}$ contains the population mean. The "cluster, $t$" column uses the cluster bootstrap variance multiplied by $G/(G - 1)$ and the $t$ critical value with $G - 1$ degrees of freedom.

Adults $G$ Readings bootstrap Cluster bootstrap Cluster, $t$ Two-stage
200 0.746 0.948 0.949 0.949
10 0.687 0.890 0.935 0.892

With 200 adults (true standard deviation of the mean 1.356) the cluster and two-stage bootstraps cover at the nominal rate, while resampling readings covers three times in four. With 10 adults (true standard deviation 6.155) the cluster bootstrap falls to 89%: the factor $(G - 1)/G = 0.9$ and the use of 1.96 in place of $t_{9,\,0.975} = 2.262$ both make its intervals too short. The corrected version restores most of the coverage. Cameron, Gelbach and Miller (2008) document the same problem for regression with clustered errors: with few clusters, five to thirty in their study, standard cluster-robust tests over-reject. They propose bootstrap-$t$ procedures, including a wild cluster bootstrap that keeps the regressors fixed and multiplies each cluster's residual vector by $+1$ or $-1$ at random. They also note that resampling whole clusters can produce samples in which a regressor that is constant within clusters takes only one value, so that the model cannot be fitted.

Permutation tests with clusters. The same principle governs Permutation Tests. When treatments are assigned to whole clusters, such as clinics or schools, the transformations under the null hypothesis relabel clusters, not individuals. Relabeling individuals would treat correlated observations as exchangeable. When observations within a cluster are positively correlated, the variance of a difference in means is larger than independent relabeling allows, by the design effect, so the p-values can be too small.

What is proved here, what is cited, what the experiment shows

Proved here. The variance of the mean of a stationary series and its long-run limit (Theorem 1), and the inconsistency of the ordinary bootstrap for it, given Theorem 4 of The Plug-In Principle and Why the Bootstrap Works and the convergence of $\hat\gamma(0)$. The autocovariances and the ratio $(1 - \phi)/(1 + \phi)$ of the autoregressive process. The exact circular block bootstrap variance (Theorem 2), the bias of its limit for a fixed block length, the exact expectation of the bootstrap variance with its centering term, and its uncorrelated case (Proposition 1), and its vanishing at $\ell = n$. The three bootstrap variances of a clustered mean (Theorem 3), the equality of the cluster bootstrap and the cluster-robust variance of a mean, and the double count of the two-stage scheme.

Cited. The moving block bootstrap, and the consistency and spectral form of the block jackknife (Künsch, 1989); Carlstein's (1986) subseries estimator; the circular block bootstrap's origin (Politis and Romano, 1992, as described by Sharipov and Wendler, 2013) and the stationary bootstrap (Politis and Romano, 1994); block-size rates (Hall, Horowitz and Jing, 1995) and their estimation (Politis and White, 2004); the sieve bootstrap (Bühlmann, 1997); the comparison of cluster bootstraps (Field and Welsh, 2007) and the few-clusters results and wild cluster bootstrap (Cameron, Gelbach and Miller, 2008). Each is reported at the level of its abstract or of the passages read, as the record of sources for this article lists.

What the experiments show. For Iowa's weekly deaths: the dependence of every block bootstrap on the block length, and, with a model of our choosing (a linear trend and three annual harmonics), that the cycle drives the rise at short blocks and the trend the values at long blocks, and that the remainder's standard errors, after an approximate allowance for the bias of the fit (exact for white noise through the same fit), are about 2.8 to 3.3 at blocks of 13 to 52 weeks, near the raw series' 2.65. For the autoregressive series, the exact expectation of the bootstrap variance at each block length and its split by Proposition 1. For the simulated autoregressive series: coverage between 66% and 91% depending on the block length, never 95%, at $n = 312$. For the NHANES readings: the design effect recovered by the cluster bootstrap, nominal coverage with 200 adults, and 89% with 10. These are observations of these data, processes and sizes.

Worked problems

The four problems are posed here, on small exact cases and on the article's processes; the article's script checks each number.

Problem 1 (a block bootstrap by hand). For the series 3, 5, 4, 8, 6, 4 ($n = 6$), find the circular block bootstrap variance of the mean with blocks of length 2, from the block means and from Theorem 2, and compare it with the ordinary bootstrap's.

Solution. The variance is 0.4722, against 0.4444 for the ordinary bootstrap. The mean is 5. On the circle the six blocks of two are $(3, 5)$, $(5, 4)$, $(4, 8)$, $(8, 6)$, $(6, 4)$ and $(4, 3)$, with means 4, 4.5, 6, 7, 5 and 3.5. Their average is 5, and their plug-in variance is $\frac{1 + 0.25 + 1 + 4 + 0 + 2.25}{6} = \frac{8.5}{6} = 1.4167$. A resampled series consists of $k = 3$ independent blocks, so $\operatorname{Var}_*(\bar x^*) = 1.4167/3 = 0.4722$. By Theorem 2: the circular autocovariances are $\hat\gamma_c(0) = 16/6 = 2.6667$ and $\hat\gamma_c(1) = \frac{(-2)(0) + (0)(-1) + (-1)(3) + (3)(1) + (1)(-1) + (-1)(-2)}{6} = \frac{1}{6} = 0.1667$, so $n\operatorname{Var}_* = \hat\gamma_c(0) + 2 \cdot \tfrac12\hat\gamma_c(1) = 2.8333$ and $\operatorname{Var}_* = 2.8333/6 = 0.4722$. The ordinary bootstrap gives $\hat\gamma_c(0)/6 = 0.4444$. The script confirms the value by listing all $6^3 = 216$ equally likely choices of three block starts.

Problem 2 (effective sample size). For the autoregressive process with $\phi = 0.6$, how many independent observations with the same variance would give a mean as precise as $n = 312$ consecutive ones, in large samples?

Solution. About 78. In large samples $\operatorname{Var}(\bar X) \approx \sigma_\infty^2/n$, and $n'$ independent observations with variance $\gamma(0)$ give $\gamma(0)/n'$. Setting the two equal, $n' = n\,\gamma(0)/\sigma_\infty^2 = n(1 - \phi)/(1 + \phi) = 312 \times 0.25 = 78$. The 312 dependent observations carry the information of 78 independent ones, and resampling them as if they were independent overstates the information fourfold.

Problem 3 (three clusters). Three clusters of two observations are $(1, 3)$, $(4, 4)$ and $(7, 9)$. Compute the three bootstrap variances of Theorem 3.

Solution. Resampling observations gives 1.1481, the cluster bootstrap 2.0741 and the two-stage bootstrap 2.1852. The overall mean is $28/6 = 4.6667$. The six deviations are $-3.6667, -1.6667, -0.6667, -0.6667, 2.3333, 4.3333$, with squares summing to 41.3333, so $\hat\sigma^2 = 41.3333/6 = 6.8889$ and scheme 1 gives $6.8889/6 = 1.1481$. The cluster means are 2, 4 and 8, with squared deviations from 4.6667 summing to $7.1111 + 0.4444 + 11.1111 = 18.6667$; scheme 2 gives $18.6667/9 = 2.0741$. The within-cluster plug-in variances are 1, 0 and 1, so scheme 3 adds $\frac{2}{9 \times 2} = 0.1111$, giving 2.1852. The clusters differ far more than the observations within them, so treating the six as independent understates the variance by almost half.

Problem 4 (one block). Show that the circular block bootstrap with $\ell = n$ has variance 0, and that the circular autocovariances of any series sum to zero over $h = 0, \dots, n - 1$.

Solution. With one block of length $n$ starting at $s$, the resampled series is $x_s, x_{s+1}, \dots, x_{s+n-1}$ around the circle, a rotation of the original, and contains every observation exactly once. Its mean is $\bar x$ for every $s$, so the variance is 0. For the second part, $\sum_{h=0}^{n-1}\hat\gamma_c(h) = \frac1n\sum_s(x_s - \bar x)\sum_h(x_{s+h} - \bar x) = \frac1n\sum_s(x_s - \bar x)\cdot 0 = 0$, because for each $s$ the inner sum runs over the whole circle. This is why long blocks pull the estimate toward zero: the Bartlett sum of Theorem 2 with $\ell$ close to $n$ approaches a sum whose total is zero.

Summary

  • The variance of the mean of a stationary series is $\frac1n\sum_{\lvert h\rvert<n}(1 - \lvert h\rvert/n)\gamma(h)$, which tends to the long-run variance over $n$. The ordinary bootstrap estimates only $\gamma(0)/n$; for an autoregression with $\phi = 0.6$ that is a quarter of the right variance.
  • The circular block bootstrap's variance of the mean is exactly a Bartlett-weighted sum of the sample autocovariances up to lag $\ell - 1$. Short blocks miss the long-lag covariances; long blocks leave too few blocks, and a single block gives zero.
  • On a simulated autoregression of length 312, the best block lengths reached 91% coverage, not 95%. On Iowa's weekly deaths, treated as stationary, the answer depended on the block length, from 3.4 to 7.5 deaths for blocks of 2 to 104 weeks, against 2.65 for the ordinary bootstrap. A seasonal cycle drives the rise and a trend the long-block values; with both removed, and allowing for the bias of the fit, the remainder's standard error is about 3 deaths.
  • For clustered data, resample clusters whole. For a mean of equal clusters the cluster bootstrap equals the cluster-robust variance; resampling observations misses the design effect (2.93 for three blood pressure readings per adult), and resampling within clusters as well counts the within-cluster variation twice.
  • With few clusters the cluster bootstrap is biased downward by $(G - 1)/G$ and needs a $t$ reference; with 10 adults it covered 89%, and 93.5% after the correction.

What we have learned

Every resampling method in this unit replaces the unknown distribution with an estimate built from units that are treated as independent. When the units of the data are not independent, the resampling unit must be the independent piece: a block of a series long enough to carry its dependence, or a whole cluster. The block bootstrap's answer then depends on a block length that trades bias against noise, and it inherits the stationarity it assumes; the cluster bootstrap needs enough clusters. The Resampling Methods unit began with The Plug-In Principle and Why the Bootstrap Works, which shows why the bootstrap works for independent data. The processes behind dependent data, and models for seasons and trends, are the subject of Stochastic Processes and Time Series. The clustered structure returns in Clustered Standard Errors, where benchmark questions share a subject, and in Paired Comparisons of Models, where two models answer the same questions.

References