Survey Sampling Designs

The variances of the three building blocks of survey design, derived from the design alone with no model for the data. Simple random sampling without replacement (the sample mean unbiased, its variance with the finite population correction, the sample variance unbiased for the population variance); stratified sampling (its variance, the exact comparison with simple random sampling under proportional allocation, and Neyman's optimal allocation proved by the Cauchy–Schwarz inequality, with and without costs); cluster and two-stage sampling (their variances, the intraclass correlation and the design effect 1 + (m − 1)ρ derived, the ratio estimator for clusters of unequal size, and the exact bias of the with-replacement variance estimator that survey software uses). Every formula is checked by Monte Carlo on the 5,182 NHANES adults treated as a finite population, and the design of NHANES itself, with its strata, its 30 locations and its masked variance units, is explained from these results. Worked problems: every sample of a six-person population, a stratification that loses to simple random sampling, an allocation with unequal costs and the effective size of a clustered sample.

The National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018 did not draw its participants by lottery from a list of Americans. It divided the country's counties into strata, chose 30 locations, chose city blocks within each location, households within the blocks and people within the households, and sent mobile examination centers to each location for weeks. Weighted, the data of the 5,182 adults whose blood pressure it measured estimate the mean systolic pressure of US adults at 123.47 mm Hg, with a standard error of 0.463 under the survey's design (our computation from the public files); a simple random sample of the same size and spread would have given about 0.250, so the design costs a factor of 3.45 in variance. Why does the design cost so much precision, and why does the survey accept the cost? Variance is a property of the design, and the derivations below show how. Kish's approximation puts the cost of unequal weights at about 2.33 (Survey Weights and the Horvitz–Thompson Estimator), but it assumes weights unrelated to the outcome. Clustering is the other main source: in an equal-probability design taking some 170 adults from each location, a within-location correlation of about one percent would multiply the variance by nearly 3 (2.74 for systolic pressure). Stratification works the other way. The factors need not multiply: their product would be $2.33 \times 2.74 = 6.4$. Without the weights, the same 15 strata and 30 locations give a design effect of 2.86 for systolic pressure, close to the clustering factor, and the weights raise it only by a further factor of 1.20, to 3.45: far less than Kish's 2.33, so his assumption of weights unrelated to the outcome does not describe these data. With 30 locations in 15 strata every one of these figures is rough: the measured 3.45 rests on 15 degrees of freedom. The survey accepts the cost of clustering because examination centers on trucks can visit only so many places.

A sampling design is the rule that gives every possible sample of a finite population its probability of being chosen. Design-based inference takes the population's values as fixed numbers and finds all its randomness in that rule: an estimator is unbiased if its average over all possible samples, weighted by their probabilities, is the population quantity, and its variance is its spread over the same samples. No model for the values is assumed. The NHANES adults' pressures need not be normal, independent or identically distributed; they need only be the numbers they are. This article derives the variances of the three building blocks of practical designs (simple random, stratified and cluster sampling) in that sense, and then reads NHANES's design as a combination of them.

First we prove the properties of simple random sampling from inclusion indicators, a route that generalizes to every design. Then we derive the variance of stratified sampling, compare it exactly with simple random sampling and prove Neyman's optimal allocation. Next come cluster and two-stage sampling, where the intraclass correlation and the design effect $1 + (m-1)\rho$ appear, and the variance estimator that survey software applies to them. Finally we explain why NHANES is a stratified multistage sample and what its masked variance units are. The article assumes Expectation, Variance and Covariance for the variance of a sum and the law of total variance from Conditional Expectation; the weights that undo unequal selection probabilities are the subject of the companion article, Survey Weights and the Horvitz–Thompson Estimator.

The population and the notation

The running population. The 5,182 adults of NHANES 2017 to 2018 who are 18 or older, not pregnant, and have a systolic and a diastolic reading are treated in this article as a finite population of their own, as in Data Collection and Sampling, which compares four designs on the same adults by simulation (National Center for Health Statistics, 2020). Their survey weights are not used here: the adults are the population, and the samples are drawn from them by the designs under study. Two variables are measured on each adult: $y_k$, the mean of the available systolic readings in mm Hg, and the hypertension indicator (1 for mean systolic pressure of 130 or more, mean diastolic of 80 or more, or taking medicine for high blood pressure; 0 otherwise). The population's mean systolic pressure is 126.48 mm Hg, its standard deviation $S = 20.07$, and its hypertension prevalence 0.5475.

Notation. The population is $U = \{1, \ldots, N\}$, its units are $k$ and $l$, and $y_k$ is the value of unit $k$. The population total is $t = \sum_{k \in U} y_k$, the mean $\bar Y = t/N$, and the population variance is written with divisor $N - 1$,

$$S^2 = \frac{1}{N-1}\sum_{k \in U}(y_k - \bar Y)^2,$$

the convention of sampling texts such as Cochran (1977), as The Sampling Distribution of the Mean notes, and the one that makes the formulas below simplest. (Course pages that use $\sigma^2$ with divisor $N$ have $S^2 = N\sigma^2/(N-1)$.) A sample $s$ is a subset of $U$ of size $n$, and the sampling fraction is $f = n/N$. The inclusion indicator $I_k$ is 1 if unit $k$ is in the sample and 0 otherwise; it is the only random quantity in design-based inference. Its expectation $\pi_k = P(k \in s)$ is the inclusion probability of unit $k$, and $\pi_{kl} = P(k \in s \text{ and } l \in s)$ is the joint inclusion probability of two units.

Simple random sampling

A simple random sample without replacement gives every subset of $n$ units the same probability, $1/\binom{N}{n}$. The course article The Sampling Distribution of the Mean proves that the sample mean is centered on the population mean and has variance $\frac{\sigma^2}{n}\cdot\frac{N-n}{N-1}$, by putting the population in a random order. Here we prove the same facts from inclusion indicators, because that route works for every design that follows. The idea in one sentence: the sample mean is a fixed linear combination of the indicators, so its mean and variance follow from the first two moments of the indicators.

Lemma 1 (indicators of a simple random sample). Under simple random sampling of $n$ from $N$, for every unit $k$ and every pair $k \ne l$,

$$\pi_k = \frac{n}{N} = f, \qquad \pi_{kl} = \frac{n(n-1)}{N(N-1)}, \qquad \operatorname{Var}(I_k) = f(1-f), \qquad \operatorname{Cov}(I_k, I_l) = -\frac{f(1-f)}{N-1}.$$

Proof. The samples containing $k$ are formed by choosing the other $n - 1$ members from the remaining $N - 1$ units, so there are $\binom{N-1}{n-1}$ of them, and $\pi_k = \binom{N-1}{n-1}/\binom{N}{n} = n/N$. The samples containing both $k$ and $l$ number $\binom{N-2}{n-2}$, so $\pi_{kl} = \binom{N-2}{n-2}/\binom{N}{n} = n(n-1)/[N(N-1)]$. An indicator has variance $\pi_k(1 - \pi_k) = f(1-f)$. The covariance is $E[I_kI_l] - \pi_k\pi_l = \pi_{kl} - f^2$, and

$$\frac{n(n-1)}{N(N-1)} - \frac{n^2}{N^2} = \frac{n}{N}\cdot\frac{N(n-1) - n(N-1)}{N(N-1)} = \frac{n}{N}\cdot\frac{n - N}{N(N-1)} = -\frac{f(1-f)}{N-1}. \qquad \blacksquare$$

Theorem 1 (the mean of a simple random sample). The sample mean $\bar y = \frac1n\sum_{k \in s} y_k$ satisfies

$$E[\bar y] = \bar Y, \qquad \operatorname{Var}(\bar y) = (1 - f)\,\frac{S^2}{n}.$$

Proof. Write $\bar y = \frac1n\sum_{k \in U} I_k y_k$, a sum over the whole population in which the $y_k$ are constants. Its expectation is $\frac1n\sum_k f y_k = \frac{n}{nN}\,t = \bar Y$. For the variance, the variance of a sum is the sum of all variances and covariances (proved in Expectation, Variance and Covariance):

$$\operatorname{Var}(\bar y) = \frac{1}{n^2}\Big[\sum_k y_k^2\operatorname{Var}(I_k) + \sum_{k \ne l} y_k y_l \operatorname{Cov}(I_k, I_l)\Big] = \frac{f(1-f)}{n^2}\Big[\sum_k y_k^2 - \frac{1}{N-1}\sum_{k \ne l} y_k y_l\Big],$$

by Lemma 1. The sum over pairs of different units is the square of the total minus the diagonal, $\sum_{k \ne l} y_ky_l = t^2 - \sum_k y_k^2$. The bracket is therefore

$$\sum_k y_k^2 - \frac{t^2 - \sum_k y_k^2}{N-1} = \frac{N}{N-1}\Big(\sum_k y_k^2 - \frac{t^2}{N}\Big) = \frac{N}{N-1}\,(N-1)S^2 = N S^2.$$

So $\operatorname{Var}(\bar y) = f(1-f)NS^2/n^2 = (1-f)S^2/n$, since $fN = n$. $\blacksquare$

The factor $1 - f$ is the finite population correction: it removes the share of the population already seen. Its square root multiplies the standard error, and in the course page's form, with $S^2 = N\sigma^2/(N-1)$, the variance is $\frac{\sigma^2}{n}\cdot\frac{N-n}{N-1}$; the two expressions are equal. To estimate the variance from a sample, $S^2$ must itself be estimated.

Theorem 2 (the sample variance is unbiased for $S^2$). Under simple random sampling with $n \ge 2$, the sample variance $s^2 = \frac{1}{n-1}\sum_{k \in s}(y_k - \bar y)^2$ has $E[s^2] = S^2$. Hence $(1-f)s^2/n$ is unbiased for $\operatorname{Var}(\bar y)$.

Proof. Expand $\sum_{k \in s}(y_k - \bar y)^2 = \sum_{k \in s} y_k^2 - n\bar y^2$. The first term is $\sum_{k \in U} I_k y_k^2$, with expectation $f\sum_k y_k^2$. For the second, $E[\bar y^2] = \operatorname{Var}(\bar y) + \bar Y^2 = (1-f)S^2/n + \bar Y^2$ by Theorem 1. So

$$E\Big[\sum_{k \in s}(y_k - \bar y)^2\Big] = f\sum_k y_k^2 - (1-f)S^2 - n\bar Y^2 = f\Big(\sum_k y_k^2 - N\bar Y^2\Big) - (1-f)S^2 = \big[f(N-1) - (1-f)\big]S^2.$$

The bracket is $(n(N-1) - N + n)/N = (nN - N)/N = n - 1$, which is the divisor of $s^2$. $\blacksquare$

The divisor $N - 1$ in $S^2$ is what makes Theorem 2 exact: if the population variance is defined with divisor $N$, as $\sigma^2$, the sample variance $s^2$ is unbiased for $S^2 = N\sigma^2/(N-1)$, not for $\sigma^2$.

Check on the running population. For samples of $n = 300$ adults, Theorem 1 gives a standard error of $\sqrt{(1 - 300/5{,}182) \times 20.07^2/300} = 1.124$ mm Hg. Drawing 2,000 simple random samples for each of 30 seeds (0 to 29), the standard deviation of the 2,000 sample means had median 1.128 over the seeds (interquartile range 1.115 to 1.138), and the average of $s^2/S^2$ over the 2,000 samples had median 1.000 (IQR 0.998 to 1.002), as Theorem 2 requires.

Stratified sampling

Here is the idea. If the population can be divided in advance into groups that differ in the outcome, sampling each group separately removes the differences between groups from the error, because every sample then contains each group in a fixed share. Stratified sampling divides $U$ into $H$ disjoint strata $U_1, \ldots, U_H$ of sizes $N_h$, and draws an independent simple random sample of $n_h$ units from each. We write $P_h = N_h/N$ for the share of the population in stratum $h$ (Data Collection and Sampling and many sampling texts write $W_h$; this unit keeps $W$ for the treatment of an experiment), $\bar Y_h$ and $S_h^2$ for the stratum's mean and variance (divisor $N_h - 1$), $f_h = n_h/N_h$ for its sampling fraction, and $\bar y_h$, $s_h^2$ for the mean and variance of its sample. The stratified mean is

$$\bar y_{\mathrm{st}} = \sum_{h=1}^H P_h\,\bar y_h.$$

Theorem 3 (stratified sampling). If $n_h \ge 1$ in every stratum, $\bar y_{\mathrm{st}}$ is unbiased for $\bar Y$ and

$$\operatorname{Var}(\bar y_{\mathrm{st}}) = \sum_{h=1}^H P_h^2\,(1 - f_h)\,\frac{S_h^2}{n_h}.$$

If $n_h \ge 2$ in every stratum, $\hat V_{\mathrm{st}} = \sum_h P_h^2 (1-f_h) s_h^2/n_h$ is unbiased for it.

Proof. By Theorem 1 within each stratum, $E[\bar y_h] = \bar Y_h$ and $\operatorname{Var}(\bar y_h) = (1-f_h)S_h^2/n_h$. Then $E[\bar y_{\mathrm{st}}] = \sum_h P_h \bar Y_h = \frac1N\sum_h N_h\bar Y_h = \bar Y$. The samples of different strata are drawn independently, so the $\bar y_h$ are independent and the variance of the weighted sum is the sum of $P_h^2\operatorname{Var}(\bar y_h)$. Unbiasedness of $\hat V_{\mathrm{st}}$ follows from Theorem 2 in each stratum and linearity. $\blacksquare$

The variance involves only the variances within strata. How much smaller that is than the variance of a simple random sample depends on how the $n$ units are allocated to the strata.

Proportional allocation, compared exactly

Proportional allocation takes $n_h = nP_h$, so every stratum has the same sampling fraction $f_h = f$, and Theorem 3 becomes

$$\operatorname{Var}_{\mathrm{prop}} = \frac{1-f}{n}\sum_h P_h S_h^2.$$

A simple random sample of the same size has $\operatorname{Var}_{\mathrm{srs}} = (1-f)S^2/n$. The comparison therefore turns on $S^2$ against $\sum_h P_hS_h^2$, and the analysis of variance splits the population's sum of squares into a within and a between part:

$$(N-1)S^2 = \sum_{k \in U}(y_k - \bar Y)^2 = \sum_h (N_h - 1)S_h^2 + \sum_h N_h(\bar Y_h - \bar Y)^2,$$

because within each stratum $\sum_{k \in U_h}(y_k - \bar Y)^2 = \sum_{k \in U_h}(y_k - \bar Y_h)^2 + N_h(\bar Y_h - \bar Y)^2$, the cross term vanishing since deviations from $\bar Y_h$ sum to zero.

Theorem 4 (proportional allocation against simple random sampling). With $n_h = nP_h$,

$$\operatorname{Var}_{\mathrm{srs}} - \operatorname{Var}_{\mathrm{prop}} = \frac{1-f}{n(N-1)}\Big[\sum_h N_h(\bar Y_h - \bar Y)^2 - \frac1N\sum_h (N - N_h)S_h^2\Big].$$

Stratification with proportional allocation is at least as precise as simple random sampling exactly when the between-strata sum of squares is at least $\frac1N\sum_h(N-N_h)S_h^2$.

Proof. Divide the decomposition by $N - 1$ and subtract $\sum_h P_hS_h^2$:

$$S^2 - \sum_h P_h S_h^2 = \sum_h\Big(\frac{N_h - 1}{N-1} - \frac{N_h}{N}\Big)S_h^2 + \frac{\sum_h N_h(\bar Y_h - \bar Y)^2}{N-1}.$$

The coefficient is $\frac{N(N_h - 1) - N_h(N-1)}{N(N-1)} = \frac{N_h - N}{N(N-1)}$. Multiplying by $(1-f)/n$ gives the result. $\blacksquare$

The second term inside the bracket is $\sum_h(1 - P_h)S_h^2$, whose weights add to $H - 1$, so it is about $H - 1$ times a typical within-stratum variance, while the first grows with $N$ whenever the strata means differ. With equal strata, $N_h = N/H$, the bracket is $(H-1)(\mathrm{MSB} - \mathrm{MSW})$ with $\mathrm{MSB} = \sum_h N_h(\bar Y_h - \bar Y)^2/(H-1)$ and $\mathrm{MSW} = \frac1H\sum_h S_h^2$, so proportional stratification is at least as precise exactly when $\mathrm{MSB} \ge \mathrm{MSW}$, and strictly more precise when $\mathrm{MSB} > \mathrm{MSW}$, the condition that Blocking and Factorial Designs finds for blocking. For strata of any size the first term dominates unless the strata means are nearly equal: stratification "helps exactly as much as the strata differ in the outcome", as Data Collection and Sampling put it for a proportion, and Theorem 4 is the exact statement of that sentence, including the small case in which it fails (Worked problem 2).

The same algebra in experiments. The decomposition into within and between sums of squares is also what makes a blocked experiment more precise than a completely randomized one. Blocking and Factorial Designs proves that version, where every unit has two potential outcomes and the comparison involves both.

Check on the running population. Stratify the 5,182 adults by age group: 1,696 aged 18 to 39, 1,574 aged 40 to 59 and 1,912 aged 60 and over. Their mean systolic pressures are 115.03, 126.35 and 136.76 mm Hg, with standard deviations $S_h$ of 12.89, 18.56 and 20.94. The between-strata term is $\sum_h N_h(\bar Y_h - \bar Y)^2/(N-1) = 81.94$ and the within correction $\sum_h (N-N_h)S_h^2/[N(N-1)] = 0.121$, so $S^2 = 402.66$ exceeds $\sum_h P_hS_h^2 = 320.84$ by $81.94 - 0.121$. Proportional allocation of $n = 300$ gives 98, 91 and 111 adults (rounded by largest remainders) and a standard error of 1.003 mm Hg, a variance ratio of 0.796 against simple random sampling. The Monte Carlo standard error, over the same 30 seeds of 2,000 samples, was 1.006 (IQR 0.997 to 1.011).

Neyman allocation

Proportional allocation spends the sample in proportion to the strata's sizes. A stratum whose values vary more deserves more of the sample, because its mean is harder to pin down. Neyman (1934) found the allocation that minimizes the variance for a fixed total sample size (as Tillé and Wilhelm, 2017, summarize it); Tschuprow (1923) had obtained the result over a decade earlier, as Wright (2014) records. Cauchy–Schwarz gives it in one line.

Theorem 5 (optimal allocation). (a) Among real allocations $n_1, \ldots, n_H > 0$ with $\sum_h n_h = n$, the variance of Theorem 3 is smallest when

$$n_h = n\,\frac{N_hS_h}{\sum_j N_jS_j} \qquad\text{(Neyman allocation)},$$

and the minimum is $\operatorname{Var}_{\mathrm{opt}} = \frac{1}{n}\big(\sum_h P_hS_h\big)^2 - \frac1N\sum_h P_hS_h^2$.

(b) $\operatorname{Var}_{\mathrm{prop}} - \operatorname{Var}_{\mathrm{opt}} = \frac1n\sum_h P_h(S_h - \bar S)^2$, where $\bar S = \sum_h P_hS_h$.

(c) If a unit in stratum $h$ costs $c_h$ and the budget is $\sum_h c_hn_h = C$, the variance is smallest when $n_h \propto N_hS_h/\sqrt{c_h}$.

Proof. (a) Expand Theorem 3: $\operatorname{Var} = \sum_h P_h^2S_h^2/n_h - \sum_h P_h^2S_h^2/N_h = \sum_h P_h^2S_h^2/n_h - \frac1N\sum_h P_hS_h^2$. The second sum does not depend on the allocation. For the first, the Cauchy–Schwarz inequality applied to the vectors $(P_hS_h/\sqrt{n_h})_h$ and $(\sqrt{n_h})_h$ gives

$$\Big(\sum_h P_hS_h\Big)^2 = \Big(\sum_h \frac{P_hS_h}{\sqrt{n_h}}\cdot\sqrt{n_h}\Big)^2 \le \Big(\sum_h \frac{P_h^2S_h^2}{n_h}\Big)\Big(\sum_h n_h\Big) = n\sum_h\frac{P_h^2S_h^2}{n_h},$$

so $\sum_h P_h^2S_h^2/n_h \ge (\sum_h P_hS_h)^2/n$ for every allocation. Equality holds exactly when the two vectors are proportional, $P_hS_h/\sqrt{n_h} = \lambda\sqrt{n_h}$, that is $n_h = P_hS_h/\lambda$; the constraint $\sum_h n_h = n$ fixes $\lambda = \sum_j P_jS_j/n$, which is the stated allocation.

(b) $\operatorname{Var}_{\mathrm{prop}} = \frac{1-f}{n}\sum_h P_hS_h^2 = \frac1n\sum_h P_hS_h^2 - \frac1N\sum_h P_hS_h^2$. Subtracting (a), the $1/N$ terms cancel and $\operatorname{Var}_{\mathrm{prop}} - \operatorname{Var}_{\mathrm{opt}} = \frac1n\big[\sum_h P_hS_h^2 - \bar S^2\big]$, which is $\frac1n\sum_h P_h(S_h - \bar S)^2$ because the $P_h$ sum to 1.

(c) Apply Cauchy–Schwarz to $(P_hS_h/\sqrt{n_h})_h$ and $(\sqrt{c_hn_h})_h$: $(\sum_h P_hS_h\sqrt{c_h})^2 \le (\sum_h P_h^2S_h^2/n_h)\,C$, with equality exactly when $P_hS_h/\sqrt{n_h} \propto \sqrt{c_hn_h}$, that is $n_h \propto P_hS_h/\sqrt{c_h}$. $\blacksquare$

Part (b) says that Neyman allocation beats proportional allocation by the variance of the strata standard deviations, so the gain is zero when all strata are equally variable. Two practical cautions follow from the proof. The optimum is a real allocation and must be rounded; and it requires $n_h \le N_h$, which can fail for a small, very variable stratum, in which case that stratum is taken whole and the rest reallocated. The second does not occur here; the first does, and the rounded allocation is used below.

Check on the running population. The age strata's standard deviations differ (12.89 against 20.94), so Neyman allocation moves adults from the young stratum to the old: 72, 96 and 132 instead of 98, 91 and 111. The standard error falls to 0.984, a variance ratio of 0.766 against simple random sampling, and the Monte Carlo value was 0.979 (IQR 0.970 to 0.988). Part (b) explains the size of the gain: with $\bar S = 17.58$, $\sum_h P_h(S_h - \bar S)^2 = 11.64$, small beside $\sum_h P_hS_h^2$.

Example (one allocation for two variables). An allocation optimal for one variable need not be for another. For hypertension the stratum standard deviations are $\sqrt{p_h(1-p_h)}$ times $\sqrt{N_h/(N_h - 1)}$, and since the middle group's prevalence is closest to one half, Neyman allocation is 95, 103 and 102. Proportional allocation gives hypertension a variance ratio of 0.770 against simple random sampling, its own Neyman allocation 0.764, and the allocation that is optimal for systolic pressure 0.803. A survey measures hundreds of variables, which is one reason surveys rarely use Neyman allocation for precision and allocate instead for the subgroups they must report on.

Cluster sampling

Stratification samples every group; cluster sampling samples some groups and measures everyone in them, or a sample of everyone. Its motive is cost, not precision. NHANES's examination centers are trucks that are driven to a location and set up for weeks, so examining about 170 adults in each of 30 locations costs far less than examining 5,182 adults scattered across the country. The price is that people in the same location resemble each other, so each additional person from a location adds less information than a person from a new one. The idea in one sentence: the cluster means, not the people, are the sampled values, and their variance depends on how much of the population's variance lies between clusters.

One-stage sampling of equal clusters

Suppose the population consists of $N$ clusters of $M$ units each, $NM$ units in all, with $y_{ij}$ the value of unit $j$ in cluster $i$, $\bar Y_i$ the cluster mean and $\bar Y$ the overall mean. A one-stage cluster sample draws $n$ clusters by simple random sampling and measures all $M$ units of each. The estimator of $\bar Y$ is the mean of the $nM$ values, which is the mean of the $n$ sampled cluster means. Three population quantities describe the clustering:

  • $S^2 = \frac{1}{NM-1}\sum_{i,j}(y_{ij} - \bar Y)^2$, the variance among all units;
  • $S_b^2 = \frac{1}{N-1}\sum_i(\bar Y_i - \bar Y)^2$, the variance among cluster means;
  • $S_w^2 = \frac{1}{N(M-1)}\sum_{i,j}(y_{ij} - \bar Y_i)^2$, the average variance within clusters.

The intraclass correlation measures how alike two units of the same cluster are. In the design-based setting it is defined from the population itself, as the average product of deviations over all pairs of different units in the same cluster, relative to $S^2$:

$$\rho = \frac{\sum_i\sum_{j \ne j'}(y_{ij} - \bar Y)(y_{ij'} - \bar Y)}{(M-1)(NM-1)\,S^2}.$$

There are $NM(M-1)$ such ordered pairs, so $\rho$ is approximately the correlation between two different units drawn from the same cluster. Clustered Standard Errors and The Binomial Distribution derive the design effect from a model with a common correlation; the theorem below derives it from the population, with no model and an exact constant.

Theorem 6 (one-stage cluster sampling). Under simple random sampling of $n$ of the $N$ clusters, with $f = n/N$:

(a) the sample mean $\bar y_{\mathrm{cl}}$ is unbiased for $\bar Y$ with $\operatorname{Var}(\bar y_{\mathrm{cl}}) = (1-f)S_b^2/n$;

(b) $S_b^2 = \dfrac{NM - 1}{M^2(N-1)}\,S^2\,[1 + (M-1)\rho]$ and $S_w^2 = \dfrac{NM-1}{NM}\,S^2(1 - \rho)$;

(c) relative to a simple random sample of the same $nM$ units, the design effect is

$$\frac{\operatorname{Var}(\bar y_{\mathrm{cl}})}{\operatorname{Var}_{\mathrm{srs}}} = \frac{NM - 1}{M(N-1)}\,[1 + (M-1)\rho] \approx 1 + (M - 1)\rho.$$

Proof. (a) The clusters are the sampling units and their means $\bar Y_i$ are the values, so Theorem 1 applied to the $N$ cluster means gives unbiasedness and the variance.

(b) The total deviation of cluster $i$ is $\sum_j(y_{ij} - \bar Y) = M(\bar Y_i - \bar Y)$. Squaring it gives the squares of the terms plus the products of pairs of different terms:

$$M^2(\bar Y_i - \bar Y)^2 = \sum_j(y_{ij} - \bar Y)^2 + \sum_{j \ne j'}(y_{ij} - \bar Y)(y_{ij'} - \bar Y).$$

Summing over clusters, the first part totals $(NM-1)S^2$ and the second is $(M-1)(NM-1)\rho S^2$ by the definition of $\rho$. So $M^2(N-1)S_b^2 = (NM-1)S^2[1 + (M-1)\rho]$, the first identity. For the second, the analysis of variance with clusters as groups gives $(NM-1)S^2 = M(N-1)S_b^2 + N(M-1)S_w^2$. Substituting the first identity,

$$N(M-1)S_w^2 = (NM-1)S^2\Big[1 - \frac{1 + (M-1)\rho}{M}\Big] = (NM-1)S^2\,\frac{(M-1)(1-\rho)}{M},$$

and dividing by $N(M-1)$ gives $S_w^2$.

(c) A simple random sample of $nM$ of the $NM$ units has sampling fraction $nM/(NM) = f$ and variance $(1-f)S^2/(nM)$. Dividing (a) by it and using (b),

$$\frac{(1-f)S_b^2/n}{(1-f)S^2/(nM)} = \frac{M S_b^2}{S^2} = \frac{NM-1}{M(N-1)}[1 + (M-1)\rho].$$

The leading factor is $\frac{NM - 1}{NM - M}$, which tends to 1 as $N$ grows. $\blacksquare$

Part (b) also bounds $\rho$. Since $S_b^2 \ge 0$, $\rho \ge -1/(M-1)$, attained when every cluster has the same mean; since $S_w^2 \ge 0$, $\rho \le 1$, attained when every cluster is internally constant. Negative values are possible, and by part (c) clustering then helps once $\rho < -1/(NM-1)$, a small threshold: a cluster that is a miniature of the population, as a household of one man and one woman is for a variable that differs by sex, gives a design effect below 1.

Check on the running population. The 30 locations that the public files identify (the masked variance units described below) hold 118 to 219 of the adults. To get equal clusters, 118 adults were kept at random from each (numpy seed 20261119), a population of 30 clusters of 118, 3,540 adults in all, with mean systolic pressure 126.47. In it the intraclass correlation is 0.00852 for systolic pressure and 0.00803 for hypertension. For a sample of $n = 5$ clusters (590 adults), Theorem 6 gives a standard error of 1.084 mm Hg for systolic pressure and 0.0265 for hypertension. The Monte Carlo values over 30 seeds of 2,000 samples were 1.086 (IQR 1.071 to 1.096) and 0.0266 (IQR 0.0262 to 0.0268). The design effects are 2.07 and 2.01; the approximation $1 + (M-1)\rho$ gives 2.00 and 1.94 (from the unrounded $\rho$), the difference being the factor $(NM-1)/(M(N-1)) = 1.0342$ for only 30 clusters. A correlation of less than one percent doubles the variance: each of the 117 further adults of a cluster repeats a little of what the first one said, and 117 small repetitions add up to a whole extra variance.

Two-stage sampling

Measuring everyone in a cluster is wasteful when $\rho > 0$: the hundredth person from a location says little that the first ninety-nine did not. A two-stage sample draws $n$ clusters, called primary sampling units (PSUs), and then a simple random sample of $m_i$ of the $M_i$ units within each selected PSU, independently from PSU to PSU. Clusters may now have unequal sizes. Write $t_i = \sum_j y_{ij}$ for the total of PSU $i$, $S_t^2$ for the variance of the $N$ PSU totals (divisor $N - 1$), $S_i^2$ for the variance within PSU $i$ (divisor $M_i - 1$), and $N_U = \sum_i M_i$ for the number of units in the population. Within a selected PSU, $\hat t_i = M_i\bar y_i$ estimates its total from the sample mean $\bar y_i$.

Theorem 7 (two-stage sampling). With simple random sampling at both stages, $f_1 = n/N$ and $f_{2i} = m_i/M_i$, the estimator $\hat t = \frac{N}{n}\sum_{i \in s}\hat t_i$ is unbiased for the population total $t$, and

$$\operatorname{Var}(\hat t) = N^2(1 - f_1)\frac{S_t^2}{n} + \frac{N}{n}\sum_{i=1}^N M_i^2(1 - f_{2i})\frac{S_i^2}{m_i}.$$

Proof. Condition on the first-stage sample $s$. Each $\hat t_i$ is $M_i$ times a simple random sample mean, so by Theorem 1, $E[\hat t_i \mid s] = t_i$ and $\operatorname{Var}(\hat t_i \mid s) = M_i^2(1-f_{2i})S_i^2/m_i =: V_i$, and the $\hat t_i$ are conditionally independent. Hence

$$E[\hat t \mid s] = \frac Nn\sum_{i \in s} t_i, \qquad \operatorname{Var}(\hat t \mid s) = \frac{N^2}{n^2}\sum_{i \in s}V_i.$$

The first is $N$ times the mean of a simple random sample of $n$ PSU totals, so by Theorem 1 its expectation is $N \cdot t/N = t$ and its variance is $N^2(1-f_1)S_t^2/n$. The second has expectation $\frac{N^2}{n^2}\cdot\frac nN\sum_{i=1}^N V_i = \frac Nn\sum_i V_i$, because each PSU is in $s$ with probability $n/N$. The law of total variance, $\operatorname{Var}(\hat t) = \operatorname{Var}(E[\hat t \mid s]) + E[\operatorname{Var}(\hat t \mid s)]$ (Conditional Expectation), adds the two. $\blacksquare$

The two terms are the between-PSU and within-PSU components. The first does not shrink as $m_i$ grows: no amount of sampling within the chosen locations makes up for having chosen few locations.

Equal clusters and the design effect. When all clusters have size $M$ and $m_i = m$, dividing by $NM$ turns $\hat t$ into the sample mean $\bar y$, and Theorem 7 becomes

$$\operatorname{Var}(\bar y) = (1 - f_1)\frac{S_b^2}{n} + (1 - f_2)\frac{S_w^2}{nm},$$

because $S_t^2 = M^2S_b^2$ and the average of the $S_i^2$ is $S_w^2$. Put in the identities of Theorem 6(b) and suppose that the number of clusters $N$ is large and the first-stage fraction $f_1$ small, so that the factors $(NM-1)/(M(N-1))$, $(NM-1)/(NM)$ and $1 - f_1$ can be replaced by 1. Then $S_b^2 \approx S^2[1 + (M-1)\rho]/M$ and $S_w^2 \approx S^2(1-\rho)$, and with $1 - f_2 = 1 - m/M$,

$$\operatorname{Var}(\bar y) \approx \frac{S^2}{n}\Big[\frac{1 + (M-1)\rho}{M} + (1-\rho)\Big(\frac1m - \frac1M\Big)\Big] = \frac{S^2}{n}\Big[\rho + \frac{1-\rho}{m}\Big] = \frac{S^2}{nm}\,\big[1 + (m-1)\rho\big],$$

because the terms in $1/M$ combine to $[1 + (M-1)\rho - (1-\rho)]/M = \rho$. A simple random sample of the same $nm$ units has variance close to $S^2/(nm)$ when $f_1$ is small. This is the design effect $1 + (m-1)\rho$ of a two-stage sample with $m$ units per PSU, relative to a simple random sample of the same $nm$ units. It depends on the number of units taken per PSU, $m$, not on the size of the PSU, $M$.

The table checks it on the equal-cluster population, drawing $n = 6$ of the 30 clusters and $m$ adults in each, for hypertension. The exact design effect is Theorem 7's variance over that of a simple random sample of $6m$ adults from the 3,540; the approximation uses $\rho = 0.00803$.

$m$ per cluster Exact design effect $1 + (m-1)\rho$
1 0.999 1.000
5 1.026 1.032
10 1.062 1.072
25 1.171 1.193
50 1.367 1.394
118 2.006 1.940

The approximation is within 2% up to 50 per cluster. At $m = 118$ every adult of a chosen cluster is taken, and the exact value is the one-stage design effect of Theorem 6 with its finite factor.

Clusters of unequal size: the ratio estimator

NHANES's locations, like most real clusters, differ in size, and Theorem 7 then holds a trap. Its estimator of the mean is $\hat t/N_U$, and its variance contains $S_t^2$, the variance of the PSU totals. Totals vary with the size of the PSU, even when the PSU means are all equal. The usual remedy divides by the estimated population size instead:

$$\hat R = \frac{\sum_{i \in s}\hat t_i}{\sum_{i \in s} M_i} = \frac{\hat t}{\hat N_U}, \qquad \hat N_U = \frac Nn\sum_{i \in s}M_i.$$

The plan of the analysis: write the error of this ratio exactly as an estimated total of deviations, then read off its variance from Theorem 7. Write $e_{ij} = y_{ij} - \bar Y$. Since $\hat t - \bar Y\hat N_U = \frac Nn\sum_{i \in s}M_i(\bar y_i - \bar Y)$ is Theorem 7's estimator applied to the $e_{ij}$,

$$\hat R - \bar Y = \frac{\hat t_e}{\hat N_U}, \qquad \hat t_e = \frac Nn\sum_{i \in s}M_i\bar e_i,$$

exactly. The estimator $\hat t_e$ is unbiased for $\sum_{i,j}e_{ij} = 0$, and $\hat N_U/N_U$ is close to 1 when the sample of PSUs is not too small, so to first order $\hat R - \bar Y \approx \hat t_e/N_U$ and $\operatorname{Var}(\hat R) \approx \operatorname{Var}(\hat t_e)/N_U^2$, which is Theorem 7 with $t_i$ replaced by $t_i - M_i\bar Y$. The within-PSU variances are unchanged by subtracting a constant. This linearized variance is the same device that Survey Weights and the Horvitz–Thompson Estimator uses for the Hájek estimator, where the remainder is studied.

Check on the running population. Draw 6 of the 30 locations and 50 adults in each, 300 adults in all, the two-stage design of Data Collection and Sampling. For systolic pressure the unbiased estimator $\hat t/N_U$ has standard error 6.770 mm Hg by Theorem 7, and 0.979 of its variance is the first-stage term: the location totals vary because the locations hold between 118 and 219 adults. The ratio estimator's linearized standard error is 1.355. The Monte Carlo values over 30 seeds of 2,000 samples were 6.745 (IQR 6.681 to 6.804) and 1.354 (IQR 1.341 to 1.364). For hypertension the first-stage term is a smaller share, 0.721, of the unbiased estimator's variance, and the two standard errors are 0.0462 and 0.0335, against Monte Carlo values of 0.0462 (IQR 0.0458 to 0.0465) and 0.0336 (IQR 0.0332 to 0.0339). The ratio estimator is the one that page simulated, and its 0.0338 is within this interquartile range. For systolic pressure the unbiased estimator has a standard error five times as large as the ratio estimator's (a variance 25 times as large): exact unbiasedness is a poor reason to choose an estimator.

Estimating the variance: PSUs treated as drawn with replacement

Theorem 7 gives the variance in terms of population quantities. To estimate it from one sample, survey software does not estimate the two components separately. It uses only the variation among the PSU estimates, as if the PSUs had been drawn with replacement; Data Collection and Sampling applies the formula to NHANES. Why is that valid, and what does it cost?

Theorem 8 (the with-replacement variance estimator). (a) Suppose $n$ PSUs are drawn independently with replacement, PSU $i$ with probability $p_i$ at each draw ($\sum_i p_i = 1$), and that each time a PSU is drawn a fresh, independent second-stage sample gives an unbiased estimate $\hat t_{(j)}$ of its total. Let $u_j = \hat t_{(j)}/p_{(j)}$ for draw $j$. Then $\hat t = \bar u = \frac1n\sum_j u_j$ is unbiased for $t$, and

$$v = \frac{1}{n(n-1)}\sum_{j=1}^n (u_j - \bar u)^2$$

is unbiased for $\operatorname{Var}(\hat t)$, whatever the second-stage design.

(b) If instead the PSUs are drawn by simple random sampling without replacement and $u_j = N\hat t_{(j)}$ (the design of Theorem 7), then

$$E[v] = \operatorname{Var}(\hat t) + f_1 N^2\frac{S_t^2}{n} \ \ge\ \operatorname{Var}(\hat t),$$

so $v$ overstates the variance by exactly $f_1N^2S_t^2/n$: the fraction $f_1/(1 - f_1)$ of the between-PSU component of Theorem 7, or equivalently the share $f_1$ of the with-replacement between-PSU term $N^2S_t^2/n$.

Proof. (a) The draws are independent and identically distributed: each picks PSU $i$ with probability $p_i$ and then estimates its total by an independent subsample. So the $u_j$ are i.i.d., with $E[u_j] = \sum_i p_i\,E[\hat t_i]/p_i = \sum_i t_i = t$. The mean of i.i.d. variables is unbiased for their common mean, with variance $\operatorname{Var}(u_1)/n$, and their sample variance with divisor $n - 1$ is unbiased for $\operatorname{Var}(u_1)$ (Expectation, Variance and Covariance). Dividing by $n$ gives the result.

(b) Condition on the first-stage sample $s$. The $u_j$ are then independent with means $Nt_i$ and variances $N^2V_i$ ($V_i$ as in the proof of Theorem 7). For independent variables with means $\mu_j$ and variances $\sigma_j^2$, $E\sum_j(X_j - \bar X)^2 = \sum_j(\mu_j - \bar\mu)^2 + (1 - \frac1n)\sum_j\sigma_j^2$, because $X_j - \bar X = (\mu_j - \bar\mu) + (\varepsilon_j - \bar\varepsilon)$ with independent mean-zero errors $\varepsilon_j$, and $E\sum_j(\varepsilon_j - \bar\varepsilon)^2 = \sum_j\sigma_j^2 - \frac1n\sum_j\sigma_j^2$. So

$$E\Big[\sum_j(u_j - \bar u)^2 \,\Big|\, s\Big] = N^2\sum_{i \in s}(t_i - \bar t_s)^2 + \Big(1 - \frac1n\Big)N^2\sum_{i \in s}V_i.$$

Over the first stage, $E\sum_{i \in s}(t_i - \bar t_s)^2 = (n-1)S_t^2$ by Theorem 2, and $E\sum_{i \in s}V_i = \frac nN\sum_iV_i$. Dividing by $n(n-1)$,

$$E[v] = N^2\frac{S_t^2}{n} + \frac Nn\sum_i V_i,$$

which exceeds Theorem 7's variance by $N^2S_t^2/n - N^2(1-f_1)S_t^2/n = f_1N^2S_t^2/n$. $\blacksquare$

Corollary 1 (strata). If the PSUs are drawn independently in each stratum $h$ as in part (a), with probabilities $p_{hi}$ that sum to 1 over the PSUs of the stratum and $n_h \ge 2$ draws giving values $u_{hj} = \hat t_{(hj)}/p_{(hj)}$, so that $\bar u_h$ is unbiased for the stratum total $t_h$, then $\hat t = \sum_h\bar u_h$ is unbiased for $t$ and $v = \sum_h\frac{1}{n_h(n_h-1)}\sum_j(u_{hj} - \bar u_h)^2$ is unbiased for $\operatorname{Var}(\hat t)$. With $n_h = 2$, a stratum's term is $(u_{h1} - u_{h2})^2/4$.

Proof. Part (a) applies within each stratum, and the strata are sampled independently, so the stratum estimators are independent and their variances add, as in Theorem 3. $\blacksquare$

Part (b) carries over stratum by stratum in the same way: with simple random sampling of PSUs without replacement in each stratum, $v$ overstates the variance by $\sum_h f_{1h}N_h^2S_{th}^2/n_h$, in the notation of Theorem 7 applied within stratum $h$.

Part (a) covers sampling with probability proportional to size with replacement as well as equal probabilities. Part (b) is the reason it is used even when PSUs are drawn without replacement: it needs nothing from the second stage, and its bias is small and conservative when few of the population's PSUs are drawn. The NHANES variance tutorial gives exactly this reason for treating its PSUs as drawn with replacement, a first-stage sampling fraction small enough that the correction is close to 1 (National Center for Health Statistics, n.d.).

Check on the running population. In the 6-of-30 design $f_1 = 0.2$ is large, and the first-stage component dominates for systolic pressure, so the overstatement shows: Theorem 8(b) gives $E[v]/\operatorname{Var}(\hat t) = 1.245$, and the Monte Carlo average of $v$ over 2,000 samples, divided by the exact variance, had median 1.240 over the 30 seeds (IQR 1.232 to 1.252). For hypertension the ratio is 1.180. With 30 PSUs drawn from all the counties of the United States, the same overstatement is negligible.

How NHANES combines the designs

NHANES since 1999 has used "multi-year, stratified, clustered four-stage samples," in the words of its 2015 to 2018 design report (Chen et al., 2020), and each stage is one of the building blocks above. The report describes the four stages as PSUs (counties, groups of tracts within counties, or groups of adjacent counties), segments within PSUs (census blocks or combinations of blocks), dwelling units within segments, and persons within dwelling units, with households screened to find the persons to be sampled.

Why strata. The noncertainty PSUs were grouped into 14 major strata, defined within four groups of states that had similar values of a state health index (built from death and infant mortality rates and the percentages of adults with high blood pressure, who are overweight or obese, who have poor nutrition and who smoke) and by census region and the share of the population living in rural areas; each major stratum was divided into 4 minor strata (Chen et al., 2020). Strata built from health indicators are strata whose means differ in the variables the survey measures, which is when Theorem 4 promises a gain. Stratifying also guarantees that every kind of place is represented in every two-year sample.

Why clusters, and why so few. The examination is done in mobile centers. The analytic guidelines state the trade-off in the terms of this article: the ideal is "to limit the correlation among sample persons within clusters by sampling more clusters with fewer people in each," but "because of operational limitations (e.g., the number of MECs, geographic distances between locations, etc.) NHANES currently samples only 30 locations (primary sampling units [PSUs]) within a 2-year survey cycle" (National Center for Health Statistics, 2018, §3.2). Fifteen PSUs are visited each year, one from each major stratum plus one PSU selected with certainty, so a two-year cycle comes from 30 PSUs (Chen et al., 2020).

Why unequal probabilities. PSUs were selected with probability proportional to a measure of size, a weighted count of residents that gives higher probability to places with larger shares of the groups the survey oversamples; within PSUs, 24 segments were drawn from each noncertainty PSU, also with probability proportional to size (Chen et al., 2020). Persons were then sampled at rates that depend on age, sex, race and Hispanic origin, and income. Unequal selection probabilities are what the weights of Survey Weights and the Horvitz–Thompson Estimator undo.

The cost argument, quantified. Theorem 7's large-population form makes the trade-off a calculation. If a PSU costs $c_1$ (travel and setup) and each examined person $c_2$, then $n$ PSUs of $m$ persons cost $C = n(c_1 + mc_2)$ and have variance proportional to $[\rho + (1-\rho)/m]/n$. The product of cost and variance is

$$(c_1 + mc_2)\Big(\rho + \frac{1-\rho}{m}\Big) = c_1\rho + c_2(1-\rho) + \frac{c_1(1-\rho)}{m} + c_2\rho\,m,$$

and the arithmetic–geometric mean inequality, $a/m + bm \ge 2\sqrt{ab}$ with equality at $m = \sqrt{a/b}$, shows that it is smallest at

$$m_{\mathrm{opt}} = \sqrt{\frac{c_1}{c_2}\cdot\frac{1-\rho}{\rho}}.$$

Minimizing the product of cost and variance is the same as minimizing variance at a fixed budget, since for fixed $m$ the variance is inversely proportional to $n$ and so to the budget. Among the 5,182 adults, the one-way analysis-of-variance estimate of the intraclass correlation within the 30 locations (with the average-size constant $n_0 = 172.6$ adults for unequal groups, as computed in Data Collection and Sampling) is 0.0108 for hypertension and 0.0101 for systolic pressure (0.01013 unrounded, which gives the clustering factor 2.74 of the opening). With $\rho = 0.0108$, a location that costs as much as 50 examinations ($c_1/c_2 = 50$) makes 68 adults per location optimal, and one that costs 200 examinations makes 135 optimal. Here the ANOVA estimate stands in for the population $\rho$ defined before Theorem 6, and the 30 locations are treated as equal clusters of their mean size. Turned around, the mean take of 172.7 adults per location is the optimum for hypertension if a location costs about 325 adult examinations. These cost ratios are illustrations of the formula, not NHANES figures; the survey's costs are not in its public reports. The price is real all the same: with 172.7 adults per location and $\rho = 0.0108$, clustering alone would multiply the variance of hypertension prevalence by $1 + 171.7 \times 0.0108 = 2.85$ in an equal-probability design. With unequal sizes the effective cluster size is $m^* = \sum_i m_i^2/\sum_i m_i = 176.0$ (Clustered Standard Errors, Theorem 2), which gives 2.89.

Masked variance units. Corollary 1 needs, for each person, the stratum and the PSU. Releasing the PSUs would let the visited counties be identified, so the public files carry masked variance units instead: a pseudo-stratum SDMVSTRA and a pseudo-PSU SDMVPSU within it, 15 pseudo-strata of 2 pseudo-PSUs each in 2017 to 2018 (National Center for Health Statistics, 2020). The analytic guidelines say that they "are not the 'true' design PSUs. They are a collection of secondary sampling units aggregated into groups for the purpose of variance estimation," and that they "produce variance estimates that closely approximate the variances that would have been estimated using the 'true' design" (National Center for Health Statistics, 2018, §3.2.1). The design report adds how the true strata were mapped to variance strata: the two noncertainty PSUs drawn from each major stratum in a two-year sample generally serve as its two variance units, while a certainty PSU, not selected within a stratum, may be split into several variance units or combined with others (Chen et al., 2020). With two pseudo-PSUs per pseudo-stratum, each stratum's term in the with-replacement estimator is a single squared difference, and the estimator has $30 - 15 = 15$ degrees of freedom, the number that NHANES intervals use (Data Collection and Sampling).

What is proved, what is cited, what the experiment shows

Proved on this page: the inclusion probabilities and indicator covariances of simple random sampling (Lemma 1); unbiasedness and the variance with the finite population correction of the sample mean (Theorem 1); unbiasedness of the sample variance for $S^2$ (Theorem 2); the variance of stratified sampling and its unbiased estimator (Theorem 3); the exact difference between simple random and proportionally stratified sampling, including when stratification loses (Theorem 4); Neyman's optimal allocation, its gain over proportional allocation, and the allocation with costs (Theorem 5); the variance of one-stage cluster sampling, the identities linking $S_b^2$ and $S_w^2$ to the intraclass correlation, and the exact design effect (Theorem 6); the variance of two-stage sampling with unequal clusters (Theorem 7) and its large-population design effect $1 + (m-1)\rho$; the exact error identity behind the linearized variance of the ratio estimator; the unbiasedness of the with-replacement variance estimator and its exact overstatement under first-stage sampling without replacement (Theorem 8), and its stratified form (Corollary 1); the optimal take per cluster under a linear cost model.

Cited: Neyman's (1934) optimal allocation, as summarized by Tillé and Wilhelm (2017), and Tschuprow's (1923) earlier derivation, as recorded by Wright (2014; Tschuprow not opened); Cochran (1977) as the standard text whose conventions are followed (not opened for this article; every result used is derived here); everything about NHANES's design, from its design report (Chen et al., 2020), its analytic guidelines (National Center for Health Statistics, 2018), its variance estimation tutorial (National Center for Health Statistics, n.d.) and the documentation of the demographic file (National Center for Health Statistics, 2020).

What the experiments show: on the 5,182 adults treated as a finite population, the standard errors of every design above agreed with their formulas to within the Monte Carlo variation of 30 seeds of 2,000 samples; the ratio estimator for unequal clusters had a standard error about one fifth of the unbiased one's for systolic pressure; the with-replacement estimator overstated the variance by the predicted factor when a fifth of the PSUs were drawn. The intraclass correlations measured on the 30 pseudo-PSUs describe these adults in these locations and not the clustering of the true design.

Worked problems

These problems are posed on the unit's NHANES population and on small exact cases constructed for the purpose; every number is recomputed in the article's script.

Problem 1 (every sample of a small population). Six adults have systolic pressures 118, 126, 131, 140, 152 and 171 mm Hg. List the simple random samples of size 3, and verify Theorems 1 and 2 by enumeration.

There are $\binom63 = 20$ samples, each with probability $1/20$. The population mean is $\bar Y = 838/6 = 139.67$ and $S^2 = 373.07$ (divisor 5). Averaging the 20 sample means gives exactly $\bar Y$, because each adult appears in $\binom52 = 10$ of the 20 samples, and so contributes $10 \times y_k/3$ to the sum of the means, whose average is then $\frac{10}{3 \cdot 20}\sum_k y_k = \bar Y$. The variance of the 20 means about $\bar Y$ is 62.18, and Theorem 1 gives $(1 - 3/6) \times 373.07/3 = 62.18$. Averaging the 20 sample variances gives exactly $S^2 = 373.07$, as Theorem 2 states; averaging the variances computed with divisor 3 instead of 2 would give two thirds of it.

Problem 2 (a stratification that loses). A population has four units with values 0, 10, 1 and 9, in two strata $\{0, 10\}$ and $\{1, 9\}$. Compare the variance of the mean of a simple random sample of 2 with that of a stratified sample taking one unit from each stratum, and check Theorem 4.

The population mean is 5 and $S^2 = (25 + 25 + 16 + 16)/3 = 27.33$. Simple random sampling gives $\operatorname{Var} = (1 - 2/4) \times 27.33/2 = 6.83$. The strata have the same mean, 5, and within-stratum variances (divisor $N_h - 1 = 1$) $S_A^2 = 50$ and $S_B^2 = 32$, so Theorem 3 with $P_h = 1/2$, $n_h = 1$, $f_h = 1/2$ gives $\operatorname{Var} = \frac14\cdot\frac12\cdot 50 + \frac14\cdot\frac12\cdot 32 = 10.25$. Stratification is worse. Theorem 4 predicts the difference: the between-strata sum of squares is 0, and the correction is $\frac14(2 \times 50 + 2 \times 32) = 41$, so $\operatorname{Var}_{\mathrm{srs}} - \operatorname{Var}_{\mathrm{prop}} = \frac{1/2}{2 \times 3}(0 - 41) = -3.42$, which is $6.83 - 10.25$. Stratification lost because it forced each sample to contain one unit of each stratum, while simple random sampling sometimes draws the pair $\{0, 10\}$ or $\{1, 9\}$, whose means are exactly right. With strata whose means do not differ, proportional stratification always loses a little, by $\frac{1-f}{n(N-1)}\sum_h(1-P_h)S_h^2$ (Theorem 4); the loss is negligible except in tiny populations.

Problem 3 (allocation with costs). A population has two strata: $N_1 = 4{,}000$ units with $S_1 = 10$ costing 1 each to measure, and $N_2 = 1{,}000$ units with $S_2 = 30$ costing 4 each. The budget is 1,200. Find the optimal allocation and compare it with proportional allocation at the same cost.

By Theorem 5(c), $n_h \propto N_hS_h/\sqrt{c_h}$: $40{,}000/1$ and $30{,}000/2 = 15{,}000$. The budget fixes the constant: $\lambda(40{,}000 \times 1 + 15{,}000 \times 4) = 1{,}200$, so $\lambda = 0.012$, $n_1 = 480$ and $n_2 = 180$, costing $480 + 720 = 1{,}200$. Theorem 3 gives $\operatorname{Var} = 0.8^2 \times 100/480 + 0.2^2 \times 900/180 - (0.8 \times 100 + 0.2 \times 900)/5{,}000 = 0.2813$. Proportional allocation spends $n(0.8 \times 1 + 0.2 \times 4) = 1.6n$ per sample, so the budget buys $n = 750$, and its variance is $(1 - 750/5{,}000)/750 \times (0.8 \times 100 + 0.2 \times 900) = 0.2947$, a ratio of 1.05 to the optimum. The optimum samples the expensive stratum less than its share of the variance alone would suggest (Neyman allocation without costs would give it 43% of the sample, $30{,}000/70{,}000$; with costs it gets 27%).

Problem 4 (the effective size of a clustered sample). Using the intraclass correlation of hypertension among the 30 NHANES locations, $\rho = 0.0108$, find the design effect and effective sample size of (a) a two-stage sample of 30 locations with 20 adults each, and (b) the 5,182 adults, with an average of 172.7 per location, considering clustering alone.

(a) $1 + 19 \times 0.0108 = 1.205$, so 600 adults are worth about $600/1.205 = 498$ independent ones. (b) Treating the 30 locations as equal clusters of the mean size, $1 + 171.7 \times 0.0108 = 2.85$, so in an equal-probability design the 5,182 adults would carry, for hypertension and through clustering alone, about the information of 1,816 independent adults (with the effective size $m^* = 176.0$ for unequal clusters, Clustered Standard Errors, the factor is 2.89). This counts the clustering only, and it does not combine with the other features of the real design by simple multiplication: the design effect of NHANES's weighted prevalence is 3.62 (Data Collection and Sampling), less than the product 6.6 of this factor and Kish's weighting factor 2.33 from Survey Weights and the Horvitz–Thompson Estimator. It is Kish's factor that does not carry over: it assumes weights unrelated to the outcome. The unweighted mean under the same strata and locations has a design effect of 3.27 (Corollary 1 with linearized values), close to the clustering factor, and the weights multiply it only by 1.11. All these figures rest on 30 locations in 15 strata and are rough.

Summary

Design Estimator of $\bar Y$ Variance Where
Simple random $\bar y$ $(1-f)S^2/n$; $E[s^2] = S^2$ Theorems 1, 2
Stratified $\sum_h P_h\bar y_h$ $\sum_h P_h^2(1-f_h)S_h^2/n_h$ Theorem 3
Proportional vs. simple random gain $\propto \sum_h N_h(\bar Y_h - \bar Y)^2 - \frac1N\sum_h(N-N_h)S_h^2$ Theorem 4
Neyman allocation $n_h \propto N_hS_h$ ($\div\sqrt{c_h}$ with costs) gain over proportional $\frac1n\sum_hP_h(S_h - \bar S)^2$ Theorem 5
One-stage cluster mean of cluster means $(1-f)S_b^2/n$; design effect $\frac{NM-1}{M(N-1)}[1 + (M-1)\rho]$ Theorem 6
Two-stage $\hat t/N_U$ or ratio $\hat R$ between plus within PSU terms; $\approx \frac{S^2}{nm}[1 + (m-1)\rho]$ Theorem 7
Variance estimation PSU totals, with replacement unbiased with replacement; overstates by $f_1N^2S_t^2/n$ without Theorem 8

What we have learned

Design-based inference needs no model of the data: every variance on this page follows from the inclusion indicators of the design and the analysis of variance of the fixed population. Stratification removes the between-strata variation and gains most when strata differ in the outcome; Neyman allocation adds a further gain equal to the variance of the strata standard deviations. Clustering costs precision by the factor $1 + (m-1)\rho$, which a correlation of one percent and some 170 adults per location would turn into a near tripling of the variance in an equal-probability design, and which the survey accepts because it cannot afford more locations; in NHANES's weighted, stratified design the unweighted mean's design effect for systolic pressure, 2.86, is close to the clustering factor, and the weights raise it to 3.45, far less than Kish's factor would suggest, since that factor assumes weights unrelated to the outcome. The variance estimator that NHANES analysts use treats its 30 locations as drawn with replacement within 15 strata, a choice that is exactly unbiased for a with-replacement design and, when PSUs are drawn by simple random sampling, conservative by $f_1N^2S_t^2/n$ (Theorem 8(b)); for NHANES, whose first-stage sampling fraction is small, NCHS treats the difference as negligible.

The designs here all gave equal probabilities within a stratum. NHANES does not: it samples some groups at several times the rate of others. How estimates are corrected for that, and why the corrected estimate is the one that reproduces published national figures, is the subject of Survey Weights and the Horvitz–Thompson Estimator.

References