The course pages that use NHANES (the National Health and Nutrition Examination Survey of 2017 to 2018) all say the same thing in one form or another: the survey's sampling weights are set aside, so the figures describe the examined adults and not the United States (Rules of Probability, Sensitivity and Specificity, The Sampling Distribution of the Mean and others). Among the 5,107 adults of the probability unit, 15.21% have diagnosed diabetes. Weighted by the examination weight that NCHS releases with each participant, the share is 11.01%. Which is right, why do they differ by four points, and what exactly does a weight do? Both are right answers to different questions. The first describes the adults who were examined; the second estimates the share among US adults. They differ because NHANES samples some groups, including groups with more diabetes, at higher rates than others, and a weight undoes that by counting each participant as many times as the people the participant stands for.
A survey weight is a number attached to each respondent that says how many people of the population the respondent stands for. Its starting point is the inverse of the respondent's probability of being in the sample. The estimator that weights by inverse inclusion probabilities is the Horvitz–Thompson estimator (Horvitz and Thompson, 1952), and it is unbiased under every probability design, whatever the values and whatever the probabilities, provided no unit had probability zero. This article proves that, derives its variance, and then follows the weight through the adjustments that turn it into NHANES's WTMEC2YR.
The route is as follows. First the Horvitz–Thompson estimator, its unbiasedness and its variance, in the general form and in the Sen–Yates–Grundy form for samples of fixed size. Then the Hájek estimator, which divides by the estimated population size: its exact error identity gives the linearized variance that survey software reports. Next, how NHANES builds its weights, and what Kish's effective sample size does and does not measure. Then the weights used at last, on the quantities the course pages computed unweighted, and a Monte Carlo check on a pseudo-population built from NHANES. Finally, regression: when the weights change the answer, and when they only add variance. The article assumes Survey Sampling Designs for inclusion indicators, simple random and stratified sampling and the with-replacement variance estimator, and Data Collection and Sampling, which first used NHANES's weights and design variables to reproduce a published prevalence.
The Horvitz–Thompson estimator
The notation is that of Survey Sampling Designs: a finite population $U = \{1, \ldots, N\}$ with fixed values $y_k$ and total $t = \sum_{k \in U}y_k$; a random sample $s$ chosen by a design that gives each subset its probability $p(s)$; inclusion indicators $I_k$, inclusion probabilities $\pi_k = P(k \in s)$ and joint inclusion probabilities $\pi_{kl} = P(k, l \in s)$, with $\pi_{kk} = \pi_k$. We write $\Delta_{kl} = \pi_{kl} - \pi_k\pi_l = \operatorname{Cov}(I_k, I_l)$, so that $\Delta_{kk} = \pi_k(1 - \pi_k)$, and $\check y_k = y_k/\pi_k$ for a value expanded by its inverse probability. The design weight of unit $k$ is $d_k = 1/\pi_k$.
Here is the idea in one sentence. A unit that had probability $\pi_k$ of being sampled is, on average over samples, present a fraction $\pi_k$ of the time, so counting it $1/\pi_k$ times when it is present counts it exactly once on average.
Theorem 1 (unbiasedness). If $\pi_k > 0$ for every unit $k$, the Horvitz–Thompson estimator
$$\hat t_{\mathrm{HT}} = \sum_{k \in s}\frac{y_k}{\pi_k} = \sum_{k \in U} I_k\,\check y_k$$
is unbiased for $t$, under any design. If some units have $\pi_k = 0$, its bias is $-\sum_{k:\,\pi_k = 0}y_k$.
Proof. Units with $\pi_k = 0$ are never in the sample, so they never appear in the sum; for the others, $E[\hat t_{\mathrm{HT}}] = \sum_{k:\,\pi_k > 0} E[I_k]\,y_k/\pi_k = \sum_{k:\,\pi_k > 0} y_k$. That is $t$ when every $\pi_k$ is positive and $t - \sum_{k:\,\pi_k = 0}y_k$ otherwise. $\blacksquare$
The second part is the formal version of undercoverage. People outside NHANES's frame (in prisons, nursing homes or military barracks, as Data Collection and Sampling lists) have probability zero, and no weighting brings them back.
Theorem 2 (variance). Under any design,
$$\operatorname{Var}(\hat t_{\mathrm{HT}}) = \sum_{k \in U}\sum_{l \in U}\Delta_{kl}\,\check y_k\check y_l,$$
and if every $\pi_{kl} > 0$, the estimator $\hat V_{\mathrm{HT}} = \sum_{k \in s}\sum_{l \in s}\frac{\Delta_{kl}}{\pi_{kl}}\check y_k\check y_l$ is unbiased for it.
Proof. $\hat t_{\mathrm{HT}} = \sum_k I_k\check y_k$ is a linear combination of the indicators with fixed coefficients, so its variance is $\sum_k\sum_l\operatorname{Cov}(I_k, I_l)\check y_k\check y_l$ (Expectation, Variance and Covariance). For the estimator, write it as $\sum_{k,l \in U}I_kI_l\,\frac{\Delta_{kl}}{\pi_{kl}}\check y_k\check y_l$; since $E[I_kI_l] = \pi_{kl}$ (and $I_k^2 = I_k$), its expectation is $\sum_{k,l}\Delta_{kl}\check y_k\check y_l$. $\blacksquare$
Simple random sampling is the special case $\pi_k = n/N$: then $\hat t_{\mathrm{HT}} = N\bar y$, and Theorem 2 with Lemma 1 of Survey Sampling Designs gives back $N^2(1-f)S^2/n$.
Samples of fixed size: the Sen–Yates–Grundy form
Most designs fix the sample size, and then the variance takes a more revealing form. The plan: the sample size $n = \sum_l I_l$ is a constant, so each indicator has zero covariance with it; that identity lets the variance be written as a sum over pairs of squared differences.
Theorem 3 (Sen–Yates–Grundy). If every sample has the same size $n$, then
$$\operatorname{Var}(\hat t_{\mathrm{HT}}) = -\frac12\sum_{k \ne l}\Delta_{kl}\,(\check y_k - \check y_l)^2,$$
and, if every $\pi_{kl} > 0$, the estimator $\hat V_{\mathrm{SYG}} = -\frac12\sum_{k \ne l \in s}\frac{\Delta_{kl}}{\pi_{kl}}(\check y_k - \check y_l)^2$ is unbiased for it. If moreover $\pi_{kl} \le \pi_k\pi_l$ for every pair, $\hat V_{\mathrm{SYG}}$ is never negative. The variance is zero if $y_k$ is proportional to $\pi_k$.
Proof. For each $k$, $\sum_{l \in U}\Delta_{kl} = \operatorname{Cov}(I_k, \sum_l I_l) = \operatorname{Cov}(I_k, n) = 0$, because $n$ is a constant. Expand the square over all ordered pairs $(k, l)$, including $k = l$, whose terms are zero anyway:
$$-\frac12\sum_{k,l}\Delta_{kl}(\check y_k - \check y_l)^2 = -\frac12\sum_k\check y_k^2\sum_l\Delta_{kl} - \frac12\sum_l\check y_l^2\sum_k\Delta_{kl} + \sum_{k,l}\Delta_{kl}\check y_k\check y_l.$$
The first two sums vanish by the identity, and the last is Theorem 2's variance. The estimator is unbiased because $E[I_kI_l] = \pi_{kl}$, as in Theorem 2. If $\pi_{kl} \le \pi_k\pi_l$, each $-\Delta_{kl}/\pi_{kl} \ge 0$ and every term of $\hat V_{\mathrm{SYG}}$ is non-negative. If $y_k = c\,\pi_k$ for all $k$, every $\check y_k = c$ and every difference is zero. $\blacksquare$
The estimator $\hat V_{\mathrm{SYG}}$ is due to Sen (1953) and Yates and Grundy (1953), as Tillé and Wilhelm (2017) attribute it; Narain (1951) and Horvitz and Thompson (1952) introduced the Horvitz–Thompson estimator itself. The last clause of Theorem 3 is the logic of sampling with probability proportional to size: if inclusion probabilities track the values, every sample gives nearly the same expanded values and the estimator nearly the true total. The two general estimators of variance can behave very differently. Worked problem 1 has a design on four units in which $\hat V_{\mathrm{HT}}$ is negative on two of the six samples while $\hat V_{\mathrm{SYG}}$ never is.
Stratified sampling as a check. Under stratified simple random sampling, $\pi_{kl} = n_h(n_h-1)/[N_h(N_h - 1)]$ for two units of stratum $h$ and $\pi_k\pi_l$ for units of different strata, so $\Delta_{kl} = 0$ across strata, and within a stratum Theorem 3 reduces to Theorem 3 of Survey Sampling Designs. The article's script computes $\hat V_{\mathrm{SYG}}$ pair by pair on a stratified sample of 999 and confirms that it equals the stratified variance estimator $\sum_h N_h^2(1 - f_h)s_h^2/n_h$ to rounding error.
Poisson sampling
In Poisson sampling each unit is included independently with its own probability $\pi_k$, so the sample size is random and $\pi_{kl} = \pi_k\pi_l$ for $k \ne l$. Theorem 2 then has only diagonal terms:
$$\operatorname{Var}(\hat t_{\mathrm{HT}}) = \sum_{k \in U}\frac{1 - \pi_k}{\pi_k}\,y_k^2.$$
The variance depends on the $y_k^2$, not only on the spread of the $y_k$: adding a constant to every value changes it (for positive values, adding a positive constant increases it), so the estimator is not shift-equivariant, unlike the Hájek estimator below (Theorem 4(a)). The estimator does not know how many units it drew, and a sample that happens to be large gives a large total. That defect is what the next estimator repairs. Poisson sampling is also the design of a randomized experiment in which each unit is treated independently with probability $e(X_k)$, and the inverse-probability-weighted estimator of Matching and Propensity Scores is Theorem 1 applied to that design.
The Hájek estimator
To estimate a mean rather than a total, $\hat t_{\mathrm{HT}}/N$ is unbiased. But the same weights also estimate the population size, $\hat N = \sum_{k \in s}d_k$, which is itself a Horvitz–Thompson estimator (of the total of $y_k = 1$), and dividing by it gives the Hájek estimator,
$$\bar y_{\mathrm H} = \frac{\sum_{k \in s}d_ky_k}{\sum_{k \in s}d_k} = \frac{\hat t_{\mathrm{HT}}}{\hat N}.$$
This is the weighted mean that every survey package computes, the $\hat p_w$ of Data Collection and Sampling. The idea: dividing by $\hat N$ cancels the part of the error that comes from drawing more or fewer weighted units than expected. The next theorem makes that exact.
Theorem 4 (the Hájek estimator). Let $e_k = y_k - \bar Y$ and $\hat t_e = \sum_{k \in s}d_ke_k$, the Horvitz–Thompson estimator of $\sum_U e_k = 0$. Then, exactly, on every sample with $\hat N > 0$,
$$\bar y_{\mathrm H} - \bar Y = \frac{\hat t_e}{\hat N}.$$
Consequently: (a) adding a constant $c$ to every $y_k$ adds exactly $c$ to $\bar y_{\mathrm H}$; (b) if $\hat N = N$ on every sample, $\bar y_{\mathrm H} = \hat t_{\mathrm{HT}}/N$ and is exactly unbiased; (c) if $\hat N \ge cN$ on every sample for some $c > 0$, the bias is bounded by a product of two standard errors,
$$\big|E[\bar y_{\mathrm H}] - \bar Y\big| \le \frac1c\,\sqrt{\operatorname{Var}\Big(\frac{\hat t_e}{N}\Big)\operatorname{Var}\Big(\frac{\hat N}{N}\Big)};$$
(d) under the condition of (c), the remainder of the first-order approximation $\bar y_{\mathrm H} - \bar Y \approx \hat t_e/N$ is bounded in mean absolute value by the same product, $E\big|(\bar y_{\mathrm H} - \bar Y) - \hat t_e/N\big| \le \frac1c\sqrt{\operatorname{Var}(\hat t_e/N)\operatorname{Var}(\hat N/N)}$. As a heuristic (made precise in asymptotic frameworks that this page does not develop), the variance of $\bar y_{\mathrm H}$ is then approximated by that of $\hat t_e/N$, the linearized variance
$$\operatorname{AV}(\bar y_{\mathrm H}) = \frac{1}{N^2}\sum_{k,l \in U}\Delta_{kl}\,\check e_k\check e_l,$$
Theorem 2 applied to the residuals $e_k$ instead of the values.
Proof. $\hat t_{\mathrm{HT}} - \bar Y\hat N = \sum_{k \in s}d_k(y_k - \bar Y) = \hat t_e$; dividing by $\hat N$ gives the identity. (a) Adding $c$ to every value leaves the $e_k$ unchanged and adds $c$ to $\bar Y$. (b) If $\hat N = N$, the identity reads $\bar y_{\mathrm H} - \bar Y = \hat t_e/N$, whose expectation is 0 by Theorem 1. (c) Write $A = \hat t_e/N$ and $B = \hat N/N$, so that $\bar y_{\mathrm H} - \bar Y = A/B$, $E[A] = 0$ and $E[B] = 1$ by Theorem 1. Then $A/B = A - A(B-1)/B$, so $E[\bar y_{\mathrm H}] - \bar Y = -E[A(B-1)/B]$. Since $B \ge c$, $|A(B-1)/B| \le |A|\,|B-1|/c$, and the Cauchy–Schwarz inequality gives $E|A(B-1)| \le \sqrt{E[A^2]\,E[(B-1)^2]} = \sqrt{\operatorname{Var}(A)\operatorname{Var}(B)}$. (d) The remainder is $A/B - A = -A(B-1)/B$, and the bound on its mean absolute value is the one just proved in (c). The variance of $A$ is Theorem 2 applied to the $e_k$, divided by $N^2$. $\blacksquare$
Under Poisson sampling the sample can be empty, so $\hat N = 0$ has positive probability, $E[\bar y_{\mathrm H}]$ is not defined, and (c) and (d) do not apply as stated. They apply conditionally on $\hat N \ge cN$, an event whose probability is essentially 1 with inclusion probabilities like those below (the expected sample size is about 1,000), and the Monte Carlo below checks the approximation directly.
Part (c) is the precise sense in which the Hájek estimator is "approximately unbiased": its bias is of the order of the product of two standard errors, roughly $1/n$, while its standard error is of order $1/\sqrt n$. Part (b) covers stratified simple random sampling with any allocation, where $\hat N = \sum_h n_h\cdot N_h/n_h = N$.
Estimating the linearized variance. Replace the unknown $\bar Y$ by $\bar y_{\mathrm H}$ and $N$ by $\hat N$. For a design with known $\pi_{kl}$ this gives $\frac{1}{\hat N^2}\sum_{k,l \in s}\frac{\Delta_{kl}}{\pi_{kl}}\check e_k\check e_l$ with $e_k = y_k - \bar y_{\mathrm H}$. For NHANES, whose joint probabilities are not published, the estimated total $\hat t_e/\hat N = \sum_{k \in s} z_k$ with $z_k = d_k(y_k - \bar y_{\mathrm H})/\hat N$ is fed to the with-replacement estimator of PSU totals within strata, Theorem 8 and its Corollary 1 of Survey Sampling Designs; NCHS's variance tutorial justifies treating the PSUs as drawn with replacement by the small first-stage sampling fraction (National Center for Health Statistics, n.d.-b). That is exactly the Taylor-linearization formula of Data Collection and Sampling, which that page used without proof: Theorem 4 says why the linearized values are $d_k(y_k - \bar y_{\mathrm H})/\hat N$, and Theorem 8 with Corollary 1 why their PSU totals within strata estimate the variance.
Example (the Hájek estimator is not always better). For designs with probabilities proportional to a size measure, the Hájek estimator usually wins when the values vary little about their mean, when the sample size is random, or when the inclusion probabilities are weakly or negatively related to the values (cases listed by Särndal, Swensson and Wretman, 1992, p. 182, as summarized in the documentation of the R package sampling; Tillé and Matei, 2025). When the values are nearly proportional to the probabilities, the Horvitz–Thompson estimator has almost no variance (Theorem 3), and dividing by a random $\hat N$ adds some. Worked problem 3 gives a four-unit design in which the Hájek estimator's mean squared error is more than four times the Horvitz–Thompson estimator's variance.
How NHANES builds its weights
A design weight $d_k = 1/\pi_k$ would make NHANES's estimates unbiased if everyone selected took part. They do not: in 2017 to 2018 about half of the selected persons completed the examination (Data Collection and Sampling gives the response rates). The examination weight WTMEC2YR is built in three steps at each of the survey's three stages (screening, interview and examination), each step multiplying the weight by a factor (Chen et al., 2020):
- Base weights. The initial base weight of a participant is "equal to the inverse of the sampling rate" within the participant's sampling domain (the oversampling groups by race and Hispanic origin, income, sex and age), adjusted for the share of dwelling units released for screening in the location, for increases in the dwelling-unit sample, and for the number of years in the sample: the 4-year base weights were multiplied by 2 for a 2-year cycle. The base weight of the interview stage is the final screening weight, and the base weight of the examination stage is the final interview weight.
- Nonresponse adjustment. Within adjustment cells, the weights of respondents are multiplied by the sum of the base weights of everyone in the cell divided by the sum of the base weights of the respondents. Cells are the segments at the screening stage, and at the interview and examination stages are formed from variables found by a classification program to be related to response propensity, separately for five age groups; cells with fewer than 30 participants were collapsed. For 2017 to 2018 the largest nonresponse factor was 3, and most were below 2.
- Trimming and calibration. Weights above four times the mean of their sampling domain were cut to that threshold in 2017 to 2018, and the other weights of the domain raised to keep its total. Then the weights were calibrated: multiplied, cell by cell, by the ratio of a population control total to the sum of the weights, so that they add up to the population totals of the American Community Survey. For 2017 to 2018 the interview and examination weights were raked (calibrated iteratively to several sets of totals) to totals by race and Hispanic origin, age and sex; by race and Hispanic origin, sex and education for adults; and by deciles of the mean household income of the census tract.
The final weight is the product of all these factors. What does each step do in the terms of this article?
Nonresponse as a second phase of sampling. Suppose that within an adjustment cell $c$, each selected person responds independently with the same probability $\phi_c$, whatever their $y$. Then response is a Poisson sample drawn from the sample, with inclusion probability $\phi_c$, and a respondent's overall probability of being in the data is $\pi_k\phi_c$. The adjustment factor $\sum_{k \in s_c}d_k / \sum_{k \in r_c}d_k$, where $s_c$ and $r_c$ are the selected persons and the respondents of the cell, is the ratio of a total to its Horvitz–Thompson estimate under the response phase: given the sample, $E[\sum_{k \in r_c}d_k] = \phi_c\sum_{k \in s_c}d_k$ by Theorem 1, so the factor estimates $1/\phi_c$, and the adjusted weight estimates $1/(\pi_k\phi_c)$. The adjusted estimator of the cell's total is $\hat N_c$ times the Hájek mean of the respondents, approximately unbiased by Theorem 4 applied to the response phase. Everything rests on the assumption that response does not depend on $y$ within cells, which the respondents' data cannot test. The design report states the condition: "Nonresponse adjustment reduces bias if response rates and survey characteristics vary from cell to cell and if participants and nonrespondents sharing the same characteristics are in the same cell" (Chen et al., 2020).
Calibration as post-stratification. Calibration to one set of cells is post-stratification. If the population counts $N_c$ of cells $c$ are known and $\hat N_c = \sum_{k \in s_c}d_k$, the post-stratified mean is $\bar y_{\mathrm{ps}} = \sum_c \frac{N_c}{N}\bar y_{\mathrm H, c}$, where $\bar y_{\mathrm H,c}$ is the Hájek mean within cell $c$. By Theorem 4 within each cell, $\bar y_{\mathrm H,c} - \bar Y_c \approx \hat t_{e,c}/N_c$ with $e_k = y_k - \bar Y_c$ for units of cell $c$, so
$$\bar y_{\mathrm{ps}} - \bar Y \approx \frac1N\sum_c\hat t_{e,c} = \frac1N\sum_{k \in s}d_k\big(y_k - \bar Y_{c(k)}\big).$$
The linearized variable is the deviation from the cell mean, not from the overall mean. Calibration therefore removes the variation between cells from the variance, as stratification does (Survey Sampling Designs, Theorem 4), and it removes the bias from any under- or overcoverage that is uniform within cells. The design report names its assumption: calibration assumes that persons missed by the survey "have the same distribution of characteristics as surveyed persons within the calibration cells," which it calls "obviously an oversimplification" (Chen et al., 2020). Raking repeats this adjustment for each set of totals in turn until all are matched.
Trimming trades a small bias for a smaller variance. The report notes that "trimming sample weights may introduce estimation bias," which is why it was applied only where inspection of a domain's weights called for it (Chen et al., 2020).
The result is a weight that the analytic guidelines describe as one that "can be considered as measures of the number of persons represented by the particular survey respondent" (National Center for Health Statistics, 2018, §3.1). The weights of the 5,182 adults of Data Collection and Sampling sum to 233,922,973, and range from 4,580 to 419,763. The NHANES weighting tutorial adds a rule: use the weight of the smallest subpopulation that contains all the variables in the analysis, which for blood pressure, body measures and anything combined with them is the examination weight (National Center for Health Statistics, n.d.-a).
Kish's effective sample size
Unequal weights cost precision even when they are right. Suppose, as a model, that the values $y_k$ of the $n$ respondents are uncorrelated with a common variance $\sigma^2$ and that the weights $w_k$ are fixed numbers unrelated to them. Then
$$\operatorname{Var}\Big(\frac{\sum_k w_ky_k}{\sum_k w_k}\Big) = \sigma^2\,\frac{\sum_k w_k^2}{(\sum_k w_k)^2} = \frac{\sigma^2}{n_{\mathrm{eff}}}, \qquad n_{\mathrm{eff}} = \frac{(\sum_k w_k)^2}{\sum_k w_k^2} = \frac{n}{1 + \mathrm{CV}^2},$$
where CV is the coefficient of variation of the weights (standard deviation with divisor $n$ over the mean); the first equality is the variance of a linear combination of uncorrelated variables, and Data Collection and Sampling proves the identity $n\sum w_k^2/(\sum w_k)^2 = 1 + \mathrm{CV}^2$. The factor $1 + \mathrm{CV}^2$ is Kish's design effect for weighting (Kish, 1965, as cited by Sarig, Galili and Eilat, 2023, who state it under the same assumption of weights unrelated to the outcome), and $n_{\mathrm{eff}}$ is the effective sample size. For the 5,182 adults the coefficient of variation is 1.153, the factor 2.33, and $n_{\mathrm{eff}} = 2{,}225$; for the 5,107 adults of the probability unit the factor is also 2.33 and $n_{\mathrm{eff}} = 2{,}190$.
The formula is a model-based summary, not a design-based variance. It ignores clustering, it ignores stratification, and it assumes that the weights carry no information about $y$. Each assumption fails somewhere below: clustering makes real design effects larger than 2.33, and stratification or a relation between weights and outcome can make them smaller.
The weights used at last
Data. The 5,182 adults (18 or older, not pregnant, with a systolic and a diastolic reading) of Data Collection and Sampling for blood pressure and hypertension; the 5,107 of them who also have a body mass index, an answer to the diabetes question and an answer on smoking, the population of Rules of Probability and the Generalized Linear Models unit, for the rest. Each estimate uses WTMEC2YR (National Center for Health Statistics, 2020) and is computed as a domain estimate: every examined person stays in the data with the domain indicator set to zero outside the domain, so that all 15 pseudo-strata and 30 pseudo-PSUs enter the variance, as the guidelines require (National Center for Health Statistics, 2018, §3.2.3.1). Standard errors are Taylor-linearized (Theorem 4 with Theorem 8 and Corollary 1 of Survey Sampling Designs); the delete-one-PSU jackknife agreed with them to within 5% in every row, as the script asserts (for systolic pressure both give 0.463). The design effect is the design-based variance divided by $\hat S_w^2/n$, the variance of a simple random sample of the same size, with $\hat S_w^2 = \sum_k w_k(y_k - \bar y_w)^2/\sum_k w_k$ the weighted estimate of the population variance. Intervals use $t_{15,\,0.975} = 2.1314$, since 30 PSUs in 15 strata give 15 degrees of freedom.
| Quantity | Unweighted | Weighted | Design SE | 95% interval | Design effect | Kish factor |
|---|---|---|---|---|---|---|
| Hypertension (5,182) | 0.5475 | 0.4822 | 0.0132 | 0.4540 to 0.5103 | 3.62 | 2.33 |
| Mean systolic pressure, mm Hg (5,182) | 126.48 | 123.47 | 0.463 | 122.48 to 124.45 | 3.45 | 2.33 |
| Mean body mass index, kg/m² (5,107) | 29.65 | 29.67 | 0.291 | 29.05 to 30.29 | 8.15 | 2.33 |
| Obesity, BMI 30 or more (5,107) | 0.4110 | 0.4195 | 0.0198 | 0.3774 to 0.4617 | 8.20 | 2.33 |
| Diagnosed diabetes (5,107) | 0.1521 | 0.1101 | 0.0055 | 0.0983 to 0.1218 | 1.59 | 2.33 |
| Current smoking (5,107) | 0.1776 | 0.1702 | 0.0120 | 0.1445 to 0.1958 | 5.24 | 2.33 |
| Never smoked (5,107) | 0.5910 | 0.5831 | 0.0170 | 0.5470 to 0.6193 | 6.06 | 2.33 |
The first two rows repeat Data Collection and Sampling, whose weighted hypertension prevalence reproduces the 48.2% published by NCHS. Three things stand out in the rest.
Weighting moves some estimates and not others. Hypertension falls by 6.5 points, diagnosed diabetes by 4.2 and systolic pressure by 3.0 mm Hg, consistent with the sample being older than the population it represents (Data Collection and Sampling compares the two compositions), since all three rise with age. Body mass index and obesity barely move. The direction and size of the change depend on how the outcome relates to the sampling rates; a variable unrelated to them is estimated without bias by the unweighted mean too. The weights cannot know in advance which variables those are, which is why they are always used for population estimates.
Design effects vary from 1.59 to 8.20 for the same people and the same weights. Kish's factor is 2.33 for every row, because it depends only on the weights. The design-based values differ because clustering does: body mass index and obesity have design effects above 8, so the 5,107 adults estimate national obesity about as precisely as 623 adults drawn at random would. That suggests the 30 locations differ more in body mass than in blood pressure. For hypertension, the ratio of the design effect to Kish's factor is 1.55. That is a ratio, not a measured component: the factors of a weighted, stratified, clustered design do not simply multiply, and the clustering factor $1 + (m-1)\rho$ of an equal-probability design with these locations would be about 2.85 (Survey Sampling Designs), not 1.55. Without the weights, the same strata and locations give the unweighted prevalence a design effect of 3.27, close to that clustering factor, so the weights add only a factor of about 1.11 for this variable, far less than Kish's 2.33. Diabetes has a design effect of 1.59, below Kish's factor, so Kish's assumptions do not hold for it: stratification, or a relation between the weights and the outcome, recovers more precision than the clustering costs. The guidelines record the same pattern in general terms: NHANES design effects are "typically greater than 1" and "differ among variables" (National Center for Health Statistics, 2018, §3.2).
The naive standard error is too small. The design-based standard error exceeds the one computed from the unweighted data as if it were a simple random sample by factors from 1.10 (diabetes) to 2.87 (obesity) in the table's rows. For systolic pressure the unweighted mean's standard error, computed as if the adults were a simple random sample, is 0.279 mm Hg, against 0.463 for the weighted mean under the design. The course pages did not claim national estimates, so their standard errors were correct for their question (samples drawn from the 5,182 as a population); for a national question they would not be.
A relative risk, weighted
The probability unit's running example compares diabetes among former and never smokers: 271/1,182 against 394/3,018 unweighted, a relative risk of 1.756 (Intervals and Adjustment for Relative Risk and Odds Ratio). Weighted, the two risks are 0.1621 and 0.0958, and the relative risk is 1.692. Its standard error on the logarithm of the relative risk, by the delete-one-PSU jackknife (the same replicate weights as in Data Collection and Sampling, recomputing both risks on each replicate), is 0.119, against 0.071 for the binomial formula used unweighted. The 95% interval, $\exp(\ln 1.692 \pm 2.1314 \times 0.119)$, runs from 1.31 to 2.18. The weighted and unweighted relative risks are close here; the weighted interval is much wider. Age confounds this comparison, as Confounding and Causal Diagrams discusses; weighting makes the estimate national, not causal.
A Monte Carlo check on a pseudo-population
The design-based standard errors above cannot be checked against the truth, because the population of US adults is not available. A pseudo-population built from the weights can at least check the formulas behind them, under designs with known inclusion probabilities. It does not exercise clustering, calibration or the with-replacement estimator over masked units that the NHANES table relies on. Each of the 5,182 adults was copied $\mathrm{round}(w_k/2{,}000)$ times, a population of 116,969 whose composition is the weighted composition of the survey: its hypertension prevalence is 0.4822 and its mean systolic pressure 123.46.
From it, samples of 999 adults were drawn by strata defined by race and Hispanic origin (six groups) and age group (three), 18 strata in all, with $n_h$ proportional to the number of NHANES participants in each stratum. The sample therefore over-represents the same groups NHANES over-represents; the highest sampling rate is 8.0 times the lowest. Two designs used these inclusion probabilities: stratified simple random sampling, of fixed size, and Poisson sampling, with random size. For each of 30 seeds (0 to 29), 2,000 samples were drawn; the figures are medians over the seeds with interquartile ranges.
The unweighted mean is biased. Its average error was +0.0556 for hypertension (IQR 0.0555 to 0.0559) and +2.46 mm Hg for systolic pressure (IQR 2.46 to 2.47), close to the gaps between the unweighted and weighted NHANES estimates.
Stratified sampling. Here $\hat N = N$, so the Hájek and Horvitz–Thompson estimators coincide (Theorem 4(b)). The standard error by Theorem 3 of Survey Sampling Designs is 0.0172 for hypertension and 0.582 for systolic pressure; the Monte Carlo values over the stratified samples were 0.0171 (IQR 0.0170 to 0.0174) and 0.579 (IQR 0.575 to 0.585). Nominal 95% intervals from the estimated variance covered the true value in 0.949 (IQR 0.947 to 0.952) and 0.949 (IQR 0.946 to 0.954) of samples. Kish's factor for these weights is 1.43, but the true design effects are 1.19 for hypertension and 1.06 for systolic pressure: the strata include age groups, which differ in both outcomes, and the stratification gain offsets most of the weighting loss.
Poisson sampling. With the same inclusion probabilities but independent inclusion, the Horvitz–Thompson estimator of the mean has standard error 0.0255 for hypertension and 4.659 mm Hg for systolic pressure by the Poisson formula; the Hájek estimator's linearized standard errors are 0.0188 and 0.638. The Monte Carlo values were 0.0256 (IQR 0.0252 to 0.0257) and 4.653 (IQR 4.594 to 4.718) for Horvitz–Thompson, 0.0188 (IQR 0.0186 to 0.0192) and 0.639 (IQR 0.630 to 0.644) for Hájek. For systolic pressure the Horvitz–Thompson estimator has a standard error seven times as large (a variance about 53 times as large), because its variance grows with $y_k^2$ and pressures are far from zero. Intervals from the estimated linearized variance covered in 0.950 (IQR 0.947 to 0.952) of samples for systolic pressure and 0.947 (IQR 0.943 to 0.952) for hypertension; the Horvitz–Thompson intervals for systolic pressure also covered in 0.950 (IQR 0.946 to 0.952), but they are about seven times as wide.
When weights matter for regression
For a mean or a prevalence the case for weights is settled: without them the estimate describes the sample, not the population. For a regression coefficient the case is weaker, and Solon, Haider and Wooldridge (2015) set out when it holds. They distinguish estimating population descriptive statistics, where "weighting is called for when it is needed to make the analysis sample representative of the target population," from estimating causal effects, where they find three possible motives for weighting (precision under heteroskedasticity, consistency under endogenous sampling, and identification of average effects under unmodeled heterogeneity) and conclude that each "sometimes does not apply in situations where practitioners often assume it does" (quotations from the working-paper version, 2013). Two results make their argument precise in the design-based setting of this article.
Write the regression of $y$ on a vector of regressors $\mathbf x_k$ (with a 1 for the intercept). The census coefficient is the least squares fit to the whole population, $\boldsymbol\beta_U = (\sum_U\mathbf x_k\mathbf x_k^{\mathsf T})^{-1}\sum_U\mathbf x_ky_k$, with census residuals $e_k = y_k - \mathbf x_k^{\mathsf T}\boldsymbol\beta_U$. Weighted least squares on the sample, $\hat{\boldsymbol\beta}_w = (\sum_s d_k\mathbf x_k\mathbf x_k^{\mathsf T})^{-1}\sum_s d_k\mathbf x_ky_k$, replaces each population sum by its Horvitz–Thompson estimator. Unweighted least squares, $\hat{\boldsymbol\beta}_{\mathrm{OLS}}$, uses the plain sample sums.
Proposition 5 (what each fit estimates). (a) Each sum in $\hat{\boldsymbol\beta}_w$ is unbiased for the corresponding population sum, so $\hat{\boldsymbol\beta}_w$ estimates $\boldsymbol\beta_U$ with an error of the order of the Horvitz–Thompson errors, as the Hájek estimator does. (b) The plain sums satisfy $E[\sum_s\mathbf x_k\mathbf x_k^{\mathsf T}] = \sum_U\pi_k\mathbf x_k\mathbf x_k^{\mathsf T}$ and $E[\sum_s\mathbf x_ky_k] = \sum_U\pi_k\mathbf x_ky_k$, so $\hat{\boldsymbol\beta}_{\mathrm{OLS}}$ estimates the $\pi$-weighted census fit $\boldsymbol\beta_\pi = (\sum_U\pi_k\mathbf x_k\mathbf x_k^{\mathsf T})^{-1}\sum_U\pi_k\mathbf x_ky_k$, and
$$\boldsymbol\beta_\pi - \boldsymbol\beta_U = \Big(\sum_U\pi_k\mathbf x_k\mathbf x_k^{\mathsf T}\Big)^{-1}\sum_U\pi_k\mathbf x_ke_k.$$
The two targets coincide exactly when the census residuals are orthogonal to $\pi_k\mathbf x_k$.
Proof. (a) and the expectations in (b) are Theorem 1 applied to each entry, with $d_k\pi_k = 1$ in (a) and $E[I_k] = \pi_k$ in (b). For the difference, substitute $y_k = \mathbf x_k^{\mathsf T}\boldsymbol\beta_U + e_k$ into $\boldsymbol\beta_\pi$: $\sum_U\pi_k\mathbf x_ky_k = (\sum_U\pi_k\mathbf x_k\mathbf x_k^{\mathsf T})\boldsymbol\beta_U + \sum_U\pi_k\mathbf x_ke_k$, and multiply by the inverse. $\blacksquare$
When are the census residuals orthogonal to $\pi_k\mathbf x_k$? The census normal equations make them orthogonal to $\mathbf x_k$ itself. If the sampling rates are constant within cells and the model contains every cell indicator multiplied by every regressor, the orthogonality holds cell by cell and the two fits agree. Short of that, it holds approximately when the model describes the mean of $y$ well and the sampling depends only on the regressors, which is Solon, Haider and Wooldridge's exogenous sampling: they write that "the issue is whether the sampling is independent of the dependent variable conditional on the explanatory variables." In that case the unweighted fit is also the more precise one.
Proposition 6 (when the model is right, the unweighted fit is more precise). Suppose that in the population $y = \mathbf x^{\mathsf T}\boldsymbol\beta + \varepsilon$ with errors that, given $\mathbf x$, are uncorrelated, of mean 0 and of common variance $\sigma^2$, and that selection depends only on $\mathbf x$, so that the same holds in the sample given the sampled regressors $X$ and their weights $D$. Then $\hat{\boldsymbol\beta}_{\mathrm{OLS}}$ and $\hat{\boldsymbol\beta}_w$ are both unbiased for $\boldsymbol\beta$, and $\operatorname{Var}(\mathbf a^{\mathsf T}\hat{\boldsymbol\beta}_{\mathrm{OLS}}) \le \operatorname{Var}(\mathbf a^{\mathsf T}\hat{\boldsymbol\beta}_w)$ for every vector $\mathbf a$.
Proof. Conditional on the sampled regressors and weights, $\hat{\boldsymbol\beta}_w = (X^{\mathsf T}DX)^{-1}X^{\mathsf T}D\mathbf y$ is linear in $\mathbf y$, and $E[\hat{\boldsymbol\beta}_w] = (X^{\mathsf T}DX)^{-1}X^{\mathsf T}DX\boldsymbol\beta = \boldsymbol\beta$. The Gauss–Markov theorem (The Gauss–Markov Theorem) says that least squares has the smallest variance among linear unbiased estimators under these assumptions, for every linear combination, conditionally on $X$ and $D$. Both estimators have conditional mean $\boldsymbol\beta$, so by the law of total variance their unconditional variances are the expectations of the conditional ones, and the inequality survives the averaging. $\blacksquare$
The NHANES regression of systolic pressure on age. The Linear Models unit fitted systolic pressure on age in the 5,176 adults with a body mass index, unweighted, and found a slope of 0.5223 mm Hg per year (The Geometry of Least Squares). Among the 5,182 adults of this article, three models were fitted with and without weights. Standard errors are linearized: the regression coefficient is a function of weighted totals, and its linearized values are the rows of $(X^{\mathsf T}DX)^{-1}\mathbf x_kd_ke_k$, whose PSU totals enter Corollary 1 of Theorem 8 in Survey Sampling Designs. The contrast between the weighted and unweighted slopes, the diagnostic that DuMouchel and Duncan (1983) proposed and Solon, Haider and Wooldridge (2015) recommend reporting, has its standard error from the delete-one-PSU jackknife with both fits recomputed on each replicate.
- With age alone, the unweighted slope is 0.521 (design-based standard error 0.016; the naive one is 0.013) and the weighted slope 0.454 (standard error 0.016). The contrast is −0.067 with standard error 0.011, $z = -6.06$: the two fits estimate different things.
- Adding sex and race and Hispanic origin, two of the variables by which NHANES oversamples (income and age group, the others, are not in this model), the slopes are 0.524 (design standard error 0.014, naive 0.013) and 0.469 (0.017); the contrast is −0.056 with standard error 0.011, $z = -4.92$. Including some design variables did not make the weights irrelevant.
- Adding also the age group (18 to 39, 40 to 59, 60 and over), the slopes are 0.553 (design standard error 0.025, naive 0.038) and 0.562 (0.043); the contrast is 0.009 with standard error 0.036, $z = 0.26$.
The Monte Carlo on the pseudo-population shows what these contrasts mean, because there the census coefficients are known: 0.453, 0.469 and 0.562 for the three models. Proposition 5 predicts the bias of the unweighted fit from the population alone: $\boldsymbol\beta_\pi - \boldsymbol\beta_U$ is 0.054, 0.043 and 0.037 for the slope in the three models.
| Model | Unweighted: average error | Weighted: average error | SD unweighted | SD weighted |
|---|---|---|---|---|
| With age alone | 0.054 (0.053 to 0.055) | 0.000 (−0.001 to 0.001) | 0.029 (0.029 to 0.030) | 0.032 (0.031 to 0.032) |
| With sex and race | 0.043 (0.043 to 0.044) | 0.000 (−0.001 to 0.001) | 0.029 (0.029 to 0.030) | 0.031 (0.031 to 0.032) |
| With age group | 0.037 (0.035 to 0.040) | 0.000 (−0.003 to 0.003) | 0.089 (0.087 to 0.091) | 0.093 (0.091 to 0.095) |
| Model true (synthetic) | −0.001 (−0.003 to 0.002) | −0.002 (−0.003 to 0.001) | 0.080 (0.079 to 0.082) | 0.096 (0.094 to 0.097) |
The table gives medians over 30 seeds of 500 stratified samples each, with interquartile ranges in parentheses. In the first three rows the unweighted slope's error agrees to three decimals with what Proposition 5 predicts, and the weighted slope is on target. The design variables are in the models with sex and race and with age group, but the line in age is not the true shape of pressure against age, and the misfit is weighted differently by the sample and by the population.
In the last row the outcome was replaced by a synthetic one for which the third model is exactly true: the census fit of that model plus a fresh normal error with the census residual standard deviation, 15.79 mm Hg. The sampling depends only on the strata, which the model contains, so Proposition 6 applies. Both fits are unbiased, and the unweighted one has the smaller standard deviation, 0.080 against 0.096, a variance ratio of 1.44.
The real-data contrast for the third model, 0.009 with standard error 0.036, cannot tell these two situations apart: in the pseudo-population the unweighted fit of that model is biased by 0.037, about the size of the contrast's standard error. A contrast near zero is weak evidence that weights do not matter. The advice of Solon, Haider and Wooldridge, to report both fits and to treat a large contrast as a sign of misspecification or of sampling that depends on the outcome, uses the contrast in the direction in which it is informative. On the real data, a large contrast cannot by itself separate misspecification from selection and nonresponse adjustment on variables left out of the model: NHANES also oversamples by income, which no model here contains, and its weights are calibrated to education and tract income (Chen et al., 2020).
What is proved, what is cited, what the experiment shows
Proved on this page: the unbiasedness of the Horvitz–Thompson estimator under any design with positive inclusion probabilities, and its bias under undercoverage (Theorem 1); its variance and an unbiased variance estimator (Theorem 2); the Sen–Yates–Grundy form for fixed-size designs, its unbiased estimator, the condition for that estimator to be non-negative, and zero variance for values proportional to the probabilities (Theorem 3); the variance of Poisson sampling; the exact error identity of the Hájek estimator, its translation equivariance, its exact unbiasedness when $\hat N = N$, a bound on its bias and on the remainder of its linearization by a product of standard errors, and its linearized variance as a first-order approximation (Theorem 4), which justifies the Taylor-linearization formula used by Data Collection and Sampling; the reading of weighting-class nonresponse adjustment as a second phase of Poisson sampling and of calibration as post-stratification with linearized variable $y_k - \bar Y_{c(k)}$; Kish's factor under its model; the targets of weighted and unweighted least squares and the exact gap between them (Proposition 5); the greater precision of the unweighted fit when the model is right and sampling depends only on the regressors (Proposition 6, from the Gauss–Markov theorem).
Cited: the estimator's origins (Narain, 1951, and Horvitz and Thompson, 1952) and the variance estimator's (Sen, 1953; Yates and Grundy, 1953), as attributed by Tillé and Wilhelm (2017); the cases favoring the Hájek estimator (Särndal, Swensson and Wretman, 1992, through Tillé and Matei, 2025); Kish's design effect (Kish, 1965, through Sarig, Galili and Eilat, 2023); the construction of NHANES's weights, its variance units and its analytic rules (Chen et al., 2020; National Center for Health Statistics, 2018, n.d.-a, n.d.-b); the framework and recommendations of Solon, Haider and Wooldridge (2015) and the contrast of DuMouchel and Duncan (1983).
What the experiments show: on NHANES, weighting moved hypertension, diabetes and systolic pressure substantially and body mass index hardly at all; design effects ranged from 1.59 to 8.20 against a Kish factor of 2.33; the weighted relative risk of diabetes for former against never smokers was 1.692 (95% interval 1.31 to 2.18). On a pseudo-population of 116,969 built from the weights, for stratified and Poisson samples with known inclusion probabilities, every variance formula matched the Monte Carlo, linearized intervals covered at their nominal rate, the Horvitz–Thompson estimator under Poisson sampling had a standard error seven times the Hájek estimator's for systolic pressure, and the bias of unweighted regression slopes agreed with Proposition 5 to three decimals. These are observations about this survey, these variables and this pseudo-population.
Worked problems
These problems use small exact cases constructed for this article and the NHANES adults of the unit; every number is recomputed in the article's script, the small cases in exact arithmetic.
Problem 1 (a design on four units, enumerated). A population has four units with values 120, 135, 150 and 190. A design selects two units, with probabilities 0.1 for $\{1,2\}$, 0.1 for $\{1,3\}$, 0.2 for $\{1,4\}$, 0.1 for $\{2,3\}$, 0.2 for $\{2,4\}$ and 0.3 for $\{3,4\}$. Find the inclusion probabilities, verify Theorems 1 to 3 by enumeration, and evaluate both variance estimators on every sample.
Summing the probabilities of the samples that contain each unit gives $\pi_1 = 0.4$, $\pi_2 = 0.4$, $\pi_3 = 0.5$ and $\pi_4 = 0.7$; they sum to 2, the sample size. The joint probabilities are the sample probabilities themselves. The total is $t = 595$. The Horvitz–Thompson estimates on the six samples are 637.5, 600.0, 571.4, 637.5, 608.9 and 571.4; for example $\{1, 4\}$ gives $120/0.4 + 190/0.7 = 300 + 271.4 = 571.4$. Their average, weighted by the sample probabilities, is exactly 595 (Theorem 1), and their variance about 595 is 680.4, which both Theorem 2 and Theorem 3 reproduce exactly.
Every $\pi_{kl}$ is below $\pi_k\pi_l$ (for instance $\pi_{12} = 0.1 < 0.16$), so $\hat V_{\mathrm{SYG}}$ is never negative: on the six samples it is 843.8, 0.0, 326.5, 1,406.3, 1,746.2 and 136.1. It is 0 on $\{1, 3\}$ because $120/0.4 = 150/0.5 = 300$: the expanded values agree, as in the last clause of Theorem 3. The estimator $\hat V_{\mathrm{HT}}$ of Theorem 2 gives 843.8, −81,000.0, 10,959.2, −89,156.3, 17,160.1 and 39,959.2: negative on 2 of the 6 samples. Both average exactly to 680.4 over the design, but only one of them is usable.
Problem 2 (a weighted relative risk). From the NHANES figures above, compute the weighted relative risk of diabetes for former against never smokers and its 95% interval, and compare with the unweighted analysis.
The weighted risks are 0.1621 and 0.0958, so the relative risk is $0.1621/0.0958 = 1.692$ (the script divides unrounded values). With the jackknife standard error of its logarithm, 0.119, and $t_{15,\,0.975} = 2.1314$, the interval is $\exp(\ln 1.692 \pm 0.254)$, from 1.31 to 2.18. Unweighted, the relative risk is $(271/1{,}182)/(394/3{,}018) = 1.756$ with a binomial standard error of the logarithm of $\sqrt{1/271 - 1/1{,}182 + 1/394 - 1/3{,}018} = 0.071$. The point estimates differ by less than 4%; the weighted standard error is 1.7 times the unweighted, because the design's clustering and unequal weights are counted.
Problem 3 (a Hájek estimator that loses). For the design of Problem 1, find the exact expectation and mean squared error of the Hájek estimator of the mean, and compare with $\hat t_{\mathrm{HT}}/N$.
The population mean is $595/4 = 148.75$. The Hájek estimates on the six samples are 127.5, 133.3, 145.5, 141.7, 155.0 and 166.7; for example $\{1, 4\}$ gives $(300 + 271.4)/(1/0.4 + 1/0.7) = 145.5$. Their expectation is 150.34, a bias of 1.59, and their mean squared error is 180.2. The unbiased $\hat t_{\mathrm{HT}}/4$ has variance $680.4/16 = 42.5$. Here the Hájek estimator is more than four times worse, a ratio of 4.24. The values rise with the inclusion probabilities (the unit with $\pi = 0.7$ has the largest value), so the expanded values $\check y_k$ are nearly constant and the Horvitz–Thompson total is already stable, while $\hat N$ varies from 3.43 to 5.00 across samples; dividing by it adds variation instead of removing it. This is the case Theorem 3's last clause describes, and the opposite of the Poisson sampling of systolic pressure above.
Problem 4 (effective sample sizes). For the 5,182 adults, find Kish's effective sample size and the effective sample size implied by the design effect of hypertension, and explain the difference.
Kish's factor is $1 + 1.153^2 = 2.33$, so $n_{\mathrm{eff}} = 5{,}182/2.33 = 2{,}225$. The design effect of hypertension is 3.62, so the 5,182 adults estimate the national prevalence as precisely as $5{,}182/3.62 = 1{,}432$ adults drawn at random would. The difference is the clustering in 30 locations, which Kish's factor ignores: their ratio, 1.55, is the extra factor that clustering and stratification together contribute for this variable. NCHS uses the design-based version, the "actual sample size divided by the design effect," in its presentation standards for proportions, which require it to be at least 30 (National Center for Health Statistics, 2018, §3.3.1).
Summary
| Result | Statement | Where |
|---|---|---|
| Horvitz–Thompson | $\sum_s y_k/\pi_k$ unbiased for $t$ if all $\pi_k > 0$ | Theorem 1 |
| Its variance | $\sum_{k,l}\Delta_{kl}\check y_k\check y_l$; estimator with $\Delta_{kl}/\pi_{kl}$ | Theorem 2 |
| Fixed size | $-\frac12\sum_{k \ne l}\Delta_{kl}(\check y_k - \check y_l)^2$; zero if $y \propto \pi$ | Theorem 3 |
| Poisson sampling | $\sum_k(1-\pi_k)y_k^2/\pi_k$ | Theorem 2 |
| Hájek | $\bar y_{\mathrm H} - \bar Y = \hat t_e/\hat N$ exactly; bias $\le$ product of two SEs; linearized variance on $e_k = y_k - \bar Y$ | Theorem 4 |
| Nonresponse, calibration | second phase with response probability; linearized variable $y_k - \bar Y_{c(k)}$ | NHANES section |
| Kish | $n_{\mathrm{eff}} = (\sum w)^2/\sum w^2 = n/(1 + \mathrm{CV}^2)$, under a model | Kish section |
| Regression | unweighted targets $\boldsymbol\beta_\pi$, gap $(\sum\pi\mathbf x\mathbf x^{\mathsf T})^{-1}\sum\pi\mathbf xe$; more precise if the model is right | Propositions 5, 6 |
What we have learned
Weighting by inverse inclusion probabilities makes a total unbiased under any probability design, because each unit is then counted once on average; the price is a variance that depends on how the probabilities relate to the values, small when they track the values and large when they do not. Dividing by the estimated population size gives the Hájek estimator, whose error is exactly an estimated total of residuals divided by $\hat N$, and that identity is the whole content of the Taylor linearization that NHANES analyses rely on. NHANES's weights add nonresponse adjustment and calibration to the design weights; both can be read as further steps of the same Horvitz–Thompson logic, each resting on an assumption about the people not observed. Used at last on the course pages' quantities, the weights lowered hypertension by 6.5 points and diabetes by 4.2, left body mass index unchanged, and widened every interval, by design effects from 1.59 to 8.20. For regression they are needed when the sampling depends on the outcome or the model is wrong, and cost precision when neither holds; the weighted-minus-unweighted contrast is a useful but weak diagnostic.
Weighting by an inverse probability is also how observational studies estimate causal effects, with the probability of treatment in place of the probability of selection. Matching and Propensity Scores takes that step, and the unit's overview, Study Design, Sampling and Causal Inference, places both in its plan.
References
- Chen, T. C., Clark, J., Riddles, M. K., Mohadjer, L. K., and Fakhouri, T. H. I. (2020). National Health and Nutrition Examination Survey, 2015–2018: Sample Design and Estimation Procedures. Vital and Health Statistics, Series 2, No. 184. National Center for Health Statistics. https://www.cdc.gov/nchs/data/series/sr_02/sr02-184-508.pdf record · 3 pages
- DuMouchel, W. H., and Duncan, G. J. (1983). Using sample survey weights in multiple regression analyses of stratified samples. Journal of the American Statistical Association, 78(383), 535–543. https://doi.org/10.1080/01621459.1983.10478006 record
- Horvitz, D. G., and Thompson, D. J. (1952). A generalization of sampling without replacement from a finite universe. Journal of the American Statistical Association, 47(260), 663–685. https://doi.org/10.1080/01621459.1952.10483446 record · 3 pages
- Kish, L. (1965). Survey Sampling. Wiley, New York. record · 2 pages
- Narain, R. D. (1951). On sampling without replacement with varying probabilities. Journal of the Indian Society of Agricultural Statistics, 3, 169–174 (pages as Tillé and Wilhelm, 2017, give them; not opened). record
- National Center for Health Statistics (2018). National Health and Nutrition Examination Survey: Analytic Guidelines, 2011–2014 and 2015–2016. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/data/nhanes/analyticguidelines/11-16-analytic-guidelines.pdf record · 3 pages
- National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018: Demographic Variables and Sample Weights (DEMO_J), Blood Pressure (BPX_J), Blood Pressure and Cholesterol (BPQ_J), Body Measures (BMX_J), Diabetes (DIQ_J) and Smoking and Cigarette Use (SMQ_J) data files and documentation. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 106 pages
- National Center for Health Statistics (n.d.-a). NHANES Tutorials: Weighting Module. https://wwwn.cdc.gov/nchs/nhanes/tutorials/weighting.aspx record
- National Center for Health Statistics (n.d.-b). NHANES Tutorials: Variance Estimation Module. https://wwwn.cdc.gov/nchs/nhanes/tutorials/VarianceEstimation.aspx record · 2 pages
- Sarig, T., Galili, T., and Eilat, R. (2023). balance -- a Python package for balancing biased data samples. arXiv:2307.06024. record
- Särndal, C.-E., Swensson, B., and Wretman, J. (1992). Model Assisted Survey Sampling. Springer, New York. record
- Sen, A. R. (1953). On the estimate of the variance in sampling with varying probabilities. Journal of the Indian Society of Agricultural Statistics, 5, 119–127. record
- Solon, G., Haider, S. J., and Wooldridge, J. M. (2015). What are we weighting for? Journal of Human Resources, 50(2), 301–316. https://doi.org/10.3368/jhr.50.2.301 (Working-paper version: NBER Working Paper 18859, 2013, https://www.nber.org/papers/w18859, from which the quotations are taken.) record
- Tillé, Y., and Matei, A. (2025). sampling: Survey Sampling. R package version 2.11, vignette "Comparing the Horvitz-Thompson estimator and Hajek estimator". https://CRAN.R-project.org/package=sampling record
- Tillé, Y., and Wilhelm, M. (2017). Probability sampling designs: Principles for choice of design and balancing. Statistical Science, 32(2), 176–189. https://doi.org/10.1214/16-STS606 record · 2 pages
- Yates, F., and Grundy, P. M. (1953). Selection without replacement from within strata with probability proportional to size. Journal of the Royal Statistical Society, Series B, 15(2), 253–261. https://doi.org/10.1111/j.2517-6161.1953.tb00140.x record