Data Collection and Sampling

How the way data are gathered decides what they can say. Populations, frames and samples; observational studies against randomized experiments, with a treatment that does nothing made to look effective by self-selection; simple random, stratified, systematic and cluster sampling compared by simulation on NHANES adults; NHANES's four-stage design, why its unweighted hypertension prevalence is 54.7% while the weighted one reproduces the published 48.2%, and design-based standard errors by Taylor linearization and the jackknife over its 30 pseudo-PSUs; nonresponse, convenience samples and the 1936 Literary Digest poll.

Of the 5,182 adults whose blood pressure the National Health and Nutrition Examination Survey (NHANES) measured in 2017 and 2018, 2,837 have hypertension by the 2017 definition of the American College of Cardiology and American Heart Association: 54.7%. The National Center for Health Statistics (NCHS), which runs the survey, reports from the same people a crude prevalence of 48.2% for US adults (Ostchega et al., 2020). Neither number is a mistake. The first describes the people who were examined; the second describes the population they were chosen to represent, and the difference of six and a half percentage points comes from how they were chosen and who agreed to take part. NHANES deliberately recruits more Hispanic, Black and Asian adults, more low-income adults and more adults aged 80 and over than a lottery over the population would, so that each of those groups is large enough to study, and it records with every participant a sampling weight that undoes the imbalance.

Most datasets carry the marks of how they were collected, though few document them as carefully. This article is about those marks: the distinction between data that record the world as it is and data from an experiment; the standard ways of drawing a sample and what each costs in precision; how a complex survey such as NHANES is analyzed, with weights and with standard errors that respect its design; and the ways a sample can fail to represent anyone, of which the Literary Digest's 1936 poll is the standard example. The later articles of this topic, from Summarizing Data with pandas to Inference for Proportions and Means, assume the data in hand were collected in a way that supports the question asked. This is the place where that assumption is examined.

Populations, frames and samples

A population is the set of units a question is about: all adults living in the United States, all orders placed on a website last year, all trees in a forest. A unit (or observational unit) is one member, on which variables are recorded. A parameter is a number describing the population, such as the share of US adults with hypertension; a statistic is the corresponding number computed from data, and it estimates the parameter. A census records every unit. A sample records some of them, usually because a census would cost too much or take too long, and the number recorded is the sample size $n$.

Between the population and the sample sits the sampling frame, the list or procedure from which units are actually drawn. NHANES's target population is the civilian, noninstitutionalized resident population of the 50 states and the District of Columbia; its frame starts from all US counties and works down to addresses (Chen et al., 2020). People in prisons, nursing homes and military barracks are outside the frame, so no amount of care in the later steps lets the survey describe them. The gap between the population of interest and the frame, called undercoverage, is the first place a sample can go wrong, and no statistical method can repair it from inside the data, because the missing units leave no trace there.

Observational studies and experiments

Data are observational when the variables are recorded as they occur, without the investigator deciding who gets what. NHANES is observational: it measures blood pressure and asks about medication, but it does not prescribe anything. Observational data show associations, and an association between two variables can come from causes other than one variable acting on the other.

The adults in NHANES who take medicine for high blood pressure have a mean systolic pressure of 136.88 mm Hg, against 122.30 for those who do not. Read naively, the medicine raises blood pressure. The obvious explanation runs the other way: people are prescribed the medicine because their pressure was high, and they are also older, with a mean age of 63.9 years against 44.4. A variable related both to who receives the "exposure" and to the outcome is a confounder. Here age is one, and the reason for the prescription is another that no column of the data records. Comparing within age groups removes part of the first: the difference falls from 14.6 mm Hg to 13.35 among adults aged 18 to 39, 8.55 among those aged 40 to 59 and 4.82 at 60 and over. It does not remove the second, and the differences still point the wrong way. The same mechanism, a comparison that changes when a third variable is held fixed, is worked through in Exploring Data with Groups and Pivot Tables, and standardization, the classical adjustment for it, in Direct and Indirect Standardization.

Randomization

In an experiment the investigator assigns the treatment, and the decisive step is to assign it at random. R. A. Fisher, who made randomization the foundation of experimental design, explained why with an experiment on tasting tea. Whatever care is taken to make the cups alike, "the uncontrolled causes which may influence the result are always strictly innumerable"; the remedy is not to list and control them but to assign the treatments to the cups at random, so that "the simple precaution of randomisation will suffice to guarantee the validity of the test of significance" (Fisher, 1935, §§9 and 10). Random assignment does not make the groups identical. It makes every difference between them, measured or not, a matter of chance whose size probability can describe.

A simulation shows the difference this makes. Take 200 adults at random from the NHANES examinees and give some of them a treatment that has no effect at all on blood pressure. In one version a random half is treated; in the other, people choose for themselves, and the chance of opting in rises with age (a logistic curve that passes one half at age 50). Then compare the mean systolic pressure of the treated and untreated groups with Welch's $t$-test, as an analyst who did not know how the groups were formed might. Each version was run 2,000 times for each of 30 seeds.

Assignment Mean difference, treated minus control (mm Hg) Share of tests with $p < 0.05$ Mean age gap (years)
Random 0.01 (IQR $-0.06$ to 0.05) 0.050 (IQR 0.046 to 0.053) 0.00
Self-selected 12.48 (IQR 12.43 to 12.51) 0.996 (IQR 0.995 to 0.997) 23.75

The figures are medians over the 30 seeds with interquartile ranges. Under random assignment the useless treatment "works" in 5% of experiments, which is what a test at the 5% level promises. Under self-selection it appears to raise blood pressure by about 12.5 mm Hg and passes the test 99.6% of the time, because the treated group is 24 years older on average.

Mean systolic pressure of treated minus untreated adults in 2,000 simulated experiments (seed 0) with a treatment that does nothing. Under random assignment the differences center on zero; when older adults are more likely to opt in, almost every experiment shows a large "effect" that is really the age difference between the groups.
Mean systolic pressure of treated minus untreated adults in 2,000 simulated experiments (seed 0) with a treatment that does nothing. Under random assignment the differences center on zero; when older adults are more likely to opt in, almost every experiment shows a large "effect" that is really the age difference between the groups.

The common summary, that only experiments can show causation, is too strong. Observational studies that measure and adjust for the confounders can support causal conclusions, and much of epidemiology and econometrics is about when such adjustment can be trusted, often for questions that could not ethically be settled by an experiment. What randomization buys is that it does not depend on knowing the confounders. In the medication example no adjustment can be complete, because the reason for each prescription is not in the data; in a randomized trial it would not need to be. Online experiments, in which visitors to a website are assigned at random to one of two versions of a page, apply the same principle at scale; the course wiki's Online Evaluation describes them for search engines.

Probability sampling

A probability sample is one in which every unit of the frame has a known, nonzero chance of selection, set by a random mechanism rather than by anyone's choice. The known chances are what make it possible to compute the standard error of an estimate from the sample itself. Four designs cover most practice. To compare them, the 5,182 NHANES adults can be treated as a small population of their own, with a known hypertension prevalence of $p = 0.5475$, and samples of $n = 300$ drawn from it by each design: 2,000 samples per design for each of 30 seeds. The spread of the 2,000 estimates is the standard error of the design.

Simple random sampling

A simple random sample of size $n$ gives every subset of $n$ units the same chance of being the sample. The estimate is the sample proportion $\hat{p}$, and its standard error is

$$\text{SE}(\hat{p}) = \sqrt{\frac{p(1-p)}{n} \cdot \frac{N-n}{N-1}},$$

where $N$ is the population size and the second factor, the finite population correction, accounts for drawing without replacement (The Sampling Distribution of the Mean derives it). For $N = 5{,}182$ and $n = 300$ the formula gives 0.0279, and the simulation gives 0.0280 (IQR 0.0277 to 0.0282). The design needs a complete list of the population, which is the reason it is rarer in practice than in textbooks.

Stratified sampling

A stratified sample divides the frame into groups called strata, using information known for every unit in advance, and draws a separate simple random sample within each. The estimate combines the strata in their population proportions:

$$\hat{p}_{\text{st}} = \sum_{h} W_h\, \hat{p}_h, \qquad \operatorname{Var}(\hat{p}_{\text{st}}) = \sum_h W_h^2\, \operatorname{Var}(\hat{p}_h),$$

where $W_h = N_h / N$ is the share of the population in stratum $h$ and $\hat{p}_h$ the proportion in its sample. The variance is a plain sum because the strata are sampled independently. With proportional allocation, $n_h = n W_h$, and ignoring the finite population correction, each $\operatorname{Var}(\hat{p}_h) = p_h(1-p_h)/n_h$, so

$$\operatorname{Var}(\hat{p}_{\text{st}}) = \frac{1}{n}\sum_h W_h\, p_h (1 - p_h).$$

Compare this with simple random sampling. The overall variance of a 0/1 variable splits into a part within the strata and a part between them (the law of total variance):

$$p(1-p) = \sum_h W_h\, p_h(1-p_h) + \sum_h W_h (p_h - p)^2.$$

A simple random sample pays for both parts; a proportionally stratified one pays only for the first. Stratification helps exactly as much as the strata differ in the outcome.

Stratifying the NHANES adults by age group shows the size of the gain. The three groups make up 32.7%, 30.4% and 36.9% of the 5,182 adults, and their prevalences are 0.2294, 0.5839 and 0.7997, very different. Of the total variance $p(1-p) = 0.248$, the within-strata part is 0.191 and the between-strata part 0.057, so stratification should multiply the variance by the within-strata share, 0.770. A sample of 300 allocated 98, 91 and 111 to the three groups gave a standard error of 0.0245 (IQR 0.0243 to 0.0248) against 0.0280 for simple random sampling, a variance ratio of 0.769. A stratified sample of 300 is as precise as a simple random sample of about 390. Stratification also guarantees that every stratum appears in the sample, which matters when the strata are themselves of interest; the strata need not then be sampled in proportion, as NHANES's are not.

Systematic sampling

A systematic sample takes every $k$-th unit from a list, starting at a random position among the first $k$, with $k = N/n$. It is easy to carry out on anything that comes in order: files in a cabinet, patients at a clinic, rows of a table. Its properties depend on the order of the list. Sorted by age, the list makes a systematic sample spread evenly over the ages, a kind of implicit stratification, and the standard error was 0.0269 (IQR 0.0268 to 0.0272), between the simple random and the stratified designs. If the list has a cycle that matches the interval, the result can be far worse. With the adults arranged alternately man, woman, man, woman, every 16th person is always of the same sex: depending on the random start, the sample is 100% men or 100% women. Real lists rarely alternate so perfectly, but weekly or seasonal cycles in time-ordered data, and lists sorted by a code that cycles, are common enough to check for.

Cluster and multistage sampling

A cluster sample draws groups of units, such as schools, villages or city blocks, and measures units within the chosen groups only. Its purpose is cost. NHANES examines people in mobile examination centers, trucks that must be driven to each location and set up for weeks; a simple random sample of the country would need a trip for almost every participant. Drawing a few locations and many people in each is far cheaper.

The price is precision, because people in the same place resemble each other. Suppose each of $m$ units sampled from a cluster has variance $\sigma^2$, and any two units in the same cluster have correlation $\rho$, the intraclass correlation. The variance of their sum is $m\sigma^2 + m(m-1)\rho\sigma^2$, so the variance of their mean is

$$\frac{\sigma^2}{m}\bigl(1 + (m-1)\rho\bigr).$$

Compared with $m$ independent units, the variance is multiplied by $1 + (m - 1)\rho$, the design effect that The Binomial Distribution derives from the law of total variance and measures for obesity across the same survey locations. Even a small correlation matters when $m$ is large. Among the NHANES adults, grouped by the 30 survey locations that the public files identify (described below), the intraclass correlation of hypertension is only 0.0108. A two-stage sample that drew 6 of the 30 locations at random and 50 adults within each, weighting each location by its size so that the estimate stays unbiased, gave a standard error of 0.0338 (IQR 0.0334 to 0.0341), a variance 1.46 times that of a simple random sample of the same size. The formula predicts $1 + 49 \times 0.0108 = 1.53$; the difference reflects the unequal sizes of the locations, which the simple formula ignores.

Standard error of the estimated hypertension prevalence from samples of 300 drawn from the 5,182 NHANES adults by four designs: median (dot) and interquartile range (bar) over 30 seeds of 2,000 samples. Stratifying by age gains precision because prevalence differs strongly by age; sampling 6 locations and 50 adults in each loses precision because people in the same location are slightly alike.
Standard error of the estimated hypertension prevalence from samples of 300 drawn from the 5,182 NHANES adults by four designs: median (dot) and interquartile range (bar) over 30 seeds of 2,000 samples. Stratifying by age gains precision because prevalence differs strongly by age; sampling 6 locations and 50 adults in each loses precision because people in the same location are slightly alike.

All four designs were unbiased: the mean of the 2,000 estimates was within 0.0003 of the true prevalence for every design (median over seeds). They differ only in how much the estimate varies, and in what each costs to carry out. Large surveys combine them: stratify to gain precision, cluster to save travel, and sample at different rates in different strata to get enough people from small groups.

How NHANES is sampled

NHANES since 1999 has used a stratified, clustered, four-stage design (Chen et al., 2020). The stages are:

  1. Primary sampling units (PSUs): counties, or groups of adjacent counties or of tracts, selected from strata formed from state-level health indicators. Fifteen locations are visited each year, so a two-year cycle such as 2017 to 2018 comes from 30 PSUs.
  2. Segments: census blocks or combinations of blocks within each PSU.
  3. Dwelling units: households within each segment, which are screened for eligible members.
  4. Persons: individuals within each household, chosen at rates that depend on age, sex, race and Hispanic origin, and income.

Some groups are sampled at higher rates than others. For 2015 to 2018 the oversampled groups were Hispanic persons; non-Hispanic Black persons; non-Hispanic, non-Black Asian persons; non-Hispanic White and other persons at or below 185% of the federal poverty level; and non-Hispanic White and other persons aged 0 to 11 years or 80 and over (Chen et al., 2020). The purpose is precision for those groups. A two-year cycle is still small for them, and the report notes that at least four years of data are needed to reach acceptable reliability for most of its sampling domains.

Weights

A unit selected with probability $\pi_k$ stands for $1/\pi_k$ units of the population, and a weighted estimate counts it that many times:

$$\hat{p}_w = \frac{\sum_k w_k y_k}{\sum_k w_k}, \qquad w_k = \frac{1}{\pi_k},$$

where $y_k$ is 1 for a person with hypertension and 0 otherwise. If some groups are sampled at twice the rate of others, their members get half the weight, and the estimate is approximately unbiased for the population whatever the sampling rates were (the weighted total $\sum_k w_k y_k$ is exactly unbiased), provided every unit had a chance of selection. NHANES's weights start from the inverse selection probabilities and are then adjusted at each stage of the survey (screening, interview and examination): for nonresponse, by increasing the weights of respondents in cells of similar people to cover those who did not take part; by trimming a few extreme weights; and by calibration, so that the weights add up to population totals from the American Community Survey. For 2017 to 2018 the examination weights were calibrated to totals by race and Hispanic origin, age and sex, by race and Hispanic origin, sex and education for adults, and by the mean household income of the census tract (Chen et al., 2020). The NHANES analytic guidelines describe the result as weights that "can be considered as measures of the number of persons represented by the particular survey respondent" (National Center for Health Statistics, 2018). Each data file comes with the right weight for its component; blood pressure is measured in the examination center, so the examination weight WTMEC2YR applies.

The weights of the 5,182 adults add up to 233,922,973, the number of US adults they represent after excluding pregnant women and those without a usable blood pressure. They range from 4,580 to 419,763, a ratio of 92 to 1, with a median of 26,443. The median weight of a non-Hispanic White adult is 58,661, against 18,945 for a non-Hispanic Black adult and 16,456 for a non-Hispanic Asian adult: each Asian participant stands for fewer people because Asian adults were sampled at a higher rate.

Composition of the 5,182 NHANES adults by age and by race and Hispanic origin, unweighted (as examined) and weighted (as estimates for US adults). The examined sample is older and much less White than the population it represents: non-Hispanic White adults are 34.7% of the sample and 62.6% of the weighted total.
Composition of the 5,182 NHANES adults by age and by race and Hispanic origin, unweighted (as examined) and weighted (as estimates for US adults). The examined sample is older and much less White than the population it represents: non-Hispanic White adults are 34.7% of the sample and 62.6% of the weighted total.

Hypertension rises steeply with age and differs between groups, so the composition changes the answer. Unweighted, the prevalence is 0.5475; weighted, it is 0.4822, which reproduces the 48.2% crude prevalence published by NCHS (Ostchega et al., 2020). The same weighting for adults aged 18 to 39 gives 0.2238, the published 22.4%. The course wiki's The Normal Approximation to the Binomial sets the unweighted shares beside the same published figures. Checks like these are worth making before any analysis of a survey: if the weighted estimate of a published figure does not come out right, something in the data preparation is wrong. NCHS's headline figure, 45.4%, is age-adjusted to the US population of 2000, a further standardization that makes figures from different years comparable and is not a property of the sample design.

Weights have a cost in precision. If the weights are unrelated to the outcome, so that the $y_k$ behave like independent values with a common variance $\sigma^2$, the weighted mean has variance $\sigma^2 \sum_k w_k^2 / (\sum_k w_k)^2$, larger than the $\sigma^2/n$ of an unweighted mean by the factor

$$\frac{n \sum_k w_k^2}{\left(\sum_k w_k\right)^2} = 1 + \text{CV}^2,$$

where CV is the coefficient of variation of the weights (their standard deviation, with divisor $n$, over their mean). The identity follows from expanding $\sum_k (w_k - \bar{w})^2 = \sum_k w_k^2 - n\bar{w}^2$. For these adults CV is 1.153 and the factor 2.33, so the 5,182 weighted observations carry about as much information as 2,225 equally weighted ones. That is the price of the oversampling for estimates of the whole population, paid to buy precision for the oversampled groups.

Standard errors that respect the design

A standard error computed as if the 5,182 adults were a simple random sample, $\sqrt{\hat{p}(1-\hat{p})/n} = 0.0069$, is wrong twice over: it ignores the unequal weights and it ignores the clustering in 30 locations. The NHANES guidelines put it directly: "Software packages or procedures that assume a simple random sample, should not be used for computing variances for NHANES" (National Center for Health Statistics, 2018). Exact formulas do not exist for designs this complex, and two approximations are standard.

Taylor linearization. The weighted proportion is a ratio of two estimated totals, $\hat{p}_w = \hat{T}_y / \hat{T}_w$ with $\hat{T}_y = \sum_k w_k y_k$ and $\hat{T}_w = \sum_k w_k$. A first-order Taylor expansion around the true values gives

$$\hat{p}_w - p \approx \frac{1}{\hat{T}_w} \sum_k w_k (y_k - p),$$

a weighted total of the linearized values $z_k = w_k (y_k - \hat{p}_w)/\hat{T}_w$. The variance of an estimated total in a stratified design with independently drawn PSUs is estimated from the PSU totals within each stratum. If $z_{hi}$ is the total of $z_k$ over PSU $i$ of stratum $h$, and stratum $h$ has $n_h$ PSUs with mean total $\bar{z}_h$,

$$\hat{V}(\hat{p}_w) = \sum_h \frac{n_h}{n_h - 1} \sum_i \left(z_{hi} - \bar{z}_h\right)^2.$$

With two PSUs per stratum, as in NHANES, each stratum's term is simply $(z_{h1} - z_{h2})^2$. Everything the formula needs is in the data: the weights and, for each person, a stratum and a PSU code. The formula treats the PSUs as drawn with replacement, the standard approximation in survey software. NCHS recommends Taylor linearization for two-year cycles and used it, through the SUDAAN software, for the published hypertension figures (Chen et al., 2020; Ostchega et al., 2020).

The jackknife. A replication method recomputes the estimate many times on modified weights and measures how much it moves. The delete-one-PSU jackknife drops one PSU at a time, doubles the weights of the other PSU in the same stratum so that the stratum still represents its share of the population, and recomputes $\hat{p}_w$. With $\hat{p}_{(hi)}$ the estimate without PSU $i$ of stratum $h$,

$$\hat{V}_{\text{JK}} = \sum_h \frac{n_h - 1}{n_h} \sum_i \left(\hat{p}_{(hi)} - \hat{p}_w\right)^2.$$

It needs no derivative, so the same code works for a median or a regression coefficient.

Masked PSUs. Releasing the real PSUs would risk identifying the counties visited, so the public files carry masked variance units instead: SDMVSTRA, a pseudo-stratum, and SDMVPSU, a pseudo-PSU within it, built so that variance estimates "closely approximate the variances that would have been estimated using the true design PSUs" (Chen et al., 2020). In 2017 to 2018 there are 15 pseudo-strata, each with 2 pseudo-PSUs.

On these units the Taylor and jackknife standard errors of the weighted prevalence agree to four decimals, 0.0132; the 30 jackknife replicates range from 0.4767 to 0.4876. The design effect, the ratio of the design-based variance to the simple random sampling variance, is 3.62: the 5,182 adults estimate the national prevalence about as precisely as a simple random sample of 1,432 would. Unequal weighting alone would account for a factor of about 2.33; the clustering in 30 locations accounts for most of the rest. For the mean systolic pressure the design effect is 3.45 (weighted mean 123.47 mm Hg, design-based standard error 0.463 against 0.250 under simple random sampling, computed like the proportion's from the weighted estimate of the population variance). The guidelines note that NHANES design effects are "typically greater than 1" and vary between variables (National Center for Health Statistics, 2018).

Intervals for design-based estimates use a $t$ quantile whose degrees of freedom are the number of PSUs minus the number of strata, here $30 - 15 = 15$, not $n - 1$ (National Center for Health Statistics, 2018, §3.2.3.2). The 95% interval for the prevalence is

$$0.4822 \pm t_{15,\,0.975} \times 0.0132 = 0.4822 \pm 2.1314 \times 0.0132 = (0.4540,\ 0.5103).$$

The naive interval from the unweighted data, 0.5339 to 0.5610, is narrower than this by more than half and does not even overlap it. It answers a question about the 5,182 people examined, not about US adults, and states its answer with a precision the design cannot deliver.

Subgroups. An estimate for a subgroup, such as adults aged 18 to 39, must still use the whole design: the Taylor formula is applied with $y_k$ and $w_k$ set to zero outside the subgroup (a domain estimate) rather than to a dataset filtered down to the subgroup first. The guidelines require this (National Center for Health Statistics, 2018, §3.2.3.1): filtering first can remove whole PSUs and so change the variance estimator. For the 1,696 adults aged 18 to 39 the estimate is 0.2238 with standard error 0.0120 and design effect 1.40. Every pseudo-PSU contains adults of that age, so filtering first happens to give the same standard error here; for smaller groups, such as one race and one age band, it often does not, and the degrees of freedom should then count only the strata and PSUs that contain the subgroup.

When samples fail

Nonresponse

Selection is only the first step; the people selected also have to take part. In 2017 to 2018 the screener response rate was 90.9%, and of the 16,211 persons selected, 51.9% completed the interview and 48.8% the examination (unweighted rates) (National Center for Health Statistics, n.d.), down from 76.3% examined in 1999 to 2000 (National Center for Health Statistics, 2018). The weight adjustments for nonresponse assume that, within each adjustment cell, those who took part resemble those who did not. That assumption cannot be checked from the respondents alone.

Nonresponse also occurs item by item. Of the 5,478 nonpregnant adults examined, 296 have no usable systolic or diastolic reading. The weights adjust for persons who refused the interview or the examination, not for missing items within a completed examination (National Center for Health Statistics, 2018), so dropping those 296, as the analyses here do, assumes their blood pressure resembles that of similar adults who were measured. The mechanisms by which values go missing, and what the common remedies do under each, are the subject of Cleaning Data.

Selection bias and convenience samples

A convenience sample is made of whoever is easiest to reach: volunteers who answer an online survey, the customers who leave reviews, the patients of one clinic, the users who happen to be logged in. Its selection probabilities are unknown and often depend on the variable being measured, a selection bias that no sample size cures. A larger convenience sample measures the wrong thing more precisely.

The NHANES adults can show this. Suppose 2,000 of the 5,182 adults volunteer for a study, drawn one at a time with chances proportional to a weight that doubles with every 20 years of age. The volunteers average 58.8 years against 50.0 in the population (seed 0), and their hypertension prevalence overstates the population's by 0.1115 (median over 30 seeds, IQR 0.1049 to 0.1167). A simple random sample of only 100 has a root mean squared error of 0.0489 (IQR 0.0484 to 0.0494). The small random sample is more than twice as accurate as the large biased one, and its error, unlike the volunteers', is known in advance from the formula for the standard error and shrinks as the sample grows.

The Literary Digest poll

The classic case is the poll the Literary Digest conducted before the 1936 presidential election. The magazine mailed about 10 million postcard ballots, using lists that included telephone directories and automobile registrations, and nearly 2.4 million were returned, a response rate of about 24%. It predicted that Alfred Landon would receive 54% of the popular vote and Franklin Roosevelt 41%; Roosevelt won 61% (Lusinchi, 2012). The usual explanation, repeated in many textbooks, is that telephone and car owners were wealthier and more Republican than the electorate, so the frame was biased.

The evidence points more to nonresponse. Squire (1988), using a 1937 Gallup survey that asked people whether they had received and returned a Digest ballot, found that owners of cars and telephones still favored Roosevelt, and concluded, in the words Lusinchi (2012) quotes, that "if everyone who had received a [Digest] ballot had returned it the results would have, at least, correctly predicted Roosevelt a winner". Lusinchi's reanalysis of the same survey, adjusted to known population figures, attributes 5.9 points of the Digest's 19.6-point error in Roosevelt's share of the two-party vote to the mailing list and 13.7 points to the difference between those who returned their ballots and those who did not: Landon's supporters returned them at a much higher rate. Lohr and Brick (2017) showed that the ballot's own question about the 1932 vote, used to weight the respondents to the 1932 result, would have predicted a majority of electoral votes for Roosevelt, though with an estimate of his share still far too low. Two lessons survive the revision. Size did not help: 2.4 million responses missed the winner by a landslide, because who responded depended on how they would vote. And a response rate of 24%, which looked poor in 1936, would be high for many surveys today; what protects them, if anything does, is weighting by variables related both to response and to the outcome, under assumptions that cannot be fully checked.

Data that were never sampled

Much of the data a data scientist meets was not sampled from anything: the transaction log of a company, the posts on a platform, the patients of one hospital system. Such data can be complete for themselves and still unrepresentative of the population a model will serve. Before using them for a population question, the useful habit is to write down which population they cover, how a unit came to be in the data, and which units could not have been, and then to look for a published figure, like the 48.2% here, against which an estimate can be checked. A model trained on the data inherits their selection, a point taken up in Training, Validation, and Test Sets and Ethics in Data Science.

Corrections to the course notes

The unit's notes describe the main designs well and are stated more precisely here in one place. They say that only experimental data with random assignment can support causal claims; randomization is the most reliable route, but careful observational studies with measured confounders have established causes too, and the article states what randomization adds rather than what observation cannot do.

In Python

import numpy as np
import pandas as pd

rng = np.random.default_rng(0)
srs = adults.sample(n=300, random_state=rng)                  # simple random, without replacement
strat = adults.groupby("age_group", observed=True).sample(frac=300 / len(adults), random_state=rng)
k = len(adults) / 300
syst = adults.iloc[np.floor(rng.uniform(0, k) + k * np.arange(300)).astype(int)]
locs = rng.choice(adults.location.unique(), 6, replace=False)  # first stage of a cluster sample

# weighted estimate and Taylor-linearized standard error (strata x PSUs, NHANES masked units)
def svy_mean(y, w, stratum, psu, domain):
    y, w, d = (np.asarray(v, float) for v in (y, w, domain))
    est = np.sum(w * d * y) / np.sum(w * d)
    z = pd.Series(w * d * (y - est) / np.sum(w * d))
    tot = z.groupby([np.asarray(stratum), np.asarray(psu)]).sum()
    v = sum(len(g) / (len(g) - 1) * ((g - g.mean()) ** 2).sum() for _, g in tot.groupby(level=0))
    return est, np.sqrt(v)

svy_mean(persons.htn.fillna(0), persons.WTMEC2YR, persons.SDMVSTRA, persons.SDMVPSU,
         persons.eligible)                                     # 0.4822, SE 0.0132

DataFrame.sample draws without replacement by default, unlike NumPy's Generator.choice, which replaces by default; Probability Fundamentals shows what that difference does. For real survey analysis, established implementations handle the details (several PSUs per stratum, strata with one PSU, replicate weights, regression): the survey package in R and the survey procedures of SAS, Stata and SUDAAN, which the NHANES documentation names (Chen et al., 2020). Weighted statistics in general-purpose libraries are not survey statistics. statsmodels' DescrStatsW treats weights as frequency weights, counts of identical observations: given the NHANES weights, it takes the sample size to be their sum, 233,922,973, and reports a standard error of 0.00003 for the weighted prevalence. Rescaling the weights to average 1 brings it to 0.0069, the simple random sampling value, which still ignores the design. The numbers in this article are computed by tools/experiments/data-science/data-collection-and-sampling.py in the site's repository, with the helper psdata.py beside it, which downloads the NHANES files from the CDC, checks their SHA-256 hashes and reads them with statdata.read_xpt() rather than pandas.read_sas alone, which returns stored zeros as tiny nonzero numbers; the figures are drawn by tools/figures/data-collection-and-sampling.py.

References