Study Design, Sampling and Causal Inference

What the way data were collected allows them to say, with every result proved or cited. Potential outcomes, the average treatment effect and Neyman's theorems for randomized experiments; what randomization balances, Fisher's randomization test and the sharp and weak null hypotheses; blocking, matched pairs and factorial designs; confounding read from causal diagrams, the backdoor criterion and the adjustment formula; propensity scores, inverse-probability weighting and double robustness; and survey sampling designs with the Horvitz–Thompson estimator, which uses the NHANES survey weights that the course pages set aside. Worked on a Cranfield science table in which every potential outcome is known and on NHANES with its strata, clusters and weights.

Two numbers can be computed the same way and mean different things. A difference in mean blood pressure between people who take a medicine and people who do not is a comparison of two groups; whether it measures what the medicine does depends on how the people came to be in the groups. A proportion computed from a survey describes the people surveyed; whether it describes a country depends on how they were chosen. This unit is about those two questions: what a study's design allows its data to conclude about causes, and about a population. It treats experiments, where the investigator assigns the treatment; observational studies, where the investigator does not; and surveys, where the investigator chooses whom to measure.

The course pages meet these questions without the theory. Data Collection and Sampling contrasts observational studies with randomized experiments, compares sampling designs by simulation on NHANES, and reproduces a published prevalence from the survey weights; Conditional Probability and the Law of Total Probability meets confounding and Simpson's paradox in diabetes, age and smoking among NHANES adults, and The Relative Risk and the Odds Ratio measures how risks differ between groups; Permutation Tests and The Paired Samples t-Test analyze designed comparisons; and the course's NHANES analyses leave the survey weights aside. This unit supplies the reasons: the definitions of causal effects, the theorems that make a randomized comparison unbiased and its uncertainty computable, the conditions under which an observational comparison can be adjusted into a causal one, and the design-based theory of surveys.

What it assumes. Random Variables, Expectation and Variance and Conditional Expectation and the Laws of Total Expectation and Variance from Probability Theory; The Sampling Distribution of the Mean for the finite population correction; Permutation Tests for the counting arguments of randomization tests; One-Way Analysis of Variance for mean squares; and, for regression adjustment and weighted regression, Linear Models.

What is proved and what is cited. Proved across the unit, in the design-based setting where the units are fixed and the probability is the experimenter's or the sampler's: that no data from an experiment can reveal a unit's effect, and that no estimator of the variance of the difference in means is unbiased (a finite-sample form of the non-identifiability of the variance of the unit effects that Ding's textbook notes); Neyman's theorems that the difference in means is unbiased under complete randomization, with its exact variance and a conservative variance estimator, exact under a constant effect, and their superpopulation version; regression adjustment with fixed slopes; the balance of every attribute under randomization; the exactness of Fisher's randomization test for any statistic and any design; the exact expected variance of the randomization distribution, which shows when a test of no effect misleads about no average effect; the Bernoulli design as complete randomization given the number treated; the exact variance of blocked randomization and its exact difference from complete randomization, the paired design's conservative variance estimator, and the variance of every contrast in a factorial design; the chain, fork and collider structures from the factorization of a causal diagram, adjustment for the parents of the treatment, and the adjustment formula from conditional exchangeability, positivity and consistency; Berkson's paradox and the bias of adjusting for a collider or a mediator, exactly in linear models; the balancing property of the propensity score and the inheritance of unconfoundedness (Rosenbaum and Rubin's theorems), inverse-probability weighting and the double robustness of the augmented estimator; the variances of simple random, stratified, cluster and two-stage sampling with the exact comparisons between them and Neyman's optimal allocation; and the unbiasedness and variance of the Horvitz–Thompson estimator under any design, with the Hájek estimator's error identity and linearized variance. Cited rather than proved: the soundness and completeness of d-separation and Pearl's backdoor theorem; the large-sample normality on which the normal intervals of the experimental articles rest; Lin's results on regression adjustment with estimated slopes; Wu and Ding's result that the randomization test with a studentized statistic is asymptotically conservative for the weak null; the large-sample theory of matching and of the estimated propensity score; and everything about how NHANES was designed and weighted, from its documentation.

Notation. Units are $i = 1, \ldots, n$, with treatment indicator $W_i$ (1 treated, 0 control) and pre-treatment covariates $X_i$. Each unit has potential outcomes $Y_i(1)$ and $Y_i(0)$, unit effect $\tau_i = Y_i(1) - Y_i(0)$, and observed outcome $Y_i = W_iY_i(1) + (1 - W_i)Y_i(0)$. In the finite-population view the average effect is $\tau_{\mathrm{fp}} = \frac1n\sum_i\tau_i$, variances have divisor $n - 1$ ($S_1^2$, $S_0^2$, $S_\tau^2$), and the difference in means $\hat\tau$ has Neyman's variance estimator $\hat V = s_1^2/n_1 + s_0^2/n_0$. In the superpopulation view the average treatment effect is $\tau = E[Y(1) - Y(0)]$, the effect on the treated $\tau_{\mathrm t} = E[Y(1) - Y(0) \mid W = 1]$, the propensity score $e(x) = P(W = 1 \mid X = x)$, and unconfoundedness is written $\{Y(1), Y(0)\} \perp\!\!\!\perp W \mid X$. Blocks of an experiment are indexed $k$, with sizes $n_{[k]}$. In surveys the population is $U$ with $N$ units $k$ and values $y_k$, the sample $s$ has inclusion probabilities $\pi_k$, strata are indexed $h$ with population shares $P_h$, and the Horvitz–Thompson estimator of a total is $\sum_{k \in s}y_k/\pi_k$.

Data. Three data sets carry the unit. The experimental articles use a science table built from the Cranfield retrieval collection: the 225 queries are the units, BM25 configurations are the treatments, and average precision is the outcome. A ranker is deterministic, so every potential outcome of every query is known, and the quantities an experiment can never see, such as the variance of the unit effects, can be computed exactly; experiments are then simulated by drawing assignments. The observational articles compare former and never smokers in the risk of diagnosed diabetes among NHANES adults of 2017 to 2018, the example of the probability unit, with age, sex, race and Hispanic origin, education and income as confounders. The survey articles treat the 5,182 NHANES adults as a finite population for checking formulas, and then use NHANES's own strata, clusters and weights. Every number is computed by a script in tools/experiments/statistics/.

Articles

Experiments

  • Potential Outcomes and the Average Treatment Effect: the potential-outcomes model, SUTVA and the science table; individual, average and on-the-treated effects; why a unit's effect is never observed, and the proof, a finite-sample form of a known non-identifiability, that no estimator of the variance of the difference in means can be unbiased; Neyman's theorems for complete randomization proved in the finite population (unbiasedness, the exact variance, the conservative variance estimator, exact under a constant effect) with a sharper bound, and the superpopulation version; regression adjustment proved for fixed slopes, with Freedman's critique and Lin's interacted estimator cited. On the Cranfield table of BM25 with and without stemming, the variance estimator is 5.8% too large on average and adjustment for a measure of query difficulty divides the variance by almost six.
  • Randomized Experiments: randomization proved to balance every attribute of the units, measured or not, in expectation and in distribution, with the size and correlation of chance imbalances; Fisher's randomization test proved exact under the sharp null for any statistic and any design; the sharp null against Neyman's weak null, with the exact expected variance of the randomization distribution showing why the test of the difference in means can lose its level under the weak null, and the studentized test, which keeps it in large samples (Wu and Ding, 2021, cited); confidence intervals for a constant effect by inverting the test; the Bernoulli design and the Horvitz–Thompson estimator; and A/B tests and interleaving read as experiments.
  • Blocking and Factorial Designs: blocked randomization proved unbiased, with the exact difference from complete randomization, a difference of mean squares that makes blocking help exactly when an analysis-of-variance F ratio exceeds one; random blocks proved neutral on average and blocks from fixed strata proved never to hurt in expectation; matched pairs as blocks of size two, with the paired variance estimator proved conservative; two-level factorial designs, with the variance of every contrast proved and the reason one experiment estimates every effect with the whole sample. On Cranfield, blocking on query difficulty divides the variance by 4.78, blocking on the number of query terms (after stop-word removal) raises it slightly, and a three-factor experiment measures the effects of stemming, length normalization and the stop list.

Observational studies

  • Confounding and Causal Diagrams: which variables an observational comparison must adjust for and which it must leave alone, read off a directed acyclic graph. The chain, fork and collider structures are proved from the graph's factorization, adjustment for the parents of the treatment from the truncated factorization, and the adjustment formula (the g-formula) from consistency, conditional exchangeability and positivity, with the standard population deciding whose effect is estimated; d-separation and Pearl's backdoor criterion are stated, cited and used on the smoking and diabetes data, where Simpson's paradox is confounding by age (risk ratio 1.756 crude, 1.200 to 1.235 by standard population). Berkson's paradox and the correlation induced by selecting on a sum of normals are proved, and M-bias and adjustment for a mediator computed exactly in linear models.
  • Matching and Propensity Scores: the probability of treatment given the covariates proved to balance them and to inherit unconfoundedness (Rosenbaum and Rubin, 1983), with every balancing score characterized; inverse-probability weighting proved unbiased with the known score (the Horvitz–Thompson estimator applied to the treated), its variance derived, and an unbiased estimator with infinite variance under positivity shown; the estimated score turning weighting into subclassification and beating the true score; weights for the effect on the treated, exact matching and the double robustness of the augmented estimator. On NHANES former and never smokers the risk ratio for diabetes falls from 1.76 to 1.12; simulations with a known answer compare eight estimators when each working model is right or wrong.

Surveys

  • Survey Sampling Designs: the variances of simple random, stratified, cluster and two-stage sampling derived from the design alone, with the finite population correction, the exact comparison of proportional stratification with simple random sampling, Neyman's optimal allocation with and without costs, the intraclass correlation and the design effect $1 + (m - 1)\rho$, the ratio estimator for unequal clusters, and the exact bias of the with-replacement variance estimator that survey software uses; every formula checked by Monte Carlo on the NHANES adults, and the design of NHANES explained from these results.
  • Survey Weights and the Horvitz–Thompson Estimator: weighting by the inverse of the inclusion probability proved unbiased under any probability design, with the Horvitz–Thompson variance in the general and Sen–Yates–Grundy forms; the Hájek ratio estimator, its error identity, a bound on its bias and its linearized variance; how NHANES builds its examination weights; Kish's effective sample size; and when weights matter for regression. Applied at last to the NHANES quantities the course pages computed unweighted, with design-based standard errors and design effects, and checked on a pseudo-population built from the weights.

Where the unit leads

The experimental articles treat a single binary treatment and units that do not interfere; experiments on networks and marketplaces, where one unit's treatment reaches another, need designs beyond this unit. The observational articles assume that the confounders are measured; when they are not, the question becomes one of sensitivity, which the propensity-score article begins with the E-value. Comparisons of models and retrieval systems, where every unit (a question or a query) receives every treatment, are the subject of Statistics of Model Evaluation, and in particular Paired Comparisons of Models; online experiments on search engines are described in Online Evaluation.

References

Each article lists its own references. The sources on which the unit draws throughout: record · 4 pages