Potential Outcomes and the Average Treatment Effect

What a causal effect is when each unit can be observed under only one treatment, and what a randomized experiment can say about it. The potential-outcomes model, SUTVA and the science table; individual, average and on-the-treated effects; why an individual effect is never observed, and the proof, a finite-sample form of a known non-identifiability, that no estimator of the variance of the difference in means can be unbiased; Neyman's theorems for complete randomization proved in the finite population (the difference in means is unbiased, its exact variance, the usual variance estimator conservative with equality exactly under a constant effect, and a sharper bound); the superpopulation version, where the same estimator is unbiased; and regression adjustment, with the fixed-slope version proved exactly and Freedman's critique and Lin's interacted estimator stated and cited. Worked on a real science table in which every potential outcome is known, the 225 Cranfield queries under BM25 with and without stemming. Worked problems on a four-unit table enumerated in full, the variance bound and an adjusted estimator.

On the 225 queries of the Cranfield collection (Cleverdon, 1967), Porter stemming raises the average precision of BM25 on 129 queries, lowers it on 83 and leaves it unchanged on 13. Its mean gain is 0.0243. Those counts exist only because a ranking function can be run twice on the same query, once with stemming and once without. A patient cannot both take a drug and not take it over the same week, and a visitor to a website sees only one version of a page. What is the effect of a treatment on a unit that can be observed under only one treatment, and what can an experiment that sees each unit once say about it?

The answer starts by giving each unit two outcomes. A potential outcome is the value the outcome would take for a given unit if it received a given treatment: $Y_i(1)$ under treatment and $Y_i(0)$ under control. Both are fixed properties of the unit before anything is assigned. An experiment reveals one of them per unit, and the effect of the treatment on the unit is their difference. This model of causation goes back to Neyman (1923), who used the notation for agricultural experiments, and to Rubin (1974), who developed it for experiments and observational studies; Ding (2024, ch. 2) gives the modern account this article follows.

Here is the route. First the model: units, potential outcomes, the assumption that makes the notation meaningful, and the effects defined from it, on the Cranfield table where every potential outcome is known. Then why an individual effect can never be observed, and a proof that even the variance of the standard estimator has no unbiased estimator. Then complete randomization and Neyman's results, proved in full: the difference in means is unbiased for the average effect, its variance has an exact formula, and the usual variance estimator is conservative unless the effect is the same for every unit. Then the superpopulation version, in which the conservativeness disappears. Finally, regression adjustment for a covariate, proved exactly for fixed slopes, with Freedman's critique and Lin's answer stated and cited. The article assumes Random Variables, Expectation and Variance and, for the superpopulation, Conditional Expectation and the Laws of Total Expectation and Variance. Later articles of the unit build on its notation and its theorems.

Two outcomes for every unit

Consider $n$ units, indexed by $i = 1, \ldots, n$, and a treatment with two levels, 1 (treatment) and 0 (control). The treatment indicator $W_i$ is 1 if unit $i$ is treated and 0 if not, and $\mathbf{W} = (W_1, \ldots, W_n)$ is the assignment. Each unit has two potential outcomes, $Y_i(1)$ and $Y_i(0)$. The $n \times 2$ array of all of them is what Rubin calls the science table (Ding, 2024, §2.2): it describes the units, and nothing the experimenter does changes it. The observed outcome is the potential outcome of the treatment actually received,

$$Y_i = W_iY_i(1) + (1 - W_i)Y_i(0).$$

The running table. The units are the 225 Cranfield queries, the treatment is Porter stemming in BM25 (with $k_1 = 1.2$, $b = 0.75$ and the 31-word stop list of the retrieval-evaluation articles), and the outcome is the query's average precision (AP). So $Y_i(1)$ is the AP of query $i$ with stemming and $Y_i(0)$ its AP without. These are the two runs compared in Comparing Systems with Significance Tests, whose mean average precisions, 0.307 with stemming and 0.282 without, are the two column means. A ranker is deterministic, so both columns are known for every query. That makes this table unusual: it is a real science table, not a simulated one, and every quantity that an experiment cannot see can be computed from it. The first five queries are

Query $Y_i(1)$, stemmed $Y_i(0)$, unstemmed $\tau_i = Y_i(1) - Y_i(0)$
1 0.218 0.233 $-0.016$
2 0.223 0.167 0.057
3 0.584 0.592 $-0.008$
4 0.529 0.591 $-0.062$
5 0.374 0.173 0.201

(rounded to three decimals; the script tools/experiments/statistics/potential-outcomes.py computes every number in this article from the full table, which build_cranfield_science_table.py writes).

Writing an outcome as $Y_i(W_i)$ assumes two things, which Rubin (1980) named together the stable unit treatment value assumption, SUTVA (as reported by Ding, 2024, Assumptions 2.1 to 2.3):

  • No interference. Unit $i$'s potential outcomes do not depend on the treatments of other units. Without it, $Y_i$ would be a function of the whole vector $\mathbf{W}$, and each unit would have $2^n$ potential outcomes rather than two.
  • No hidden versions of the treatment. "Treatment" means one thing. If two different stemmers could both be called stemming, $Y_i(1)$ would not be a single number.

For the Cranfield table both hold by construction: each query is ranked on its own, and the stemmer is fixed. For people they can fail. Ding (2024, §2.2) gives the examples of vaccination, where a friend's flu shot lowers one's own risk, and of advertising seen by one's friends. On a website, two versions of a page that compete for the same limited stock of goods interfere through the stock. Randomized Experiments returns to this for online experiments.

Example (the same person twice). A person with a headache who took nothing yesterday and takes aspirin today might seem to have observed both potential outcomes. Ding (2024, §2.2.2) points out that the two days are two different units, "$i$, before" and "$i$, after", each with its own pair of potential outcomes, of which one each is observed. The comparison of the two days measures the effect only under the extra assumption that the untreated outcome would have been the same on both days. The Cranfield table needs no such assumption, because running a query twice does not change the query.

Effects: individual, average, on the treated

The unit effect (individual treatment effect) is $\tau_i = Y_i(1) - Y_i(0)$. On Cranfield it runs from $-0.453$ (query 118, which stemming hurts most) to 0.716 (query 205, which it helps most), and its standard deviation over the 225 queries is 0.109. The effect is far from constant.

The usual estimand is the finite-population average treatment effect,

$$\tau_{\mathrm{fp}} = \frac1n\sum_{i=1}^n \tau_i = \bar Y(1) - \bar Y(0), \qquad \bar Y(w) = \frac1n\sum_{i=1}^n Y_i(w).$$

The second form says that the average of the differences is the difference of the averages, which is just the linearity of a sum. For Cranfield $\tau_{\mathrm{fp}} = 0.0243$, the difference between the two mean average precisions.

Example (the median is not linear). The same identity fails for medians. The median of the 225 unit effects is 0.0053, while the difference between the median of $Y(1)$ and the median of $Y(0)$ is 0.0168. The first describes the typical query's gain; the second compares two distributions and describes no query. Ding (2024, Problem 2.2) makes this distinction; both are causal estimands, but only the average can be estimated by comparing two groups' means, as the next sections show.

Two more estimands appear in the unit. The effect on the treated is the average of $\tau_i$ over the units that were in fact treated,

$$\tau_{\mathrm{fp,t}} = \frac{1}{n_1}\sum_{i:\,W_i = 1}\tau_i, \qquad n_1 = \sum_i W_i,$$

and the effect on the controls is defined in the same way over the $n_0 = n - n_1$ controls. The weighted average of the two, with weights $n_1/n$ and $n_0/n$, is $\tau_{\mathrm{fp}}$. In a randomized experiment the effect on the treated is a random quantity, since it depends on which units were treated. In the running experiment below, 113 queries are stemmed: for them the average effect is 0.0317, for the 112 others it is 0.0169, and $(113 \times 0.0317 + 112 \times 0.0169)/225$ returns 0.0243. In an observational study the effect on the treated is often the natural question (what did the treatment do for those who chose it?), and Matching and Propensity Scores estimates it.

When the units are regarded as a sample from a larger population, the estimands are population quantities: the average treatment effect $\tau = E[Y(1) - Y(0)]$ and the average effect on the treated $\tau_{\mathrm t} = E[Y(1) - Y(0) \mid W = 1]$, where $(Y(1), Y(0), W)$ describes a unit drawn at random. This is the superpopulation view. The finite-population view treats the $n$ units as fixed and puts all randomness in the assignment; it is the view of this article until the section on the superpopulation, and it is called design-based because the only probability in it is the one the experimenter creates.

Why an individual effect is never observed

An experiment observes, for each unit, either $Y_i(1)$ or $Y_i(0)$, never both. Dasgupta, Pillai and Rubin (2015, §2) quote Rubin's (1978) name for this, the "fundamental problem facing inference for causal effects". Here is what the problem rules out, in three precise statements.

Proposition 1 (the missing outcome is unrestricted). Let $\mathbf{w}$ be the realized assignment and $y_1, \ldots, y_n$ the observed outcomes. For any unit $i$ and any real number $c$, there is a science table that produces exactly these observed data under $\mathbf{w}$ and has $\tau_i = c$.

Proof. If $w_i = 1$, set $Y_i(1) = y_i$ and $Y_i(0) = y_i - c$; if $w_i = 0$, set $Y_i(0) = y_i$ and $Y_i(1) = y_i + c$. Fill every other unit's observed potential outcome with its observed value and its missing one arbitrarily. The table reproduces the data and has $\tau_i = c$. $\square$

So no function of the observed data equals $\tau_i$ for every science table: the data cannot distinguish a unit that gained 0.7 from one that lost 0.4. What can be done is to be right on average over the randomization, and for a single unit even that is possible but useless.

Example (an unbiased estimate of one unit's effect). Suppose unit $i$ is treated with probability $p$, $0 < p < 1$. The quantity $\hat\tau_i = W_iY_i/p - (1 - W_i)Y_i/(1 - p)$ has expectation $p \cdot Y_i(1)/p - (1 - p)\cdot Y_i(0)/(1 - p) = \tau_i$. Its two possible values are $Y_i(1)/p$ and $-Y_i(0)/(1 - p)$. For query 1 with $p = 1/2$ these are 0.436 and $-0.467$, while $\tau_1 = -0.016$. Averaging such estimates over many units averages away their wild individual errors, and that is the route to estimating $\tau_{\mathrm{fp}}$. The average of these single-unit estimates is the Horvitz–Thompson estimator, which Survey Weights and the Horvitz–Thompson Estimator proves unbiased for any design and Randomized Experiments compares with the difference in means.

The third statement concerns not $\tau_i$ but the spread of the unit effects, which will turn out to govern the precision of every experiment. Let

$$S_1^2 = \frac{1}{n - 1}\sum_i\big(Y_i(1) - \bar Y(1)\big)^2, \quad S_0^2 = \frac{1}{n - 1}\sum_i\big(Y_i(0) - \bar Y(0)\big)^2, \quad S_\tau^2 = \frac{1}{n - 1}\sum_i(\tau_i - \tau_{\mathrm{fp}})^2$$

be the finite-population variances of the two columns and of the unit effects, all with divisor $n - 1$, and $S_{10}$ the covariance of the two columns with the same divisor. Expanding $(\tau_i - \tau_{\mathrm{fp}})^2 = \big((Y_i(1) - \bar Y(1)) - (Y_i(0) - \bar Y(0))\big)^2$ and summing gives

$$S_\tau^2 = S_1^2 + S_0^2 - 2S_{10}. \tag{1}$$

On Cranfield, $S_1^2 = 0.0573$, $S_0^2 = 0.0519$, $S_{10} = 0.0486$ (a correlation of 0.892 between a query's two APs) and $S_\tau^2 = 0.0120$. An experiment reveals each column on part of the units, so $S_1^2$ and $S_0^2$ can be estimated. $S_{10}$ and $S_\tau^2$ need both outcomes of the same unit. Theorem 4 below shows that the consequence is sharp: no estimator of the variance of the difference in means is unbiased for every science table.

Complete randomization and the difference in means

In a completely randomized experiment the experimenter fixes $n_1$ and $n_0 = n - n_1$, with $1 \le n_1 \le n - 1$, and chooses the treated set uniformly among the $\binom{n}{n_1}$ subsets of size $n_1$ (Ding, 2024, Definition 3.1). Write $p = n_1/n$. The assignment is chosen without looking at the potential outcomes, which are fixed numbers; all probabilities below are over the $\binom{n}{n_1}$ equally likely assignments. The estimator is the difference in means

$$\hat\tau = \bar Y_1 - \bar Y_0, \qquad \bar Y_1 = \frac{1}{n_1}\sum_{i:\,W_i = 1}Y_i, \quad \bar Y_0 = \frac{1}{n_0}\sum_{i:\,W_i = 0}Y_i.$$

The running experiment. The script draws one complete randomization (seed 2026) of the 225 queries: 113 are stemmed and 112 are not. The treated queries' mean AP is 0.2971 and the controls' 0.2992, so $\hat\tau = -0.0021$. This experiment would report that stemming slightly lowers average precision, while the true average effect on these very queries is $+0.0243$. Nothing went wrong: the controls happen to be easier queries. How far from $\tau_{\mathrm{fp}}$ can such an estimate be expected to fall, and can the experiment itself say so? That is what the next three theorems answer.

Everything follows from the first two moments of the indicators.

Lemma 1 (the assignment indicators). Under complete randomization, for $i \ne j$,

$$E[W_i] = p, \qquad \operatorname{Var}(W_i) = p(1 - p), \qquad \operatorname{Cov}(W_i, W_j) = -\frac{p(1 - p)}{n - 1}.$$

Proof. The number of treated sets containing unit $i$ is $\binom{n - 1}{n_1 - 1}$, so $E[W_i] = P(W_i = 1) = \binom{n - 1}{n_1 - 1}\big/\binom{n}{n_1} = n_1/n = p$. Since $W_i^2 = W_i$, $\operatorname{Var}(W_i) = p - p^2$. The number containing both $i$ and $j$ is $\binom{n - 2}{n_1 - 2}$, so $E[W_iW_j] = \frac{n_1(n_1 - 1)}{n(n - 1)}$ and

$$\operatorname{Cov}(W_i, W_j) = \frac{n_1(n_1 - 1)}{n(n - 1)} - \frac{n_1^2}{n^2} = \frac{n_1\big(n(n_1 - 1) - n_1(n - 1)\big)}{n^2(n - 1)} = \frac{n_1(n_1 - n)}{n^2(n - 1)} = -\frac{p(1 - p)}{n - 1}. \qquad\square$$

The covariance is negative because the count of treated units is fixed: one more treated unit here means one fewer elsewhere.

Lemma 2 (variance of a weighted count). For fixed numbers $c_1, \ldots, c_n$ with mean $\bar c$ and variance $S_c^2 = \frac{1}{n - 1}\sum_i(c_i - \bar c)^2$,

$$\operatorname{Var}\Big(\sum_{i=1}^n W_ic_i\Big) = n\,p(1 - p)\,S_c^2.$$

Proof. By Lemma 1,

$$\operatorname{Var}\Big(\sum_i W_ic_i\Big) = p(1 - p)\sum_i c_i^2 - \frac{p(1 - p)}{n - 1}\sum_{i \ne j}c_ic_j = \frac{p(1 - p)}{n - 1}\Big((n - 1)\sum_i c_i^2 - \Big(\big(\textstyle\sum_i c_i\big)^2 - \sum_i c_i^2\Big)\Big),$$

and $n\sum_i c_i^2 - (\sum_i c_i)^2 = n\sum_i(c_i - \bar c)^2 = n(n - 1)S_c^2$. $\square$

Lemma 2 is the variance of a sample total under sampling without replacement, the finite population correction of The Sampling Distribution of the Mean, in another form: the treated set is a simple random sample of the units. Survey Sampling Designs proves the same indicator moments (its Lemma 1) and the unbiasedness of the sample variance (its Theorem 2) in the sampling setting, where the sample plays the part of the treated group.

Theorem 1 (the difference in means is unbiased). Under complete randomization, $E[\hat\tau] = \tau_{\mathrm{fp}}$, for every science table.

Proof. The observed outcome of a treated unit is $Y_i(1)$ and of a control $Y_i(0)$, so

$$\hat\tau = \frac{1}{n_1}\sum_i W_iY_i(1) - \frac{1}{n_0}\sum_i(1 - W_i)Y_i(0).$$

By Lemma 1, $E[W_i] = n_1/n$ and $E[1 - W_i] = n_0/n$, so $E[\hat\tau] = \frac1n\sum_i Y_i(1) - \frac1n\sum_i Y_i(0) = \tau_{\mathrm{fp}}$. $\square$

No model for the outcomes was used: not normality, not a constant effect, not even independence between units. The only probability is the randomization. That is what it means to say that randomization licenses the causal reading of a difference in means, and Randomized Experiments takes the idea further, to the balance of every covariate.

For the variance, a representation does the work. Since $W_i/n_1 = W_i/(np)$ and $(1 - W_i)/n_0 = (1 - W_i)/(n(1 - p))$,

$$\hat\tau - \tau_{\mathrm{fp}} = \frac1n\sum_i\Big(\frac{W_iY_i(1)}{p} - \frac{(1 - W_i)Y_i(0)}{1 - p} - Y_i(1) + Y_i(0)\Big) = \frac1n\sum_i(W_i - p)\,a_i, \qquad a_i = \frac{Y_i(1)}{p} + \frac{Y_i(0)}{1 - p}. \tag{2}$$

The last step collects terms: $\frac{W_iY_i(1)}{p} - Y_i(1) = \frac{(W_i - p)Y_i(1)}{p}$ and $-\frac{(1 - W_i)Y_i(0)}{1 - p} + Y_i(0) = \frac{(W_i - p)Y_i(0)}{1 - p}$.

Theorem 2 (Neyman's variance). Under complete randomization,

$$V = \operatorname{Var}(\hat\tau) = \frac{S_1^2}{n_1} + \frac{S_0^2}{n_0} - \frac{S_\tau^2}{n} = \frac{p(1 - p)}{n}\,S_a^2,$$

where $S_a^2$ is the variance (divisor $n - 1$) of the $a_i$ of (2).

Proof. By (2), $\operatorname{Var}(\hat\tau) = \frac{1}{n^2}\operatorname{Var}(\sum_i W_ia_i)$, since the $p\,a_i$ are constants. Lemma 2 gives $\frac{1}{n^2}\,n\,p(1 - p)\,S_a^2$, the second form. For the first, $S_a^2 = S_1^2/p^2 + S_0^2/(1 - p)^2 + 2S_{10}/(p(1 - p))$, so

$$\frac{p(1 - p)}{n}S_a^2 = \frac{(1 - p)S_1^2}{np} + \frac{p\,S_0^2}{n(1 - p)} + \frac{2S_{10}}{n}.$$

Now $\frac{1 - p}{np} = \frac{1}{n_1} - \frac1n$ and $\frac{p}{n(1 - p)} = \frac{1}{n_0} - \frac1n$, so the variance is $\frac{S_1^2}{n_1} + \frac{S_0^2}{n_0} - \frac{S_1^2 + S_0^2 - 2S_{10}}{n}$, and by (1) the last numerator is $S_\tau^2$. $\square$

The theorem is Neyman's (1923), in the form and with the proof strategy Ding gives (2024, Theorem 4.1 and §4.3). The first form differs from the textbook variance of a difference of two independent sample means, $\sigma_1^2/n_1 + \sigma_0^2/n_0$ in The Two-Sample t-Test and Welch's t-Test, by the term $-S_\tau^2/n$. The more the unit effects vary, the smaller the variance, for fixed column variances. Ding (2024, §4.2) explains why: with $S_{10} > 0$, a treated group that happens to have large $Y(1)$ values has large $Y(0)$ values too, which are then missing from the control group, so the two errors add; a smaller covariance (more varied effects) makes them partly cancel.

On Cranfield with 113 treated queries, $V = 0.000917$, a standard deviation of 0.0303. The running experiment's error, $\hat\tau - \tau_{\mathrm{fp}} = -0.0265$ (computed before rounding the two terms), is less than one standard deviation. The script checks the theorem by drawing 20,000 assignments for each of 30 seeds: the variance of the 20,000 estimates divided by $V$ has median 1.005 over the seeds (interquartile range 0.999 to 1.010), and the mean estimate differs from $\tau_{\mathrm{fp}}$ by a median of 0.00002 (bias interquartile range $-0.00020$ to 0.00014), as Theorem 1 requires.

Example (an experiment that sees both outcomes). The evaluation articles observe both columns for every query, so for these 225 queries $\tau_{\mathrm{fp}}$ is computed, not estimated. The standard error that Comparing Systems with Significance Tests attaches to the paired comparison, $S_\tau/\sqrt n = 0.0073$ here, describes how the mean difference would vary over new sets of queries from a population of queries. An experiment that sees each query under one condition only has, for the same queries, a standard deviation of 0.0303, about four times larger, because it compares different queries. Designs between these extremes, which use blocks or pairs of similar units, are the subject of Blocking and Factorial Designs.

Neyman's variance estimator is conservative

The experiment cannot compute $V$, since $S_\tau^2$ needs both outcomes of each unit. The natural estimate replaces the column variances by the sample variances within each group,

$$\hat V = \frac{s_1^2}{n_1} + \frac{s_0^2}{n_0}, \qquad s_1^2 = \frac{1}{n_1 - 1}\sum_{i:\,W_i = 1}(Y_i - \bar Y_1)^2, \quad s_0^2 = \frac{1}{n_0 - 1}\sum_{i:\,W_i = 0}(Y_i - \bar Y_0)^2,$$

(with $n_1, n_0 \ge 2$), the same formula as Welch's standard error.

Theorem 3 (conservativeness). Under complete randomization, $E[s_1^2] = S_1^2$ and $E[s_0^2] = S_0^2$, so

$$E[\hat V] = \frac{S_1^2}{n_1} + \frac{S_0^2}{n_0} = V + \frac{S_\tau^2}{n} \ge V,$$

with equality if and only if every unit has the same effect, $\tau_i = \tau_{\mathrm{fp}}$ for all $i$.

Proof. Write $y_i = Y_i(1)$ and $\bar y = \bar Y(1)$. Since $\sum_{W_i = 1}(y_i - \bar Y_1)^2 = \sum_i W_iy_i^2 - n_1\bar Y_1^2$,

$$(n_1 - 1)\,E[s_1^2] = p\sum_i y_i^2 - n_1E[\bar Y_1^2].$$

By Theorem 1's argument $E[\bar Y_1] = \bar y$, and by Lemma 2 with $c_i = y_i$, $\operatorname{Var}(\bar Y_1) = \frac{1}{n_1^2}\,n\,p(1 - p)\,S_1^2 = \frac{(1 - p)S_1^2}{n_1}$. So $n_1E[\bar Y_1^2] = (1 - p)S_1^2 + n_1\bar y^2$, and

$$(n_1 - 1)\,E[s_1^2] = p\Big(\sum_i y_i^2 - n\bar y^2\Big) - (1 - p)S_1^2 = p(n - 1)S_1^2 - (1 - p)S_1^2 = (n_1 - 1)S_1^2,$$

because $p(n - 1) - (1 - p) = pn - 1 = n_1 - 1$. The control group is a simple random sample of size $n_0$ as well, so the same computation gives $E[s_0^2] = S_0^2$. Then $E[\hat V] - V = S_\tau^2/n$ by Theorem 2, which is zero exactly when all $\tau_i$ equal their mean. $\square$

This is the third part of Neyman's theorem (Ding, 2024, Theorem 4.1). On Cranfield $E[\hat V] = 0.000970$, a standard deviation of 0.0311 against the true 0.0303: the ratio of variances is 1.058, the excess being $S_\tau^2/n = 0.000053$. In the simulation, the mean of the 20,000 variance estimates divided by $E[\hat V]$ has median 1.000 (interquartile range 1.000 to 1.000), and the 95% interval $\hat\tau \pm 1.96\sqrt{\hat V}$ covered $\tau_{\mathrm{fp}}$ in a median 0.955 of the assignments (interquartile range 0.954 to 0.956). A normal approximation with the true variance predicts 0.956. When the script replaces the table by one with a constant effect, $Y_i(1) = Y_i(0) + 0.0243$, the coverage falls to 0.948 (interquartile range 0.947 to 0.950), close to the nominal level; the remaining shortfall is the normal approximation's, since the variance estimator is then unbiased. (For the running experiment, $s_1^2 = 0.0586$, $s_0^2 = 0.0537$, the estimated standard error is 0.0316, and the interval runs from $-0.064$ to 0.060: it covers 0.0243, and it also covers zero.)

Conservativeness is the price of the fundamental problem, and it cannot be removed by a cleverer estimator. Ding (2024, §4.2) notes that $S_\tau^2$ and $S_{10}$ are not identifiable from the data, because $Y_i(1)$ and $Y_i(0)$ are never observed together. The next theorem is a finite-sample form of that remark, for unbiased estimation under any design.

Theorem 4 (no unbiased variance estimator). Let the design be any probability distribution over assignments, fixed in advance (not depending on the potential outcomes), in which each unit receives one treatment. No function of the assignment and the observed outcomes has expectation equal to $V$ for every science table (when $V$ is the variance of $\hat\tau$ under complete randomization); the same holds for $S_\tau^2$.

Proof. Let $f(\mathbf{W}, Y_1, \ldots, Y_n)$ be any function of the observed data. Its expectation is $\sum_{\mathbf{w}}P(\mathbf{w})\,f(\mathbf{w}, y^{\mathbf w})$, where $y^{\mathbf w}_i = Y_i(w_i)$. Fix every entry of the science table except unit $i$'s two potential outcomes, and write $a = Y_i(1)$ and $b = Y_i(0)$. A term with $w_i = 1$ depends on $(a, b)$ only through $a$, and a term with $w_i = 0$ only through $b$. So the expectation has the form $G(a) + H(b)$, and for any $a \ne a'$ and $b \ne b'$ its mixed difference vanishes:

$$\big(G(a) + H(b)\big) - \big(G(a') + H(b)\big) - \big(G(a) + H(b')\big) + \big(G(a') + H(b')\big) = 0.$$

Now compute the mixed difference of $S_\tau^2$ as a function of $(a, b)$. Write $(n - 1)S_\tau^2 = \sum_j\tau_j^2 - n\tau_{\mathrm{fp}}^2$, where $\tau_i = a - b$ and $n\tau_{\mathrm{fp}} = a - b + C$ with $C$ the sum of the other units' effects. The mixed difference of $(a - b)^2$ is $-2(a - a')(b - b')$; that of $-n\tau_{\mathrm{fp}}^2 = -\frac1n(a - b + C)^2$ is $+\frac2n(a - a')(b - b')$; the other $\tau_j^2$ do not depend on $(a, b)$. So the mixed difference of $S_\tau^2$ is

$$\frac{1}{n - 1}\Big(-2 + \frac2n\Big)(a - a')(b - b') = -\frac2n\,(a - a')(b - b') \ne 0.$$

Hence no estimator is unbiased for $S_\tau^2$. In $V = S_1^2/n_1 + S_0^2/n_0 - S_\tau^2/n$, the term $S_1^2$ depends on $(a, b)$ only through $a$ and $S_0^2$ only through $b$, so their mixed differences vanish, and the mixed difference of $V$ is $\frac{2}{n^2}(a - a')(b - b') \ne 0$. No estimator is unbiased for $V$ either. $\square$

The proof shows exactly where the obstruction lies: the expectation of anything an experiment computes is a sum of a function of each unit's treated outcome and a function of its control outcome, while $V$ contains the product $Y_i(1)Y_i(0)$ through the covariance $S_{10}$. The data cannot reach a product of two outcomes that are never seen together. The same argument applies to any target whose mixed difference in some unit's two outcomes is not zero: the variances of the blocked and paired estimators in Blocking and Factorial Designs contain the same kind of term, (for blocks, $-\sum_k(n_{[k]}/n)^2S_{\tau[k]}^2/n_{[k]}$, whose mixed difference in a unit's two outcomes is not zero), so they have no unbiased estimator either.

What can be improved is the bound. $\hat V$ estimates the bound obtained by dropping $-S_\tau^2/n$, that is, by assuming the worst case $S_\tau^2 = 0$. Given only $S_1$ and $S_0$, a tighter bound holds.

Corollary 1 (a sharper bound). Under complete randomization,

$$V \le \frac1n\Big(\sqrt{\tfrac{n_0}{n_1}}\,S_1 + \sqrt{\tfrac{n_1}{n_0}}\,S_0\Big)^2 \le E[\hat V],$$

with equality on the left if and only if the two columns are perfectly positively linearly related (correlation 1, or one column constant), and on the right if and only if $S_1 = S_0$.

Proof. By Theorem 2, $V = \frac{p(1 - p)}{n}\big(S_1^2/p^2 + S_0^2/(1 - p)^2 + 2S_{10}/(p(1 - p))\big)$. The Cauchy–Schwarz inequality gives $S_{10} \le S_1S_0$, with equality exactly in the stated cases, so $V \le \frac{p(1 - p)}{n}\big(S_1/p + S_0/(1 - p)\big)^2$, which is the middle expression because $\sqrt{(1 - p)/p} = \sqrt{n_0/n_1}$. Expanding the square, $E[\hat V]$ minus the middle expression is $\frac1n\big(S_1^2 + S_0^2 - 2S_1S_0\big) = \frac1n(S_1 - S_0)^2 \ge 0$. $\square$

Ding (2024, Problem 4.5) poses this bound as an exercise and reports that Aronow et al. (2014) derived the sharpest possible bound given the two marginal distributions. On Cranfield the bound is 0.000969, a ratio of 1.057 to $V$: hardly better than $E[\hat V]$, because $S_1$ and $S_0$ are close. It helps when the two groups' spreads differ.

The superpopulation view

Suppose now that the $n$ units are a random sample from a large population: the vectors $(Y_i(1), Y_i(0))$ are independent and identically distributed, with variances $\sigma_1^2$, $\sigma_0^2$ and average effect $\tau = E[Y(1) - Y(0)]$, and the assignment is a complete randomization independent of them.

Theorem 5 (the superpopulation version). Then $E[\hat\tau] = \tau$, $\operatorname{Var}(\hat\tau) = \sigma_1^2/n_1 + \sigma_0^2/n_0$, and $E[\hat V] = \operatorname{Var}(\hat\tau)$: Neyman's estimator is unbiased.

Proof. Condition on the sampled units, that is, on the science table. Theorems 1 to 3 give $E[\hat\tau \mid \text{table}] = \tau_{\mathrm{fp}}$, $\operatorname{Var}(\hat\tau \mid \text{table}) = S_1^2/n_1 + S_0^2/n_0 - S_\tau^2/n$ and $E[\hat V \mid \text{table}] = S_1^2/n_1 + S_0^2/n_0$. For independent and identically distributed draws, the sample variances with divisor $n - 1$ are unbiased: $E[S_1^2] = \sigma_1^2$, $E[S_0^2] = \sigma_0^2$ and $E[S_\tau^2] = \sigma_\tau^2 = \operatorname{Var}(Y(1) - Y(0))$; and $E[\tau_{\mathrm{fp}}] = \tau$ with $\operatorname{Var}(\tau_{\mathrm{fp}}) = \sigma_\tau^2/n$. By the laws of total expectation and total variance (Conditional Expectation and the Laws of Total Expectation and Variance),

$$E[\hat\tau] = E[\tau_{\mathrm{fp}}] = \tau, \qquad \operatorname{Var}(\hat\tau) = E\Big[\frac{S_1^2}{n_1} + \frac{S_0^2}{n_0} - \frac{S_\tau^2}{n}\Big] + \operatorname{Var}(\tau_{\mathrm{fp}}) = \frac{\sigma_1^2}{n_1} + \frac{\sigma_0^2}{n_0} - \frac{\sigma_\tau^2}{n} + \frac{\sigma_\tau^2}{n},$$

and $E[\hat V] = E\big[E[\hat V \mid \text{table}]\big] = \sigma_1^2/n_1 + \sigma_0^2/n_0$. $\square$

The conservative term has not vanished; it has been spent. The part of the variance that the finite-population estimator over-counts, $S_\tau^2/n$, is exactly the variance with which the sample's own average effect $\tau_{\mathrm{fp}}$ scatters around the population's $\tau$. Ding (2024, §9.1) states the result. In the terms of his Definitions 9.1 and 9.2, randomization secures conclusions about the units in the experiment (internal validity), and how the units were obtained decides whether the conclusions extend to a broader population (external validity); Data Collection and Sampling describes how samples are drawn so that they do.

Check on Cranfield. The script treats the 225 queries as an infinite population, each equally likely, so that $\sigma_1^2$ and $\sigma_0^2$ are the column variances with divisor 225. Each simulated experiment draws $n = 60$ queries with replacement and stems 30 of them; Theorem 5 predicts $\operatorname{Var}(\hat\tau) = 0.003622$. Over 30 seeds of 20,000 experiments, the observed variance divided by this has median 1.002 (interquartile range 0.996 to 1.006), and the mean of $\hat V$ divided by it has median 1.000 (0.999 to 1.001), in the superpopulation, with no conservative excess.

The superpopulation form also states the identification argument that observational studies imitate. If $W$ is independent of $(Y(1), Y(0))$, then $E[Y \mid W = 1] = E[Y(1) \mid W = 1] = E[Y(1)]$ and likewise for the control, so $\tau = E[Y \mid W = 1] - E[Y \mid W = 0]$: an average causal effect equals a difference of observable means. Confounding and Causal Diagrams asks when independence can be had conditionally on measured covariates instead of by design.

Regression adjustment

A covariate measured before treatment can sharpen the estimate. The TF-IDF average precision of a query, from a system not in the experiment, measures how hard the query is; its correlation with $Y(0)$ over the 225 queries is 0.868. An experiment that happens to stem the harder queries will underestimate the effect, as the running experiment did (Randomized Experiments measures its imbalance); a covariate that tracks difficulty measures part of such an imbalance, and adjustment removes that part on average. Here is the idea in one sentence: subtract from the difference in outcomes a multiple of the difference in the covariate, which has mean zero, and choose the multiple to cancel as much of the outcomes' chance imbalance as possible.

Let $X_i$ be a covariate that the treatment cannot change (measured before assignment), with mean $\bar X$ and variance $S_X^2$, and write $\bar X_1$ and $\bar X_0$ for its means in the treated and control groups. Let $S_{1X}$ and $S_{0X}$ be the covariances of $Y(1)$ and $Y(0)$ with $X$, and $\beta_1 = S_{1X}/S_X^2$, $\beta_0 = S_{0X}/S_X^2$ the slopes of the two population least-squares lines of each column on $X$.

Theorem 6 (regression adjustment with fixed slopes). For a fixed number $\beta$, let $\hat\tau(\beta) = \hat\tau - \beta(\bar X_1 - \bar X_0)$. Under complete randomization:

  1. $E[\hat\tau(\beta)] = \tau_{\mathrm{fp}}$ for every $\beta$;
  2. $\operatorname{Var}(\hat\tau(\beta)) = V - 2\beta\Big(\dfrac{S_{1X}}{n_1} + \dfrac{S_{0X}}{n_0}\Big) + \beta^2S_X^2\Big(\dfrac{1}{n_1} + \dfrac{1}{n_0}\Big)$;
  3. the variance is smallest at $\beta^* = (1 - p)\beta_1 + p\beta_0$, where it equals $V - \dfrac{\beta^{*2}S_X^2}{n\,p(1 - p)}$, and, when $\beta^* \ne 0$, it is below $V$ exactly when $\beta$ lies strictly between 0 and $2\beta^*$;
  4. adjusting each group with its own fixed slope, $\big(\bar Y_1 - \beta_1'(\bar X_1 - \bar X)\big) - \big(\bar Y_0 - \beta_0'(\bar X_0 - \bar X)\big)$, equals $\hat\tau(\beta)$ with $\beta = (1 - p)\beta_1' + p\beta_0'$.

Proof. $\hat\tau(\beta)$ is the difference in means computed from the modified science table $\tilde Y_i(w) = Y_i(w) - \beta X_i$, whose unit effects are unchanged. Theorem 1 applied to that table gives part 1. Theorem 2 applied to it gives $\tilde S_1^2/n_1 + \tilde S_0^2/n_0 - S_\tau^2/n$ with $\tilde S_w^2 = S_w^2 - 2\beta S_{wX} + \beta^2S_X^2$, which is part 2. Part 2 is a quadratic in $\beta$ with leading coefficient $S_X^2(1/n_1 + 1/n_0) = S_X^2/(np(1 - p))$, minimized at

$$\beta^* = \frac{S_{1X}/n_1 + S_{0X}/n_0}{S_X^2/(np(1 - p))} = \frac{(1 - p)S_{1X} + p\,S_{0X}}{S_X^2} = (1 - p)\beta_1 + p\beta_0,$$

and its value there is $V - \beta^{*2}S_X^2/(np(1 - p))$. The quadratic equals $V$ at $\beta = 0$ and $\beta = 2\beta^*$ and lies below $V$ strictly between them. For part 4, $\bar X = p\bar X_1 + (1 - p)\bar X_0$ gives $\bar X_1 - \bar X = (1 - p)(\bar X_1 - \bar X_0)$ and $\bar X_0 - \bar X = -p(\bar X_1 - \bar X_0)$, so the two adjustments combine to $\big((1 - p)\beta_1' + p\beta_0'\big)(\bar X_1 - \bar X_0)$. $\square$

The optimal slope weights each group's slope by the share of the other group. The smaller group's mean covariate lies farther from $\bar X$, so its adjustment matters more, a point Lin (2013, remark (ii) after Corollary 1.1) makes for the estimated version.

On Cranfield, $\beta_1 = 0.960$ and $\beta_0 = 0.878$. With 113 treated queries, $\beta^* = 0.919$ and the optimally adjusted estimator has a variance 5.87 times smaller than the unadjusted one: a pre-treatment measure of difficulty removes most of the imbalance between the groups.

Estimated slopes: Freedman's critique and Lin's answer. The slopes are not known and must be fitted, and then exactness is lost. Two estimators are common.

  • The pooled regression (analysis of covariance) fits $Y_i$ on $1$, $W_i$ and $X_i$ by ordinary least squares and reports the coefficient of $W_i$. Freedman (2008a) analyzed it under the randomization model of this article, with no regression model assumed, and argued, in the words of Lin's (2013) abstract, that adjustment "can lead to worsened asymptotic precision, invalid measures of precision, and small-sample bias"; Lin's abstract and summary (§1) cover this paper and a companion on several treatments (Freedman, 2008b); in that summary these are a possible loss of precision, an inconsistent conventional OLS standard error, and a bias in small samples. The pooled slope tends to $p\beta_1 + (1 - p)\beta_0$ (Lin, 2013, remark after Corollary 1.2), which weights the groups the wrong way round compared with $\beta^*$.
  • Lin's estimator fits $Y_i$ on $1$, $W_i$, $X_i - \bar X$ and the interaction $W_i(X_i - \bar X)$, which is the same as fitting a separate line in each group and comparing them at $\bar X$, as in part 4 of Theorem 6 with estimated slopes. Lin (2013, Theorem 1 and Corollaries 1.1 and 1.2) proves, under Freedman's regularity conditions, that it is asymptotically normal and never less precise asymptotically than either the unadjusted or the pooled estimator; it and the pooled estimator have equal asymptotic precision when $p = 1/2$ or $\beta_1 = \beta_0$ (Corollary 1.2). His Theorem 2 shows that the sandwich (Huber–White) standard error is consistent or asymptotically conservative for both regressions. For the unadjusted difference in means, the HC2 version of the sandwich standard error equals $\sqrt{\hat V}$ exactly (Ding, 2024, §4.4).

These are cited, not proved here; Theorem 6 is the exact core of the argument, and the estimated slopes converge to the fixed ones. The script measures the estimated versions on Cranfield over 30 seeds of 5,000 assignments, as a fraction of the exact unadjusted variance $V$:

Treated queries Optimal fixed slope (Theorem 6) Lin's estimator Pooled regression
113 of 225 0.170 0.174 (0.172 to 0.177) 0.174 (0.171 to 0.176)
45 of 225 0.169 0.174 (0.173 to 0.177) 0.173 (0.171 to 0.174)

The ranges are interquartile ranges over the seeds. With 45 treated queries $\beta^* = 0.943$, while the pooled slope tends to 0.894; at that fixed slope the variance would be larger than at $\beta^*$ by a factor of 1.0135, a loss of 1.35%. In this simulation that small asymptotic advantage of Lin's estimator is outweighed by the cost of fitting two slopes, one of them from 45 queries, and the pooled regression came out slightly ahead. The slopes of the two columns are close here, which is the case Lin's Corollary 1.2 identifies as harmless for the pooled regression. Both adjusted estimators came within a few percent of the optimal fixed slope.

The covariate must come before the treatment. Theorem 6 needs $X_i$ to be the same whichever treatment unit $i$ receives. A variable measured after treatment, and affected by it, has two potential values; subtracting a multiple of its observed difference can remove part of the effect itself and introduces bias. Confounding and Causal Diagrams treats such variables (mediators and colliders) in general.

What is proved here, what is cited, what the experiment shows

Proved here. That the missing potential outcome is unrestricted by the data (Proposition 1), and the single-unit unbiased estimator. The moments of the assignment indicators and the variance of a weighted count under complete randomization (Lemmas 1 and 2). Neyman's three results: the difference in means is unbiased (Theorem 1), its exact variance in two forms (Theorem 2), and the expectation of the usual variance estimator, conservative by exactly $S_\tau^2/n$ (Theorem 3). That no estimator of the variance or of $S_\tau^2$ is unbiased, under any fixed design that gives each unit one treatment (Theorem 4), a finite-sample form of the non-identifiability of $S_\tau^2$ that Ding (2024, §4.2) notes. The sharper bound (Corollary 1). The superpopulation version, in which the estimator is unbiased for the population effect and the variance estimator is unbiased (Theorem 5), given the laws of total expectation and variance proved on the linked page. Regression adjustment with fixed slopes: unbiasedness, exact variance, the optimal slope and the range of slopes that help (Theorem 6).

Cited. The potential-outcomes notation and Neyman's theorem as Ding (2024) reports them from Neyman (1923); SUTVA as Rubin's (1980) assumption, through Ding; the science table and the aspirin example from Ding; the non-identifiability of $S_\tau^2$ and $S_{10}$ (Ding, 2024, §4.2); the sharp bound of Aronow et al. (2014), as Ding reports it; Rubin's (1978) phrase for the fundamental problem, as Dasgupta, Pillai and Rubin (2015) quote it; Freedman's (2008a, 2008b) results as Lin (2013) summarizes them; Lin's Theorems 1 and 2 and Corollaries 1.1 and 1.2 on estimated slopes; the equality of the HC2 standard error and $\sqrt{\hat V}$ (Ding, 2024, §4.4).

What the experiments show. All quantities of the Cranfield science table, computed exactly. For one drawn assignment, an estimate of the wrong sign whose interval covers the truth. Over 30 seeds of 20,000 assignments, the agreement of the simulated mean and variance with Theorems 1 and 2, the coverage of the 95% interval (0.955 for the real table, 0.948 for a constant-effect table), and the superpopulation check of Theorem 5. The variances of Lin's and the pooled regression estimators with estimated slopes, which are observations for this table and these group sizes.

Worked problems

These problems are posed here on a four-unit science table small enough to enumerate; the article's script checks every number exactly.

Problem 1 (a four-unit experiment, enumerated). Four units have $Y(1) = (7, 5, 4, 2)$ and $Y(0) = (3, 4, 1, 2)$. Two are treated by complete randomization. List the difference in means for every assignment, and verify Theorems 1 to 3.

Solution. The unit effects are $\tau = (4, 1, 3, 0)$, so $\tau_{\mathrm{fp}} = 2$. There are $\binom42 = 6$ assignments. Treating $\{1, 2\}$ gives $(7 + 5)/2 - (1 + 2)/2 = 9/2$; in the order $\{1,2\}, \{1,3\}, \{1,4\}, \{2,3\}, \{2,4\}, \{3,4\}$ the six estimates are 9/2, 5/2, 2, 2, 3/2, -1/2. Their mean is $12/6 = 2 = \tau_{\mathrm{fp}}$ (Theorem 1). Their variance (divisor 6, since the six assignments are equally likely) is $\frac16\big((5/2)^2 + (1/2)^2 + 0 + 0 + (1/2)^2 + (5/2)^2\big) = 13/6$. The column variances are $S_1^2 = 13/3 = 4.3333$ (the values 7, 5, 4, 2 have mean 4.5 and squared deviations summing to 13), $S_0^2 = 5/3 = 1.6667$, and $S_\tau^2 = 10/3 = 3.3333$. Theorem 2 gives $\frac{13/3}{2} + \frac{5/3}{2} - \frac{10/3}{4} = \frac{13}{6} + \frac{5}{6} - \frac{5}{6} = 13/6$, matching the enumeration. Theorem 3 gives $E[\hat V] = 13/6 + 5/6 = 3$, which is also the average of the six values of $s_1^2/2 + s_0^2/2$ computed by the script, so $\hat V$ overstates the variance by $S_\tau^2/n = 5/6$.

Problem 2 (the effect on the treated is random; the bound). In Problem 1, find the effect on the treated for each assignment, and compare $V$ with the bound of Corollary 1.

Solution. The average of $\tau_i$ over the treated pair, in the same order of assignments, is 5/2, 7/2, 2, 2, 1/2, 3/2. It ranges from 1/2 to 7/2, and its mean over the assignments is 2, equal to $\tau_{\mathrm{fp}}$, because each unit is treated in half of the assignments. With $n_1 = n_0 = 2$ the bound is $\frac14(S_1 + S_0)^2 = \frac14\big(\sqrt{13/3} + \sqrt{5/3}\big)^2 = 2.8437$. So $V = 13/6 \approx 2.1667 \le 2.8437 \le 3 = E[\hat V]$. The bound is not attained, since the columns are not perfectly correlated, and the gap to $E[\hat V]$ is $\frac14(S_1 - S_0)^2 = 0.1563$, as the proof of Corollary 1 says.

Problem 3 (adjusting for a covariate). Each unit of Problem 1 also has a pre-treatment covariate $X = (3, 2, 1, 0)$. Find the optimal fixed slope of Theorem 6 and the adjusted estimates for all six assignments.

Solution. $\bar X = 3/2$ and $S_X^2 = 5/3$. The covariances are $S_{1X} = \frac13\big((2.5)(1.5) + (0.5)(0.5) + (-0.5)(-0.5) + (-2.5)(-1.5)\big) = 8/3$ and $S_{0X} = \frac13\big((0.5)(1.5) + (1.5)(0.5) + (-1.5)(-0.5) + (-0.5)(-1.5)\big) = 1$, so $\beta_1 = 8/5$ and $\beta_0 = 3/5$. With $p = 1/2$, $\beta^* = \frac12\cdot\frac85 + \frac12\cdot\frac35 = 11/10$. The adjusted estimate is $\hat\tau - \frac{11}{10}(\bar X_1 - \bar X_0)$. For the treated pair $\{1, 2\}$, $\bar X_1 - \bar X_0 = 5/2 - 1/2 = 2$, so the estimate is $9/2 - 11/5 = 23/10$. In the order of Problem 1 the six adjusted estimates are 23/10, 7/5, 2, 2, 13/5, 17/10. Their mean is 2, and their variance is 3/20, against 13/6 unadjusted. Theorem 6 predicts $V - \beta^{*2}S_X^2/(np(1 - p)) = \frac{13}{6} - \frac{121}{100}\cdot\frac{5/3}{1} = \frac{13}{6} - \frac{121}{60} = \frac{9}{60} = 3/20$, as the enumeration finds. The covariate tracks the outcomes closely here, which is why the variance falls by a factor of about 14.

Summary

  • Each unit has a potential outcome under each treatment; SUTVA (no interference, no hidden versions) makes two per unit enough. The unit effect is $\tau_i = Y_i(1) - Y_i(0)$, and the average effect $\tau_{\mathrm{fp}}$ is the difference of the column means.
  • An experiment sees one outcome per unit, so no unit's effect is ever observed (Proposition 1), and no estimator of $S_\tau^2$ or of the variance of the difference in means is unbiased (Theorem 4).
  • Under complete randomization the difference in means is unbiased for $\tau_{\mathrm{fp}}$ with no model (Theorem 1), its variance is $S_1^2/n_1 + S_0^2/n_0 - S_\tau^2/n$ (Theorem 2), and Neyman's estimator $s_1^2/n_1 + s_0^2/n_0$ overstates it by exactly $S_\tau^2/n$ (Theorem 3).
  • For units sampled from a population the same estimator is unbiased for the population effect and the variance estimator is unbiased (Theorem 5).
  • Adjusting for a pre-treatment covariate with the slope $(1 - p)\beta_1 + p\beta_0$ reduces the variance by $\beta^{*2}S_X^2/(np(1 - p))$ (Theorem 6); with estimated slopes, Lin's interacted regression is asymptotically never worse than no adjustment.
  • On the Cranfield table: $\tau_{\mathrm{fp}} = 0.0243$, the standard deviation of the difference in means is 0.0303 with 113 of 225 queries stemmed, the variance estimator is 5.8% too large on average, and adjustment for TF-IDF difficulty cuts the variance by a factor of almost six.

What we have learned

A causal effect is a comparison of two potential outcomes of the same unit, and an experiment never sees both. Randomization does not reveal the missing outcomes; it makes the observed group means unbiased for the two column means, and it makes the uncertainty computable, up to the one term that would need both outcomes. That term is why randomized experiments report conservative intervals, and Theorem 4 shows that no estimator can avoid it.

The argument of this article used only the first two moments of the assignment. Randomized Experiments uses its whole distribution: it shows that randomization balances every covariate, measured or not, and turns the assignment distribution into an exact test of the hypothesis that the treatment affects no unit at all. It also compares Bernoulli and complete randomization and treats online experiments.

References

  • Aronow, P. M., Green, D. P., and Lee, D. K. K. (2014). Sharp bounds on the variance in randomized experiments. The Annals of Statistics, 42(3), 850–871. https://doi.org/10.1214/13-AOS1200 (cited as reported by Ding, 2024) record
  • Cleverdon, C. W. (1967). The Cranfield tests on index language devices. Aslib Proceedings, 19(6), 173–194. https://doi.org/10.1108/eb050097 record · 11 pages
  • Dasgupta, T., Pillai, N. S., and Rubin, D. B. (2015). Causal inference from $2^K$ factorial designs by using potential outcomes. Journal of the Royal Statistical Society, Series B, 77(4), 727–753. https://doi.org/10.1111/rssb.12085 (arXiv:1211.2481) (lemma, theorem and corollary numbers are those of the arXiv version) record · 2 pages
  • Ding, P. (2024). A First Course in Causal Inference. Boca Raton: CRC Press. arXiv:2305.18793 (chapter, section and theorem numbers are those of the arXiv version) record · 6 pages
  • Freedman, D. A. (2008a). On regression adjustments to experimental data. Advances in Applied Mathematics, 40(2), 180–193. https://doi.org/10.1016/j.aam.2006.12.003 (as summarized by Lin, 2013) record
  • Freedman, D. A. (2008b). On regression adjustments in experiments with several treatments. The Annals of Applied Statistics, 2(1), 176–196. https://doi.org/10.1214/07-AOAS143 (as summarized by Lin, 2013) record
  • Lin, W. (2013). Agnostic notes on regression adjustments to experimental data: Reexamining Freedman's critique. The Annals of Applied Statistics, 7(1), 295–318. https://doi.org/10.1214/12-AOAS583 (arXiv:1208.2301) record
  • Neyman, J. (1923). On the application of probability theory to agricultural experiments. Essay on principles. Section 9. Translated and edited by D. M. Dabrowska and T. P. Speed (1990), Statistical Science, 5(4), 465–472. https://doi.org/10.1214/ss/1177012031 (cited as Ding, 2024, reports it) record
  • Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5), 688–701. https://doi.org/10.1037/h0037350 (cited as Ding, 2024, reports it) record
  • Rubin, D. B. (1978). Bayesian inference for causal effects: The role of randomization. The Annals of Statistics, 6(1), 34–58. https://doi.org/10.1214/aos/1176344064 (quoted as Dasgupta, Pillai and Rubin, 2015, quote it) record
  • Rubin, D. B. (1980). Comment on "Randomization analysis of experimental data: The Fisher randomization test" by D. Basu. Journal of the American Statistical Association, 75(371), 591–593. https://doi.org/10.2307/2287653 (cited as Ding, 2024, reports it) record · 2 pages