Blocking and Factorial Designs

How the design of an experiment, before any data are seen, decides how precise its estimates will be. Blocked (stratified) randomization proved unbiased, with its exact finite-population variance; the exact difference from complete randomization proved, which with equal blocks is a multiple of the between-blocks minus the within-blocks mean square, so that blocking helps exactly when an analysis-of-variance F ratio exceeds one; random blocks proved neutral on average, and blocks drawn from fixed strata of a superpopulation proved never to hurt in expectation. The paired design as blocks of size two, with its conservative variance estimator proved. Two-level factorial designs in the potential-outcomes model: main effects and interactions as orthogonal contrasts, the variance of every contrast under complete randomization into several groups proved, and the reason one experiment estimates every effect with the whole sample. Worked on the Cranfield science table, blocked on a measure of query difficulty and run as a 2 x 2 x 2 factorial of stemming, length normalization and the stop list. Worked problems on good and bad pairings of four units, the paired variance estimator and a 2 x 2 design enumerated over all 2,520 assignments.

The randomized stemming experiment of Potential Outcomes and the Average Treatment Effect estimates the effect of Porter stemming on the average precision of 225 Cranfield queries (Cleverdon, 1967) with a standard deviation of 0.0303, larger than the effect itself (0.0243). Most of that uncertainty comes from the queries, not the stemmer: some queries are easy under any ranker and some hard, and a random split can put more of the easy ones on one side. Sort 224 of the queries by how well a different system, TF-IDF, does on them, cut the list into eight blocks of 28, and stem 14 queries chosen at random within each block: the variance of the estimate falls by a factor of 4.78. Sort them instead by the number of terms in the query (counted after stop-word removal) and do the same, and the variance rises slightly. When does grouping units before randomizing help, by exactly how much, and can it hurt? And when an experiment has several treatments at once, can one experiment answer all of them?

A block is a group of units formed before treatment is assigned, using information that the treatment cannot change; blocked randomization (also called stratified randomization, and in agricultural experiments the randomized block design) randomizes separately within each block. Box, Hunter and Hunter's advice, "Block what you can and randomize what you cannot" (1978, p. 103, as quoted by Ding, 2024, ch. 5), summarizes the practice. A factorial design varies several treatments, called factors, at once, and assigns each unit one combination of their levels.

Here is the route. First blocked randomization, its unbiasedness and its exact variance. Then the exact comparison with complete randomization, which reduces to an analysis-of-variance comparison of block means; random blocks are neutral on average, and blocks from fixed strata cannot hurt in expectation. Then the paired design, as blocks of size two. Then two-level factorial designs: effects defined as contrasts of potential outcomes, the variance of every contrast proved, and the reason all effects share the whole sample, measured on a three-factor experiment on Cranfield. The article assumes Potential Outcomes and the Average Treatment Effect, whose notation, lemmas and theorems it uses throughout, and One-Way Analysis of Variance for mean squares.

Blocked randomization

The setting is that of the potential-outcomes article: $n$ units with potential outcomes $Y_i(1)$ and $Y_i(0)$, unit effects $\tau_i$ and average effect $\tau_{\mathrm{fp}}$. The units are partitioned into $K$ blocks; block $k$ has $n_{[k]}$ units, of which $n_{[k]1}$ are to be treated and $n_{[k]0}$ not. A blocked randomization runs an independent complete randomization in each block (Ding, 2024, Definition 5.1). Within block $k$, write $\tau_{[k]}$ for the average effect of its units, $\hat\tau_{[k]}$ for the difference in means of its treated and control units, and $V_{[k]}$ and $\hat V_{[k]}$ for Neyman's variance and variance estimator computed within the block. Since $\tau_{\mathrm{fp}} = \sum_k (n_{[k]}/n)\,\tau_{[k]}$, the natural estimator is

$$\hat\tau_{\mathrm B} = \sum_{k=1}^K \frac{n_{[k]}}{n}\,\hat\tau_{[k]}.$$

Theorem 1 (blocked randomization). Under blocked randomization, $E[\hat\tau_{\mathrm B}] = \tau_{\mathrm{fp}}$ and

$$V_{\mathrm B} = \operatorname{Var}(\hat\tau_{\mathrm B}) = \sum_k \Big(\frac{n_{[k]}}{n}\Big)^2V_{[k]}, \qquad V_{[k]} = \frac{S_{1[k]}^2}{n_{[k]1}} + \frac{S_{0[k]}^2}{n_{[k]0}} - \frac{S_{\tau[k]}^2}{n_{[k]}},$$

with the variances computed within block $k$. If every block has at least two treated and two control units, $\hat V_{\mathrm B} = \sum_k (n_{[k]}/n)^2\hat V_{[k]}$ has expectation $V_{\mathrm B} + \sum_k (n_{[k]}/n)^2S_{\tau[k]}^2/n_{[k]} \ge V_{\mathrm B}$.

Proof. Within block $k$ the assignment is a complete randomization of the block's units, so Theorems 1 to 3 of the potential-outcomes article apply to the block as a population of its own: $E[\hat\tau_{[k]}] = \tau_{[k]}$, $\operatorname{Var}(\hat\tau_{[k]}) = V_{[k]}$ and $E[\hat V_{[k]}] = V_{[k]} + S_{\tau[k]}^2/n_{[k]}$. The randomizations in different blocks are independent, so the $\hat\tau_{[k]}$ are independent; the expectation and variance of the weighted sum follow. $\square$

When every block treats the same share, $n_{[k]1} = p\,n_{[k]}$, the blocked estimator is simply the difference in means of all treated and all control units, because each treated unit then carries weight $\frac{n_{[k]}}{n}\cdot\frac{1}{p\,n_{[k]}} = \frac{1}{pn} = \frac{1}{n_1}$ (Ding, 2024, eq. 5.3). Blocking changes the design, not the estimator: it rules out the assignments in which the blocks are unevenly treated.

Blocks: the exact gain

Here is the idea in one sentence. Under complete randomization, the chance imbalance between the groups comes from all the variation among units; under blocking, the variation between blocks can no longer be split unevenly, and only the variation within blocks remains. The proof makes this exact, and also shows the small price that blocking pays.

Assume a common share $p = n_1/n$ treated in every block. The potential-outcomes article (Theorem 2) wrote Neyman's variance as $V = \frac{p(1 - p)}{n}S_a^2$, where $S_a^2$ is the variance of

$$a_i = \frac{Y_i(1)}{p} + \frac{Y_i(0)}{1 - p}.$$

For $p = 1/2$, $a_i = 2\big(Y_i(1) + Y_i(0)\big)$: a unit's level, the sum of its two outcomes. Let $\bar a_{[k]}$ and $S_{a[k]}^2$ be the mean and variance (divisor $n_{[k]} - 1$) of $a$ in block $k$, and $\bar a$ the overall mean.

Theorem 2 (blocking against complete randomization). With the same share $p$ treated in every block,

$$V_{\mathrm{CR}} - V_{\mathrm B} = \frac{p(1 - p)}{n^2(n - 1)}\Big(n\sum_k n_{[k]}\big(\bar a_{[k]} - \bar a\big)^2 - \sum_k\big(n - n_{[k]}\big)S_{a[k]}^2\Big).$$

With $K$ blocks of equal size, this is

$$V_{\mathrm{CR}} - V_{\mathrm B} = \frac{p(1 - p)(K - 1)}{n(n - 1)}\,\big(\mathrm{MSB} - \mathrm{MSW}\big),$$

where $\mathrm{MSB} = \frac{1}{K - 1}\sum_k n_{[k]}(\bar a_{[k]} - \bar a)^2$ and $\mathrm{MSW} = \frac{1}{n - K}\sum_k(n_{[k]} - 1)S_{a[k]}^2$ are the between-blocks and within-blocks mean squares of the $a_i$. Blocking reduces the variance exactly when $\mathrm{MSB} > \mathrm{MSW}$, that is, when the one-way analysis-of-variance $F$ ratio of the $a_i$ across blocks exceeds 1.

Proof. Within block $k$ the treated share is $p$, so Theorem 2 of the potential-outcomes article, applied to the block, gives $V_{[k]} = \frac{p(1 - p)}{n_{[k]}}S_{a[k]}^2$, and by Theorem 1, $V_{\mathrm B} = \sum_k\frac{n_{[k]}^2}{n^2}\cdot\frac{p(1 - p)}{n_{[k]}}S_{a[k]}^2 = \frac{p(1 - p)}{n^2}\sum_k n_{[k]}S_{a[k]}^2$. The total sum of squares splits into within- and between-block parts, as in One-Way Analysis of Variance:

$$(n - 1)S_a^2 = \sum_k(n_{[k]} - 1)S_{a[k]}^2 + B, \qquad B = \sum_k n_{[k]}(\bar a_{[k]} - \bar a)^2.$$

So

$$\frac{n^2}{p(1 - p)}\big(V_{\mathrm{CR}} - V_{\mathrm B}\big) = \frac{n}{n - 1}\Big(\sum_k(n_{[k]} - 1)S_{a[k]}^2 + B\Big) - \sum_k n_{[k]}S_{a[k]}^2 = \frac{nB}{n - 1} - \sum_k\Big(n_{[k]} - \frac{n(n_{[k]} - 1)}{n - 1}\Big)S_{a[k]}^2,$$

and $n_{[k]} - \frac{n(n_{[k]} - 1)}{n - 1} = \frac{n - n_{[k]}}{n - 1}$, which gives the first formula. With $K$ blocks of size $m = n/K$, $nB = n(K - 1)\,\mathrm{MSB}$, and since $\mathrm{MSW} = \frac1K\sum_kS_{a[k]}^2$, $\sum_k(n - m)S_{a[k]}^2 = (n - m)K\,\mathrm{MSW} = n(K - 1)\,\mathrm{MSW}$, because $(n - m)K = m(K - 1)K = n(K - 1)$. $\square$

The first term is the gain: the variation between block means that complete randomization lets into the comparison and blocking keeps out. The second is the price: each block is a smaller population, and a smaller population has a slightly larger finite-population variance for the same spread (the factor $n_{[k]}/(n_{[k]} - 1)$ against $n/(n - 1)$). Pashley and Miratrix (2022, eq. 1) give the same difference in another notation and stress that the second term can win. Ding (2024, §5.3.3) compares the two designs with large blocks, where the price vanishes; Theorem 2 is the exact version. The algebra is the within-and-between decomposition of Survey Sampling Designs (section "Stratified sampling", Theorem 4, the comparison of proportional allocation with simple random sampling, whose formula is the first formula of Theorem 2 with $(1 - f)/n$ ($n$ there the sample size) in place of $p(1 - p)/n$, the population size $N$ in place of $n$, strata sizes $N_h$ in place of blocks $n_{[k]}$, and $y_k$ in place of $a_i$), where strata play the part of blocks and a sample the part of the treated group.

Blocks on Cranfield. The units are the first 224 of the 225 Cranfield queries (224 = 8 × 28, which makes the blocks, the pairs and the factorial cells below all equal in size; query 225 is left out), and the comparison is stemming against no stemming with $p = 1/2$. On these 224 queries $\tau_{\mathrm{fp}} = 0.0244$, and complete randomization with 112 treated has $V_{\mathrm{CR}} = 0.000922$. Eight blocks of 28 are formed by sorting the queries on a covariate, ties broken by query number. The script tools/experiments/statistics/factorial-and-block-designs.py computes every number exactly from the science table:

Blocks formed by MSB of $a$ MSW of $a$ $F$ $V_{\mathrm B}$ $V_{\mathrm{CR}}/V_{\mathrm B}$
TF-IDF average precision 20.986 0.173 121.47 0.000193 4.782
number of relevant documents 1.485 0.805 1.85 0.000898 1.027
number of query terms 0.674 0.831 0.81 0.000927 0.994
every eighth query in TF-IDF order 0.143 0.848 0.17 0.000947 0.974

TF-IDF difficulty is a strong blocking variable: its correlation with $a$ is 0.911, and blocking on it divides the variance by 4.782. The number of query terms (the query's words after stop-word removal and stemming, counted with repeats, keeping those that occur in the collection) is nearly unrelated to $a$ (correlation $-0.050$); its blocks differ less than chance would make them, $F = 0.81$, and blocking on it increases the variance by about 0.6%. The last row is a deliberately bad design: dealing the queries in difficulty order into eight blocks, like cards, makes every block a miniature of the whole and the block means nearly equal, so the price is paid and nothing is gained (variance 2.7% higher).

The unit level a = 2(Y(1) + Y(0)) of each of the 224 queries for the stemming comparison. Left: against the query's rank in TF-IDF average precision (rank 1 is the lowest), with the eight blocks of 28 adjacent ranks and their means; notice how far apart the block means are, which is the variation blocking keeps out of the comparison. Right: the same values dealt into eight blocks, every eighth rank; the block means are nearly level and each block is as spread as the whole, so blocking removes nothing.
The unit level a = 2(Y(1) + Y(0)) of each of the 224 queries for the stemming comparison. Left: against the query's rank in TF-IDF average precision (rank 1 is the lowest), with the eight blocks of 28 adjacent ranks and their means; notice how far apart the block means are, which is the variation blocking keeps out of the comparison. Right: the same values dealt into eight blocks, every eighth rank; the block means are nearly level and each block is as spread as the whole, so blocking removes nothing.

The script also checks Theorem 1 for the TF-IDF blocks by simulation: over 30 seeds of 20,000 blocked assignments, the variance of $\hat\tau_{\mathrm B}$ divided by $V_{\mathrm B}$ has median 1.000 (interquartile range 0.993 to 1.008).

Example (the conservative excess does not shrink). The 95% interval $\hat\tau_{\mathrm B} \pm 1.96\sqrt{\hat V_{\mathrm B}}$ covered $\tau_{\mathrm{fp}}$ in a median 0.971 of the simulated assignments (interquartile range 0.970 to 0.972), more than the 0.955 of complete randomization. The reason is in Theorem 1: blocking removes the variation of the unit levels between blocks but not the variation of the unit effects, so the excess $\sum_k(n_{[k]}/n)^2S_{\tau[k]}^2/n_{[k]}$, about $S_\tau^2/n$, is now a larger share of a smaller variance. Here $E[\hat V_{\mathrm B}]/V_{\mathrm B} = 1.266$.

Can blocking hurt on average? Two results answer this, for two ways in which blocks arise.

Corollary 1 (random blocks are neutral on average). With the same share $p$ treated in every block (as in Theorem 2) and at least two units per block, if the blocks, of fixed sizes, are formed by partitioning the units uniformly at random, independently of the science table, then the average of $V_{\mathrm B}$ over the partitions equals $V_{\mathrm{CR}}$.

Proof. Under a uniformly random partition, the units of block $k$ are a simple random sample of $n_{[k]}$ of the $n$ units, so the expectation of their variance is the population's: $E[S_{a[k]}^2] = S_a^2$ (the first statement of Theorem 3 of the potential-outcomes article, applied to the numbers $a_i$). Hence $E[V_{\mathrm B}] = \frac{p(1 - p)}{n^2}\sum_k n_{[k]}S_a^2 = \frac{p(1 - p)}{n}S_a^2 = V_{\mathrm{CR}}$. $\square$

A blocking variable unrelated to the outcomes behaves like a random partition. For 30 random partitions of the 224 queries into eight blocks of 28, the ratio $V_{\mathrm{CR}}/V_{\mathrm B}$ had median 0.995 (interquartile range 0.988 to 1.004): a given random partition can lose or gain a little, as the query-terms blocks lost 0.6%. Pashley and Miratrix (2022, Theorem 3.1) prove a related statement for a covariate independent of the potential outcomes in a superpopulation.

Corollary 2 (blocks from fixed strata never hurt in expectation). With the same share $p$ treated in every block (as in Theorem 2) and at least two units per block, suppose the units of block $k$ are drawn independently from a stratum of a superpopulation in which $a$ has mean $\mu_{[k]}$ and variance $\sigma_{[k]}^2$, with the block sizes fixed in advance. Then, averaging the finite-population variances over the draw of the units,

$$E\big[V_{\mathrm{CR}} - V_{\mathrm B}\big] = \frac{p(1 - p)}{n(n - 1)}\sum_k n_{[k]}\big(\mu_{[k]} - \bar\mu\big)^2 \ge 0, \qquad \bar\mu = \sum_k\frac{n_{[k]}}{n}\mu_{[k]}.$$

Proof. Take expectations in the first formula of Theorem 2. Within a block the units are independent and identically distributed, so $E[S_{a[k]}^2] = \sigma_{[k]}^2$. For the between-block sum, $B = \sum_k n_{[k]}\bar a_{[k]}^2 - n\bar a^2$ with $\bar a = \sum_k (n_{[k]}/n)\bar a_{[k]}$; the block means are independent with $E[\bar a_{[k]}^2] = \mu_{[k]}^2 + \sigma_{[k]}^2/n_{[k]}$, and $E[\bar a^2] = \bar\mu^2 + \sum_k n_{[k]}\sigma_{[k]}^2/n^2$. So

$$E[B] = \sum_k n_{[k]}\mu_{[k]}^2 - n\bar\mu^2 + \sum_k\sigma_{[k]}^2 - \sum_k\frac{n_{[k]}}{n}\sigma_{[k]}^2 = \sum_k n_{[k]}(\mu_{[k]} - \bar\mu)^2 + \sum_k\frac{n - n_{[k]}}{n}\sigma_{[k]}^2,$$

and $nE[B] - \sum_k(n - n_{[k]})\sigma_{[k]}^2 = n\sum_k n_{[k]}(\mu_{[k]} - \bar\mu)^2$. Multiplying by $\frac{p(1 - p)}{n^2(n - 1)}$ gives the result. $\square$

This is Pashley and Miratrix's (2022) Theorem 3.2 in the present notation. They compare the variances of the two estimators over the draw of the units as well as the assignment, while Corollary 2 compares the expected finite-population variances; the two comparisons agree, because given the units both estimators are unbiased for the same $\tau_{\mathrm{fp}}$, so the term $\operatorname{Var}(\tau_{\mathrm{fp}})$ of the law of total variance is the same for both designs and cancels. It explains the two literatures they reconcile: in a fixed finite population blocking can cost a little, and over repeated draws from fixed strata the cost is exactly offset, on average, by the chance variation of block means that it removes.

Pairs: blocks of size two

The finest blocking puts two units in each block and treats one of them: the paired design (matched-pairs experiment). It is the case $K = n/2$, $m = 2$, $p = 1/2$ of Theorem 2. For Cranfield, pairing adjacent queries in TF-IDF order gives 112 pairs, with $\mathrm{MSB} = 1.500$ and $\mathrm{MSW} = 0.159$ for $a$, and a variance 5.21 times smaller than complete randomization, $V_{\mathrm{pair}} = 0.000177$; finer blocks have bought a little more than the eight blocks did.

A pair has one treated and one control unit, so the within-block variance estimators of Theorem 1 do not exist. The variance must be estimated from the spread of the pair differences instead. Let pair $j$ ($j = 1, \ldots, m$, with $m = n/2$) contain units $u$ and $v$, let $d_j$ be the treated unit's outcome minus the control's, and let $\tau_j = (\tau_u + \tau_v)/2$ be the pair's average effect. The estimator is $\hat\tau_{\mathrm P} = \bar d = \frac1m\sum_j d_j$.

Theorem 3 (the paired design). With a fair coin deciding, independently in each pair, which unit is treated:

  1. $E[\hat\tau_{\mathrm P}] = \tau_{\mathrm{fp}}$, and $\operatorname{Var}(\hat\tau_{\mathrm P}) = \frac{1}{m^2}\sum_j V_j$ with $V_j = \frac14\big((Y_u(1) + Y_u(0)) - (Y_v(1) + Y_v(0))\big)^2$;
  2. $\hat V_{\mathrm P} = \frac{1}{m(m - 1)}\sum_j(d_j - \bar d)^2$ satisfies $E[\hat V_{\mathrm P}] = \operatorname{Var}(\hat\tau_{\mathrm P}) + \frac{1}{m(m - 1)}\sum_j(\tau_j - \tau_{\mathrm{fp}})^2$.

Proof. In pair $j$, $d_j$ is $Y_u(1) - Y_v(0)$ or $Y_v(1) - Y_u(0)$, each with probability 1/2. Its mean is $\frac12(Y_u(1) - Y_u(0) + Y_v(1) - Y_v(0)) = \tau_j$, and its variance, a quarter of the squared difference of its two values, is $\frac14\big((Y_u(1) + Y_u(0)) - (Y_v(1) + Y_v(0))\big)^2 = V_j$. The $d_j$ are independent, and $\frac1m\sum_j\tau_j = \tau_{\mathrm{fp}}$; this gives part 1. For part 2, $\sum_j(d_j - \bar d)^2 = \sum_j d_j^2 - m\bar d^2$, so

$$E\Big[\sum_j(d_j - \bar d)^2\Big] = \sum_j\big(V_j + \tau_j^2\big) - m\Big(\frac{1}{m^2}\sum_jV_j + \tau_{\mathrm{fp}}^2\Big) = \Big(1 - \frac1m\Big)\sum_jV_j + \sum_j(\tau_j - \tau_{\mathrm{fp}})^2.$$

Dividing by $m(m - 1)$ gives $\frac{1}{m^2}\sum_jV_j + \frac{1}{m(m - 1)}\sum_j(\tau_j - \tau_{\mathrm{fp}})^2$. $\square$

Part 2 and this proof are Ding's (2024, Theorem 7.1); part 1 is his Problem 7.1. Ding credits the comparison of paired and completely randomized designs to Imai (2008). The estimator is the squared standard error of the paired $t$-test, $s_d^2/m$, which The Paired Samples t-Test derives for a sample of pairs from a population. In the randomization model it is conservative, by the spread of the pair effects rather than of the unit effects. On Cranfield the excess is 0.000057, so $E[\hat V_{\mathrm P}]/V_{\mathrm{pair}} = 1.320$. Over 30 seeds of 20,000 paired assignments, the variance of $\hat\tau_{\mathrm P}$ divided by $V_{\mathrm{pair}}$ had median 1.001 (interquartile range 0.996 to 1.004), the mean of $\hat V_{\mathrm P}$ divided by its expectation 1.000 (0.999 to 1.000), and the 95% interval covered $\tau_{\mathrm{fp}}$ in a median 0.974 of assignments (0.973 to 0.975).

The paired-samples article measures the cost of pairing in a normal model: half the degrees of freedom of two independent groups, which pairs pay back only when the two members are correlated. Theorem 2 gives the same lesson without a model: pairs on an irrelevant variable have $\mathrm{MSB}$ near $\mathrm{MSW}$, and pay the price for nothing. Pairs also limit the randomization test: with $m$ pairs there are only $2^m$ assignments, and Ding (2024, §7.6) notes that this can make a paired experiment unable to reach a small p-value with few units where a completely randomized one can; Permutation Tests (Worked problem 3) finds the smallest attainable two-sided p-value of a sign-flip test.

Example (when every unit receives both treatments). The retrieval comparisons of Comparing Systems with Significance Tests and the model comparisons of Paired Comparisons of Models run both systems on every query or question. That is the limit of blocking: each unit is its own block and receives every treatment, so for the units at hand $\tau_{\mathrm{fp}}$ is observed exactly. Their standard errors describe a different uncertainty, the variation over new samples of queries or questions from a population of them. People and plots of land rarely allow this, which is why the blocked and paired designs above exist.

Several factors at once

A ranking function has many settings, and a treatment in medicine or education often has several components. Varying one at a time answers one question per experiment. A factorial design varies all of them in one experiment. The potential-outcomes formulation used here is that of Dasgupta, Pillai and Rubin (2015).

There are $K$ factors, each with a low level $-1$ and a high level $+1$, so $J = 2^K$ treatment combinations $z \in \{-1, +1\}^K$. Unit $i$ has a potential outcome $Y_i(z)$ for each combination (SUTVA now meaning no interference and one version of each combination), collected in a vector $\mathbf Y_i$ of length $J$. A factorial effect is defined by a vector $g$ of $\pm1$ entries indexed by the combinations, half $+1$ and half $-1$:

  • the main effect of factor $k$ has $g(z) = z_k$;
  • the interaction of factors $k$ and $l$ has $g(z) = z_kz_l$, and higher interactions multiply more factors' levels.

Unit $i$'s effect is $\tau_{ij} = 2^{-(K-1)}g_j^{\mathsf T}\mathbf Y_i$: the mean of its outcomes over the $2^{K-1}$ combinations with $g_j = +1$ minus the mean over those with $g_j = -1$. The average factorial effect is $\tau_j = \frac1N\sum_i\tau_{ij}$, with $N$ units (Dasgupta et al., 2015, eqs. 1 and 2). For the main effect of factor $k$, this compares the average outcome at its high level with the average at its low level, averaging over all levels of the other factors. The $2^K - 1$ vectors $g_j$ and the vector of ones are mutually orthogonal: the product of two different ones is again a product of factor levels, which is $+1$ on exactly half of the combinations.

In a completely randomized factorial experiment, $N = rJ$ units are divided at random into $J$ groups of $r$, one per combination. Let $\bar Y_z$ be the mean outcome of group $z$ and $S_z^2$ the finite-population variance (divisor $N - 1$) of $Y_i(z)$ over all $N$ units. The estimate of effect $j$ is $\hat\tau_j = 2^{-(K-1)}g_j^{\mathsf T}\bar{\mathbf Y}$. Its variance follows from a general theorem about several groups.

Theorem 4 (contrasts under complete randomization into several groups). Let $N$ units be divided at random into groups of fixed sizes $n_z$ ($\sum_z n_z = N$), every division equally likely, and let $c_z$ be constants. Then $\sum_z c_z\bar Y_z$ is unbiased for $\sum_z c_z\bar Y(z)$, and

$$\operatorname{Var}\Big(\sum_z c_z\bar Y_z\Big) = \sum_z\frac{c_z^2S_z^2}{n_z} - \frac{S_c^2}{N},$$

where $S_c^2$ is the finite-population variance (divisor $N - 1$) of the unit-level contrasts $\sum_z c_zY_i(z)$.

Proof. Let $W_i(z) = 1$ if unit $i$ is in group $z$. Each group, on its own, is a simple random sample of size $n_z$, so $E[W_i(z)] = n_z/N$ and, by Lemma 2 and the proof of Theorem 3 in the potential-outcomes article, $E[\bar Y_z] = \bar Y(z)$ and $\operatorname{Var}(\bar Y_z) = \big(\frac{1}{n_z} - \frac1N\big)S_z^2$. For two different groups $z \ne z'$: a unit cannot be in both, so $\operatorname{Cov}(W_i(z), W_i(z')) = -\frac{n_zn_{z'}}{N^2}$; for $i \ne j$, $P(W_i(z) = W_j(z') = 1) = \frac{n_z}{N}\cdot\frac{n_{z'}}{N - 1}$, so $\operatorname{Cov}(W_i(z), W_j(z')) = \frac{n_zn_{z'}}{N^2(N - 1)}$. Writing $y_i = Y_i(z)$ and $y_i' = Y_i(z')$,

$$\operatorname{Cov}(\bar Y_z, \bar Y_{z'}) = \frac{1}{n_zn_{z'}}\sum_{i,j}\operatorname{Cov}(W_i(z), W_j(z'))\,y_iy_j' = \frac{1}{N^2}\Big(-\sum_iy_iy_i' + \frac{1}{N - 1}\Big(\sum_iy_i\sum_jy_j' - \sum_iy_iy_i'\Big)\Big) = -\frac{S_{zz'}}{N},$$

where $S_{zz'}$ is the covariance (divisor $N - 1$) of $Y(z)$ and $Y(z')$; the last step uses $\sum_iy_iy_i' - \frac1N\sum_iy_i\sum_jy_j' = (N - 1)S_{zz'}$. Then

$$\operatorname{Var}\Big(\sum_zc_z\bar Y_z\Big) = \sum_zc_z^2\Big(\frac{1}{n_z} - \frac1N\Big)S_z^2 - \frac1N\sum_{z \ne z'}c_zc_{z'}S_{zz'} = \sum_z\frac{c_z^2S_z^2}{n_z} - \frac1N\Big(\sum_zc_z^2S_z^2 + \sum_{z \ne z'}c_zc_{z'}S_{zz'}\Big),$$

and the bracket is the variance of $\sum_zc_zY_i(z)$. $\square$

With two groups and $c = (1, -1)$ this is Theorem 2 of the potential-outcomes article. The indicator covariances are Dasgupta et al.'s (2015) Lemmas 3.1 and 3.2.

Corollary 3 (factorial effects). In a completely randomized $2^K$ experiment with $r$ units per combination, $\hat\tau_j$ is unbiased for $\tau_j$ and

$$\operatorname{Var}(\hat\tau_j) = \frac{1}{2^{2(K-1)}\,r}\sum_zS_z^2 - \frac{S_j^2}{N},$$

where $S_j^2$ is the variance of the unit effects $\tau_{ij}$. If the effects are additive, so that every $Y_i(z) - Y_i(z')$ is the same for all units, then all $S_z^2$ equal a common $S^2$, every $S_j^2 = 0$, and $\operatorname{Var}(\hat\tau_j) = 4S^2/N$ for every effect.

Proof. Apply Theorem 4 with $c_z = 2^{-(K-1)}g_j(z)$ and $n_z = r$: $\sum_zc_z^2S_z^2/r = 2^{-2(K-1)}\sum_zS_z^2/r$, and the unit-level contrast is $\tau_{ij}$. Under additivity $Y_i(z) - Y_i(z')$ is the same constant for every unit, so the variances agree and each $\tau_{ij}$ is the same for all units; then $\frac{1}{2^{2(K-1)}r}\cdot2^KS^2 = \frac{2^KS^2}{2^{2K-2}\,r} = \frac{4S^2}{2^Kr} = \frac{4S^2}{N}$. $\square$

These are Dasgupta et al.'s (2015) Theorems 4.1 and 4.2 and Corollary 4.2.1. They also show (Corollary 4.3.1) that under additivity the estimated effects are uncorrelated.

Why one experiment estimates every effect with the whole sample. A two-arm experiment that devoted all $N$ units to factor 1 alone, $N/2$ at each level, would estimate its effect with variance $S^2/(N/2) + S^2/(N/2) = 4S^2/N$ under the same additivity. The factorial experiment achieves that variance for every one of its $2^K - 1$ effects at once, because each effect's contrast compares all $N/2$ units on one side with all $N/2$ on the other: each unit serves in the estimate of every effect. Varying one factor at a time instead, with the units split among $K$ separate experiments, gives each experiment $N/K$ units and each estimate a variance $K$ times larger. It also answers a different question, the effect of one factor with the others held at chosen levels, and it cannot estimate interactions at all. Dasgupta et al. (2015, §1) summarize the case for factorial designs: they "allow the relative effects of several factors and their interactions to be assessed efficiently".

Three factors on Cranfield

The science table holds the average precision of every Cranfield query under all eight BM25 configurations of three factors, each at two levels: stemming (Porter stemmer on or off), length normalization ($b = 0.75$ or $b = 0$) and the stop list (the 31 stop words removed or kept). The mean average precision of each combination over the 224 queries (1 = high level: stemming on, normalization on, stop words removed):

Stemming Normalization Stop words removed MAP
0 0 0 0.2508
0 0 1 0.2585
0 1 0 0.2782
0 1 1 0.2831
1 0 0 0.2687
1 0 1 0.2715
1 1 0 0.3068
1 1 1 0.3075

The seven factorial effects, and for each the standard deviation of the unit effects $S_j$ and the exact standard deviation of its estimate in a completely randomized experiment with 28 queries per combination (Corollary 3):

Effect $\tau_j$ $S_j$ sd of $\hat\tau_j$
Stemming 0.0210 0.101 0.0295
Normalization 0.0316 0.087 0.0297
Stop list 0.0040 0.018 0.0303
Stemming × normalization 0.0055 0.038 0.0302
Stemming × stop list $-0.0023$ 0.013 0.0303
Normalization × stop list $-0.0012$ 0.013 0.0303
All three 0.0002 0.011 0.0303

Length normalization has the largest main effect, then stemming; removing stop words matters little under BM25, whose idf weights already discount common words. The stemming main effect, 0.0210, is smaller than the effect of stemming with normalization and stop list both at their high levels, 0.0244: the positive stemming × normalization interaction says that stemming helps more when document length is normalized, so averaging over $b = 0$ lowers it. The additive formula $4\bar S^2/N$, with $\bar S^2$ the average of the eight $S_z^2$, gives 0.0303. The actual standard deviations are equal or smaller, most visibly for the two main effects whose unit effects vary most, as Corollary 3's $-S_j^2/N$ term implies; Dasgupta et al. (2015, §4) note that under their compound-symmetry model the variance is largest under additivity.

The script simulates the experiment, 30 seeds of 20,000 random divisions of the 224 queries into eight groups of 28. For each of the seven effects the variance of the estimates divided by the exact variance has a median between 0.997 and 1.001; for stemming, 1.000 (interquartile range 0.994 to 1.004). It also simulates one factor at a time: the queries are split at random into thirds, and the first third, 75 queries, is used for a stemming experiment with the other two factors at their high levels (37 stemmed, 38 not). Its estimate has a variance 3.28 times that of the factorial estimate of the stemming main effect (interquartile range 3.27 to 3.30), close to the factor of three that the count of units predicts. The ratio is also exact: splitting at random into thirds and then 37 and 38 is a complete randomization into three groups of 37, 38 and 149 (the last unused), so Theorem 4 gives the variance $S_{111}^2/37 + S_{011}^2/38 - S_\tau^2/224 = 0.0028625$, where $S_{111}^2$ and $S_{011}^2$ are the variances of the two configurations' AP and $S_\tau^2$ that of their difference over the 224 queries, against 0.00087172 for the factorial main effect, a ratio of 3.284; and it estimates the conditional effect, 0.0244, not the main effect.

The factorial estimates are also the coefficients of a regression. Code each factor as $\pm1$ and add the products of the codes as interaction columns. With equal numbers per combination the columns, together with the column of ones, are orthogonal and each has squared length $N$, so least squares gives each coefficient as $\frac1N\sum_i g_j(z_i)Y_i = \frac{r}{N}\sum_z g_j(z)\bar Y_z = \hat\tau_j/2$: half the effect, since a factor moves from $-1$ to $+1$. Under the normal linear model, tests of the effects are then $F$ tests of The General Linear Hypothesis, and with two factors they are the tests of Two-Way Analysis of Variance. The randomization argument of Corollary 3 needs no such model, at the price of a conservative variance estimate, $2^{-2(K-1)}\sum_z s_z^2/r$ with the sample variances within the groups, which estimates the first term of Corollary 3 without bias by Theorem 3 of the potential-outcomes article.

Latin squares

Blocking can be done in two directions at once. A Latin square design for $T$ treatments arranges $T^2$ units in $T$ rows and $T$ columns, such as plots on a field with a fertility gradient in each direction, and assigns treatments so that each appears exactly once in every row and every column; the randomization chooses among such arrangements. It removes the variation of row means and of column means from the comparison with only $T^2$ units. Its analysis was the subject of the dispute between Neyman and Fisher in 1935. Sabbaghi and Rubin (2014, abstract) report that Neyman, using potential outcomes, asserted that the standard analysis-of-variance $F$ test is valid for randomized complete block designs but invalid for Latin squares, detecting differences among treatments that did not exist on average more often than it should; they show that Neyman's expressions for the expected mean residual sum of squares are generally incorrect for both designs, and that without further assumptions on the potential outcomes the Type I error of the $F$ test cannot be determined from expected sums of squares. The design-based analysis of Latin squares is beyond this article.

What is proved here, what is cited, what the experiment shows

Proved here. Unbiasedness and the exact variance of blocked randomization, with the conservativeness of its variance estimator (Theorem 1). The exact difference between complete and blocked randomization with a common treated share, and its form with equal blocks as a difference of mean squares (Theorem 2). That random blocks are neutral on average (Corollary 1), and that blocks from fixed strata of a superpopulation cannot increase the expected variance (Corollary 2). The paired design's variance and the conservativeness of the paired variance estimator (Theorem 3). The variance of any contrast under complete randomization into several groups (Theorem 4), and the variances of factorial effects with their additive special case (Corollary 3). That factorial effects are orthogonal contrasts and that least squares on $\pm1$ codes returns half-effects.

Cited. Definitions of blocked randomization and the paired design and the large-block comparison (Ding, 2024, ch. 5 and 7); part 2 of Theorem 3 as Ding's Theorem 7.1 and part 1 as his Problem 7.1, with the credit to Imai (2008) as Ding gives it; Pashley and Miratrix (2022), eq. 1 and Theorems 3.1 and 3.2, for the same comparisons in their notation; Box, Hunter and Hunter's maxim as Ding quotes it; Dasgupta et al. (2015) for the potential-outcomes definitions of factorial effects, their Lemmas 3.1 and 3.2, Theorems 4.1 and 4.2 and Corollaries 4.2.1 and 4.3.1; Sabbaghi and Rubin (2014) for the Neyman–Fisher controversy on Latin squares, from their abstract.

What the experiments show. All variances of the table above are exact computations on the Cranfield science table. The simulations confirm Theorems 1, 3 and Corollary 3 to within a percent and measure the coverage of the conservative intervals. The comparisons among blocking variables, the random partitions and the one-factor-at-a-time experiment are observations for these queries and these configurations.

Worked problems

These problems are posed here, on the four-unit table of the potential-outcomes article and on a small factorial design; the article's script checks every number exactly.

Problem 1 (a good and a bad pairing). The four units of the potential-outcomes article have $Y(1) = (7, 5, 4, 2)$ and $Y(0) = (3, 4, 1, 2)$, and complete randomization of two of them has $V_{\mathrm{CR}} = 13/6$. Find the variance of the paired design with pairs $\{1, 2\}, \{3, 4\}$, and with pairs $\{1, 4\}, \{2, 3\}$, both directly and by Theorem 2.

Solution. With $p = 1/2$, $a_i = 2(Y_i(1) + Y_i(0)) = (20, 18, 10, 8)$, with mean 14. Pairs $\{1, 2\}, \{3, 4\}$. Directly: pair 1 gives $d = 7 - 4 = 3$ or $5 - 3 = 2$, variance $1/4$; pair 2 gives $4 - 2 = 2$ or $2 - 1 = 1$, variance $1/4$; so $V_{\mathrm{pair}} = (1/4 + 1/4)/2^2$ = 1/8. By Theorem 2, the block means of $a$ are 19 and 9, so $\mathrm{MSB} = 2\big((19 - 14)^2 + (9 - 14)^2\big)/1$ = 100, and the within-block variances are 2 and 2, so $\mathrm{MSW}$ = 2. Then $V_{\mathrm{CR}} - V_{\mathrm{pair}} = \frac{(1/4)(K - 1)}{n(n - 1)}(\mathrm{MSB} - \mathrm{MSW}) = \frac{1/4}{12}\cdot98$ = 49/24, and $13/6 - 49/24 = 3/24 = 1/8$. Pairs $\{1, 4\}, \{2, 3\}$. Directly: pair $\{1, 4\}$ gives $7 - 2 = 5$ or $2 - 3 = -1$, variance 9; pair $\{2, 3\}$ gives $5 - 1 = 4$ or $4 - 4 = 0$, variance 4; so $V_{\mathrm{pair}} = 13/4$, half again as large as complete randomization. By Theorem 2, the block means of $a$ are both 14, so $\mathrm{MSB}$ = 0, the within-block variances are 72 and 32, $\mathrm{MSW}$ = 52, and the difference is $\frac{1/4}{12}(0 - 52)$ = -13/12, so $V_{\mathrm{pair}} = 13/6 + 13/12 = 13/4$. Pairing the highest-level unit with the lowest makes every pair a copy of the whole, which is the worst a pairing can do here.

Problem 2 (the paired variance estimator). For the good pairing of Problem 1, list $\hat V_{\mathrm P}$ for the four assignments and check Theorem 3.

Solution. With $m = 2$, $\hat V_{\mathrm P} = \frac12\sum_j(d_j - \bar d)^2$. The four assignments give $(d_1, d_2) = (3, 2), (3, 1), (2, 2), (2, 1)$, so $\hat V_{\mathrm P}$ = 1/4, 1, 0, 1/4 respectively; for $(3, 1)$, $\bar d = 2$ and $\frac12(1 + 1) = 1$. Their mean is $E[\hat V_{\mathrm P}]$ = 3/8. Theorem 3 predicts $V_{\mathrm{pair}} + \frac{1}{m(m - 1)}\sum_j(\tau_j - \tau_{\mathrm{fp}})^2$ with pair effects $\tau_1 = (4 + 1)/2 = 5/2$ and $\tau_2 = (3 + 0)/2 = 3/2$ and $\tau_{\mathrm{fp}} = 2$: $\frac18 + \frac12\big(\frac14 + \frac14\big) = \frac18 + \frac14 = \frac38$, as found.

Problem 3 (a $2^2$ factorial, enumerated). Eight units have baseline values $b = (1, 2, 4, 5, 7, 8, 10, 13)$ and potential outcomes $Y_i(z_1, z_2) = b_i + 3z_1 + z_2$ for $z_1, z_2 \in \{-1, +1\}$. Two units are assigned to each of the four combinations, completely at random. Find the main effect of factor 1 and the exact variance of its estimate, and compare with a two-arm experiment on factor 1 using all eight units.

Solution. Every unit's outcome rises by 6 when $z_1$ moves from $-1$ to $+1$, whatever $z_2$, so the main effect of factor 1 is 6 for every unit and $\tau_1 = 6$; the effects are additive. There are $8!/(2!)^4$ = 2,520 equally likely assignments, and the script evaluates $\hat\tau_1 = \frac12\big(\bar Y_{(+,+)} + \bar Y_{(+,-)} - \bar Y_{(-,+)} - \bar Y_{(-,-)}\big)$ in each: the mean is 6 and the variance is 33/4. Corollary 3 predicts $4S^2/N$, where $S^2$ is the common variance of the columns, which is the variance of the baselines: their mean is 6.25, the squared deviations sum to 115.5, and $S^2 = 115.5/7$ = 33/2. So $4S^2/N = 4 \cdot \frac{33}{2}/8 = \frac{33}{4}$, as enumerated. A two-arm experiment on factor 1 with four units at each level has variance $S^2/4 + S^2/4 = 33/4$ (Theorem 2 of the potential-outcomes article, with $S_\tau^2 = 0$): the same, although the factorial experiment estimates the effect of factor 2 and the interaction from the same eight units as well.

Summary

  • Blocked randomization randomizes separately within groups formed before assignment; the weighted difference in means is unbiased with variance $\sum_k(n_{[k]}/n)^2V_{[k]}$ (Theorem 1).
  • With a common treated share, the gain over complete randomization is exactly a between-blocks sum of squares of $a_i = Y_i(1)/p + Y_i(0)/(1 - p)$ minus a small within-blocks price; with equal blocks, blocking helps exactly when $\mathrm{MSB} > \mathrm{MSW}$ (Theorem 2). Random blocks are neutral on average (Corollary 1), and blocks from fixed strata never hurt in expectation (Corollary 2).
  • Pairs are blocks of size two; the paired variance estimator is the paired $t$-test's, conservative by the spread of pair effects (Theorem 3).
  • In a $2^K$ factorial design every effect is an orthogonal contrast of the $2^K$ group means; its variance follows from Theorem 4, and under additivity every effect is estimated with the precision of a two-arm experiment using all units (Corollary 3).
  • On Cranfield, blocking the stemming comparison on TF-IDF difficulty divided the variance by 4.78 and pairing by 5.21, while blocking on the number of query terms raised it by 0.6%; in the three-factor experiment, normalization (0.0316) and stemming (0.0210) had the largest main effects, with a positive interaction between them.

What we have learned

Design decides precision before any outcome is measured. Blocking moves the variation between groups of similar units out of the comparison, and the exact accounting shows both the gain and the small price paid when the groups are not really different. Factorial designs make each unit serve every question at once. Both rest on the same fact as the rest of this unit's experimental articles: the randomization, and nothing else, supplies the probability.

Outside experiments, nobody assigns the treatment, and the groups can differ in ways no block was formed to control. Confounding and Causal Diagrams asks which differences must be adjusted for, and which must not, when the data are observational.

References

  • Box, G. E. P., Hunter, W. G., and Hunter, J. S. (1978). Statistics for Experimenters: An Introduction to Design, Data Analysis, and Model Building. New York: Wiley. (quoted as Ding, 2024, quotes it) record
  • Cleverdon, C. W. (1967). The Cranfield tests on index language devices. Aslib Proceedings, 19(6), 173–194. https://doi.org/10.1108/eb050097 record · 11 pages
  • Dasgupta, T., Pillai, N. S., and Rubin, D. B. (2015). Causal inference from $2^K$ factorial designs by using potential outcomes. Journal of the Royal Statistical Society, Series B, 77(4), 727–753. https://doi.org/10.1111/rssb.12085 (arXiv:1211.2481) (lemma, theorem and corollary numbers are those of the arXiv version) record · 2 pages
  • Ding, P. (2024). A First Course in Causal Inference. Boca Raton: CRC Press. arXiv:2305.18793 (chapter, section and theorem numbers are those of the arXiv version) record · 6 pages
  • Imai, K. (2008). Variance identification and efficiency analysis in randomized experiments under the matched-pair design. Statistics in Medicine, 27(24), 4857–4873. https://doi.org/10.1002/sim.3337 (cited as Ding, 2024, reports it) record
  • Pashley, N. E., and Miratrix, L. W. (2022). Block what you can, except when you shouldn't. Journal of Educational and Behavioral Statistics, 47(1), 69–100. https://doi.org/10.3102/10769986211027240 (arXiv:2010.14078) (equation and theorem numbers are those of the arXiv version) record
  • Sabbaghi, A., and Rubin, D. B. (2014). Comments on the Neyman–Fisher controversy and its consequences. Statistical Science, 29(2), 267–284. https://doi.org/10.1214/13-STS454 (arXiv:1409.2675) record