Scheffé (1959, p. 140) reports a foster-feeding experiment from Bailey's thesis of 1953, as the documentation of the R data set genotype (Venables and Ripley, 2002) records. In it, 61 rat litters of four genotypes were each raised by a foster mother of one of the same four genotypes, and the litter's average weight gain at 28 days was recorded. The 16 combinations hold between 2 and 5 litters each. Fit the litter's genotype first and it accounts for a sum of squares of 60.16. Fit the mother's genotype first and the litter's genotype then accounts for 63.63. A third standard computation credits it with 27.66. Which is the sum of squares for the litter's genotype, and what does each one test?
A design whose cells hold unequal numbers of observations is unbalanced. In a balanced two-way layout, Two-Way Analysis of Variance splits the total sum of squares into parts for each factor, their interaction and the residual. The parts do not depend on the order of fitting. In an unbalanced layout there is no such split. Statistical software offers several sums of squares for each factor, called Type I, Type II and Type III in the terms of the SAS documentation (SAS Institute, 2015). This article proves that each is the $F$ test of a definite linear hypothesis about the cell means, and finds the hypotheses. They differ in how they weight the cells, and in an unbalanced design that is a difference in the question asked.
First we set up the two-way layout as a linear model and define the three types by the reductions in residual sum of squares that produce them. Then we prove a lemma that reads off the hypothesis behind any such sum of squares, and apply it to each type and to the interaction. Next we prove that the types coincide exactly when the cell counts are proportional, which supplies the proof that the course page leaves to Scheffé. We also prove that under an additive model Types II and III test the same hypothesis, with different power. Finally we compare them on the rat data, in an exact calculation and an experiment, and record the arguments for each. The article assumes The General Linear Hypothesis and The Normal Linear Model.
The rat data
The rows are the litter's genotype (A, B, I, J) and the columns the foster mother's. The cell counts:
| Litter | Mother A | Mother B | Mother I | Mother J | All |
|---|---|---|---|---|---|
| A | 5 | 3 | 4 | 5 | 17 |
| B | 4 | 5 | 4 | 2 | 15 |
| I | 3 | 3 | 5 | 3 | 14 |
| J | 4 | 3 | 3 | 5 | 15 |
| All | 16 | 14 | 16 | 15 | 61 |
and the cell means of weight gain, in grams:
| Litter | Mother A | Mother B | Mother I | Mother J |
|---|---|---|---|---|
| A | 63.68 | 52.40 | 54.13 | 48.96 |
| B | 52.33 | 60.64 | 53.93 | 45.90 |
| I | 47.10 | 64.37 | 51.60 | 49.43 |
| J | 54.35 | 56.10 | 54.53 | 49.06 |
This is not the unit's NHANES data. It earns its place for two reasons. It is a genuinely unbalanced design of a size that can be read whole. And Venables (2000) published analyses of it in several orders, against which the script checks every sum of squares below. The NHANES samples of the course page are balanced by construction; the third worked problem draws an unbalanced one.
The two-way layout as a linear model
Let factor A have levels $i = 1, \ldots, a$ and factor B levels $j = 1, \ldots, b$, let cell $(i, j)$ hold $n_{ij} \ge 1$ observations $y_{ijk}$, and write $n_{i\cdot} = \sum_j n_{ij}$, $n_{\cdot j} = \sum_i n_{ij}$ and $n = \sum_{ij} n_{ij}$. Every cell is assumed to hold at least one observation; empty cells change the estimable functions and are not treated here. The cell-means model is
$$y_{ijk} = \mu_{ij} + \varepsilon_{ijk},$$
with independent normal errors of mean 0 and common variance $\sigma^2$. It is a one-way layout with $ab$ groups. By The General Linear Hypothesis the least-squares estimate of $\mu_{ij}$ is the cell mean $\bar{y}_{ij}$, and the residual sum of squares $SS_E = \sum_{ijk}(y_{ijk} - \bar{y}_{ij})^2$ has $n - ab$ degrees of freedom. For the rat data $SS_E = 2{,}440.82$ on $61 - 16 = 45$ degrees of freedom, so the residual mean square is $MS_E = 54.24$.
Four subspaces of the $n$-dimensional space of data vectors describe the models in play. Let $\mathbf{l}_i$ be the indicator of level $i$ of A (1 on the observations at that level, 0 elsewhere), $\mathbf{m}_j$ the indicator of level $j$ of B, and $\mathbf{1}$ the vector of ones. Then
- $V_1 = \operatorname{span}(\mathbf{1})$, the model with one common mean, of dimension 1;
- $V_A = \operatorname{span}(\mathbf{l}_1, \ldots, \mathbf{l}_a)$, the model in which the mean depends on A only, of dimension $a$; $V_B$ likewise, of dimension $b$;
- $V_{A+B} = V_A + V_B$, the additive model $\mu_{ij} = \theta + \alpha_i + \beta_j$ (here $\beta_j$ are column effects and $\mu_{ij}$ cell means, not the regression coefficients and mean vector of the earlier articles);
- $V_{AB}$, spanned by the $ab$ cell indicators, the full cell-means model.
They are nested: $V_1 \subset V_A \subset V_{A+B} \subset V_{AB}$, and $V_1 \subset V_B \subset V_{A+B}$. Write $P_V$ for the projection onto $V$, as in The Geometry of Least Squares, and $\mathrm{RSS}(V) = \lVert\mathbf{y} - P_V\mathbf{y}\rVert^2$ for the residual sum of squares of the model $V$.
The dimension of the additive model. $V_{A+B}$ has dimension $a + b - 1$. A vector in both $V_A$ and $V_B$ takes a value $r_i$ on level $i$ of A and a value $c_j$ on level $j$ of B. So $r_i = c_j$ for every cell that holds an observation. Every cell is nonempty, so $r_i = c_1$ for all $i$ and the vector is constant: $V_A \cap V_B = V_1$. The dimension of a sum of two subspaces is the sum of their dimensions minus that of their intersection, $a + b - 1$.
The reductions. Following the SAS documentation's notation (SAS Institute, 2015, "Introduction to Reduction Notation"), write $R(\,\cdot \mid \cdot\,)$ for the drop in residual sum of squares when terms are added to a model:
$$R(A) = \mathrm{RSS}(V_1) - \mathrm{RSS}(V_A), \qquad R(A \mid B) = \mathrm{RSS}(V_B) - \mathrm{RSS}(V_{A+B}), \qquad R(AB \mid A, B) = \mathrm{RSS}(V_{A+B}) - \mathrm{RSS}(V_{AB}),$$
and $R(B)$, $R(B \mid A)$ likewise. For nested subspaces $V_0 \subset V$, the difference $\mathrm{RSS}(V_0) - \mathrm{RSS}(V)$ equals $\lVert(P_V - P_{V_0})\mathbf{y}\rVert^2$, and $P_V - P_{V_0}$ is the projection onto $V \ominus V_0$, the part of $V$ perpendicular to $V_0$ (The Geometry of Least Squares, Theorem 4).
The three types for factor A. The SAS documentation (SAS Institute, 2015, sections on Type I and Type II sums of squares) defines them by reductions:
- Type I (sequential), with A entered first: $R(A)$; then $R(B \mid A)$ for B and $R(AB \mid A, B)$ for the interaction. Entered in the other order, A receives $R(A \mid B)$. "The Type I SS are model-order dependent; each effect is adjusted only for the preceding effects in the model."
- Type II: $R(A \mid B)$ for A, $R(B \mid A)$ for B, $R(AB \mid A, B)$ for the interaction. Each main effect is adjusted for the other, but not for the interaction that contains it.
- Type III is defined there through estimable functions. The documentation states that the Type III hypotheses "are precisely the hypotheses being tested by programs that reparameterize using the usual assumptions (for example, constraining all parameters for an effect to sum to zero)", and that with no missing cells the Type III sums of squares coincide with "Yates' weighted squares-of-means technique". This article takes the first statement as its working definition: with no empty cell, the Type III sum of squares for A is the increase in the residual sum of squares when the columns of A are deleted from the full model written with sum-to-zero constraints. The section "The parametrization that produces Type III" proves that this is the sum of squares of the general linear hypothesis that the unweighted means $\bar\mu_{i\cdot} = \frac1b\sum_j\mu_{ij}$ are all equal. The equivalence with Yates's technique is the documentation's statement (Yates, 1934, not read here).
Each is tested by $F = (SS/\text{df})/MS_E$ with the full model's residual mean square, on $a - 1$ degrees of freedom for A, $b - 1$ for B and $(a - 1)(b - 1)$ for the interaction.
The rat data. For the litter's genotype (L) and the mother's (M), the script finds:
| Source | Sum of squares | df | $F$ | p |
|---|---|---|---|---|
| L, entered first: $R(L)$ | 60.16 | 3 | 0.370 | 0.7752 |
| M after L: $R(M \mid L)$ | 775.08 | 3 | 4.763 | 0.0057 |
| M, entered first: $R(M)$ | 771.61 | 3 | 4.742 | 0.0059 |
| L after M: $R(L \mid M)$ | 63.63 | 3 | 0.391 | 0.7600 |
| L, Type III | 27.66 | 3 | 0.170 | 0.916 |
| M, Type III | 671.74 | 3 | 4.128 | 0.0114 |
| Interaction: $R(LM \mid L, M)$ | 824.07 | 9 | 1.688 | 0.120 |
| Residual | 2,440.82 | 45 |
The total sum of squares about the grand mean is 4,100.13. The sequential sums in either order add to it with the interaction and residual; the Type II and Type III sums do not. These are the values Venables (2000, §5.2) printed for the two sequential analyses and for the order-independent "drop one term" analysis, to the precision printed there. The mother's genotype matters by every type, the litter's by none, and the interaction is not significant. Here the types agree in their conclusions. The question is what each one asks.
Which hypothesis a sum of squares tests
Each of these sums of squares is the squared length of the projection of the data onto some subspace $V$ of the full model's space $V_{AB}$. One lemma gives its distribution and the hypothesis it tests.
Lemma 1 (a sum of squares inside the model). Let $V \subset V_{AB}$ be a subspace of dimension $r$, and let $\mathbf{m} = E[\mathbf{y}]$, the vector with the entry $\mu_{ij}$ on every observation of cell $(i, j)$. Under the cell-means model, $F = (\lVert P_V\mathbf{y}\rVert^2/r)/MS_E$ has the noncentral $F$ distribution with $r$ and $n - ab$ degrees of freedom and noncentrality $\lVert P_V\mathbf{m}\rVert^2/\sigma^2$. It has the central $F$ distribution exactly when $\mathbf{m}$ is perpendicular to $V$.
Proof. $P_V$ and $I - P_{V_{AB}}$ are projections with $P_V(I - P_{V_{AB}}) = P_V - P_V = 0$, since $V \subset V_{AB}$ gives $P_VP_{V_{AB}} = P_V$. The data vector has independent normal coordinates with variance $\sigma^2$ and mean $\mathbf{m}$, and $(I - P_{V_{AB}})\mathbf{m} = \mathbf{0}$. The projection theorem (Sampling Distributions from the Normal, Theorem 2) makes $\lVert P_V\mathbf{y}\rVert^2/\sigma^2$ noncentral chi-squared with $r$ degrees of freedom and noncentrality $\lVert P_V\mathbf{m}\rVert^2/\sigma^2$. It is independent of $SS_E/\sigma^2$, which is central chi-squared with $n - ab$ degrees of freedom. The noncentrality is zero exactly when $P_V\mathbf{m} = \mathbf{0}$, that is, when $\mathbf{m}$ is perpendicular to $V$. $\square$
So "the hypothesis tested by a sum of squares" has a precise meaning: the set of cell-mean vectors perpendicular to $V$. The test has level exactly $\alpha$ there and power above $\alpha$ everywhere else, because the power of an $F$ test increases with the noncentrality (One-Way Analysis of Variance proves this). An inner product with an indicator is a weighted sum of cell means: $\mathbf{l}_i\cdot\mathbf{m} = \sum_j n_{ij}\mu_{ij}$, $\mathbf{m}_j\cdot\mathbf{m} = \sum_i n_{ij}\mu_{ij}$ and $\mathbf{1}\cdot\mathbf{m} = \sum_{ij}n_{ij}\mu_{ij}$. Two kinds of weighted mean will appear:
$$\tilde\mu_{i\cdot} = \frac{1}{n_{i\cdot}}\sum_j n_{ij}\mu_{ij} \quad\text{(row $i$, weighted by its cell counts)}, \qquad \tilde\mu_{\cdot j} = \frac{1}{n_{\cdot j}}\sum_i n_{ij}\mu_{ij} \quad\text{(column $j$)},$$
and the unweighted row mean $\bar\mu_{i\cdot} = \frac1b\sum_j\mu_{ij}$.
Theorem 1 (the hypotheses of the three types). Under the cell-means model with no empty cell:
- Type I, A first. $R(A)$ tests that the weighted row means $\tilde\mu_{1\cdot}, \ldots, \tilde\mu_{a\cdot}$ are all equal.
- Type II. $R(A \mid B)$ tests that $\sum_j n_{ij}\,(\mu_{ij} - \tilde\mu_{\cdot j}) = 0$ for every $i$: in each row, the cell means, compared with their column's weighted mean and weighted by the row's cell counts, balance to zero.
- Type III. The Type III sum of squares tests that the unweighted row means $\bar\mu_{1\cdot}, \ldots, \bar\mu_{a\cdot}$ are all equal.
Each is referred to the $F$ distribution with $a - 1$ and $n - ab$ degrees of freedom, central under its hypothesis and noncentral otherwise.
The idea: each type projects onto a subspace spanned by modified row indicators. Perpendicularity to a row indicator is a statement about a weighted sum of the cell means in that row.
Proof. Type I. $R(A) = \lVert P_V\mathbf{y}\rVert^2$ with $V = V_A \ominus V_1$. The vectors $\mathbf{l}_i - \frac{n_{i\cdot}}{n}\mathbf{1}$ lie in $V_A$ and are perpendicular to $\mathbf{1}$, since $\mathbf{1}\cdot\mathbf{l}_i = n_{i\cdot}$. Any $\mathbf{v} = \sum_i c_i\mathbf{l}_i$ in $V$ satisfies $\sum_i c_in_{i\cdot} = 0$ and equals $\sum_i c_i(\mathbf{l}_i - \frac{n_{i\cdot}}{n}\mathbf{1})$, so these vectors span $V$. Hence $\mathbf{m} \perp V$ if and only if, for every $i$,
$$\Big(\mathbf{l}_i - \frac{n_{i\cdot}}{n}\mathbf{1}\Big)\cdot\mathbf{m} = n_{i\cdot}\tilde\mu_{i\cdot} - \frac{n_{i\cdot}}{n}\sum_{i'}n_{i'\cdot}\tilde\mu_{i'\cdot} = 0,$$
that is, if and only if every $\tilde\mu_{i\cdot}$ equals the overall weighted mean $\frac1n\sum_{i'}n_{i'\cdot}\tilde\mu_{i'\cdot}$. That holds exactly when the $\tilde\mu_{i\cdot}$ are all equal: if they are, each equals their weighted average; conversely, the condition says that each equals the same number. By Lemma 1 the claim follows.
Type II. $R(A \mid B) = \lVert P_V\mathbf{y}\rVert^2$ with $V = V_{A+B} \ominus V_B$. Every vector of $V_{A+B}$ is $\mathbf{u} + \mathbf{w}$ with $\mathbf{u} \in V_A$ and $\mathbf{w} \in V_B$, and its component perpendicular to $V_B$ is $(I - P_{V_B})\mathbf{u}$. So $V$ is spanned by the vectors $(I - P_{V_B})\mathbf{l}_i$. The projection onto the column indicators replaces each entry by the average of the entries in its column, so $P_{V_B}\mathbf{m}$ replaces each entry of $\mathbf{m}$ by the weighted mean $\tilde\mu_{\cdot j}$ of its column. Since $I - P_{V_B}$ is symmetric,
$$\big((I - P_{V_B})\mathbf{l}_i\big)\cdot\mathbf{m} = \mathbf{l}_i\cdot(I - P_{V_B})\mathbf{m} = \sum_j n_{ij}\,(\mu_{ij} - \tilde\mu_{\cdot j}).$$
So $\mathbf{m} \perp V$ exactly when all these sums vanish, and Lemma 1 gives the claim.
Type III. By the section "The parametrization that produces Type III" below (whose argument does not use this theorem), the Type III sum of squares is the extra sum of squares of the general linear hypothesis $\bar\mu_{1\cdot} = \bar\mu_{i\cdot}$ for $i = 2, \ldots, a$ in the cell-means model. The $a - 1$ restrictions are linearly independent, since the $i$th involves cell $(i, 1)$ and no other restriction does. By Theorem 2 of The General Linear Hypothesis its statistic has the noncentral $F$ distribution with $a - 1$ and $n - ab$ degrees of freedom, central exactly under the hypothesis. $\square$
The hypotheses of Types I and II involve the cell counts; that of Type III does not. That is the whole difference. A hypothesis that depends on $n_{ij}$ is a statement about the population that depends on how many observations the experimenter happened to put in each cell.
For factor B the computations are these. The subspace of $R(B \mid A)$, the Type I and Type II sum for B, is $V_{A+B} \ominus V_A$, spanned by the vectors $(I - P_{V_A})\mathbf{m}_j$. The projection $P_{V_A}\mathbf{m}$ replaces each entry of $\mathbf{m}$ by the weighted mean $\tilde\mu_{i\cdot}$ of its row, so $\big((I - P_{V_A})\mathbf{m}_j\big)\cdot\mathbf{m} = \mathbf{m}_j\cdot(I - P_{V_A})\mathbf{m} = \sum_i n_{ij}(\mu_{ij} - \tilde\mu_{i\cdot})$, and by Lemma 1 $R(B \mid A)$ tests that these sums vanish for every $j$. The subspace of $R(B)$ is $V_B \ominus V_1$, spanned by the vectors $\mathbf{m}_j - \frac{n_{\cdot j}}{n}\mathbf{1}$, whose inner products with $\mathbf{m}$ are $n_{\cdot j}\tilde\mu_{\cdot j} - \frac{n_{\cdot j}}{n}\sum_{j'}n_{\cdot j'}\tilde\mu_{\cdot j'}$; they all vanish exactly when the weighted column means $\tilde\mu_{\cdot j}$ are equal, by the argument of part 1 with columns in place of rows.
The rat data. For the litter's genotype the three kinds of row mean, computed from the cell means, are:
| Litter | Weighted mean $\tilde\mu_{i\cdot}$ (Type I) | Unweighted mean $\bar\mu_{i\cdot}$ (Type III) |
|---|---|---|
| A | 55.11 | 54.79 |
| B | 54.67 | 53.20 |
| I | 52.91 | 53.13 |
| J | 52.97 | 53.51 |
with row totals 17, 15, 14 and 15 litters. The two kinds of mean differ by up to 1.47 g, and the tests weigh different differences: $R(L) = 60.16$ against 27.66 for Type III. For the mother's genotype the weighted column means are 55.40, 58.70, 53.36 and 48.68 g. The unweighted are 54.36, 58.38, 53.55 and 48.34 g, with column totals 16, 14, 16 and 15.
These are general linear hypotheses in the full sense. Lemma 1 shows that Types I and II have $F$ distributions with the stated null hypotheses. More is true: each is exactly the extra sum of squares of The General Linear Hypothesis, Theorem 1, for its hypothesis written as $C\boldsymbol\mu = \mathbf{0}$ in the cell-means model. In the proof of Theorem 2 there, the extra sum of squares is $\lVert P\mathbf{y}\rVert^2$ with $P$ the projection onto the columns of $X(X^{\mathsf{T}}X)^{-1}C^{\mathsf{T}}$. In the cell-means model $X^{\mathsf{T}}X$ is diagonal with entries $n_{ij}$, so a restriction with coefficient $c_{ij}$ on $\mu_{ij}$ becomes the vector with entry $c_{ij}/n_{ij}$ on every observation of cell $(i, j)$. For Type I the restriction $\tilde\mu_{i\cdot} - \tilde\mu_{a\cdot} = 0$ has $c_{ij} = n_{ij}/n_{i\cdot}$ in row $i$ and $-n_{aj}/n_{a\cdot}$ in row $a$, which becomes $\mathbf{l}_i/n_{i\cdot} - \mathbf{l}_a/n_{a\cdot}$. These $a - 1$ vectors lie in $V_A \ominus V_1$, which has dimension $a - 1$, and are independent, so they span it. For Type II, the restriction for row $i$ has $c_{i'j} = n_{ij}(\mathbf{1}\{i' = i\} - n_{i'j}/n_{\cdot j})$, which becomes the vector with entry $\mathbf{1}\{i' = i\} - n_{ij}/n_{\cdot j}$ on cell $(i', j)$, that is, $(I - P_{V_B})\mathbf{l}_i$. The $a$ conditions sum to zero, so one is redundant; the first $a - 1$ are independent and their vectors span $V_{A+B} \ominus V_B$, of dimension $(a + b - 1) - b = a - 1$. The script confirms on the rat data that both general linear hypotheses reproduce $R(L) = 60.16$ and $R(L \mid M) = 63.63$.
The interaction is the same in every type
All three types assign the interaction the same sum of squares, $R(AB \mid A, B)$. It tests the hypothesis that the effect of one factor does not depend on the level of the other.
Theorem 2 (no interaction). With no empty cell, the following are equivalent for a vector of cell means $\mu_{ij}$:
- $\mu_{ij} = \theta + \alpha_i + \beta_j$ for some numbers $\theta$, $\alpha_i$, $\beta_j$ (the additive model);
- $\mu_{ij} - \mu_{i1} - \mu_{1j} + \mu_{11} = 0$ for all $i \ge 2$ and $j \ge 2$;
- $\mu_{ij} - \mu_{ij'} - \mu_{i'j} + \mu_{i'j'} = 0$ for all $i, i', j, j'$: the difference between any two levels of B is the same at every level of A.
The $(a - 1)(b - 1)$ restrictions in (2) are linearly independent, and their general linear hypothesis has extra sum of squares $R(AB \mid A, B)$, tested on $(a - 1)(b - 1)$ and $n - ab$ degrees of freedom.
Proof. (1) implies (3): substituting, the $\theta$, $\alpha$ and $\beta$ terms cancel in pairs. (3) implies (2) as a special case. (2) implies (1): put $\theta = \mu_{11}$, $\alpha_i = \mu_{i1} - \mu_{11}$ and $\beta_j = \mu_{1j} - \mu_{11}$; then $\theta + \alpha_i + \beta_j = \mu_{i1} + \mu_{1j} - \mu_{11}$, which equals $\mu_{ij}$ by (2), and the identity also holds when $i = 1$ or $j = 1$. The restriction for $(i, j)$ in (2) is the only one involving $\mu_{ij}$ with $i, j \ge 2$, so they are independent. Under them the cell means range over the additive model, so the restricted least-squares fit is the projection onto $V_{A+B}$ and $\mathrm{RSS}_0 = \mathrm{RSS}(V_{A+B})$. The unrestricted fit is the projection onto $V_{AB}$. By Theorem 1 of The General Linear Hypothesis the extra sum of squares is $\mathrm{RSS}(V_{A+B}) - \mathrm{RSS}(V_{AB}) = R(AB \mid A, B)$, and the degrees of freedom are $ab - (a + b - 1) = (a - 1)(b - 1)$. $\square$
For the rat data the interaction sum of squares is 824.07 on 9 and 45 degrees of freedom, with $F = 1.688$ and $p = 0.120$. The data do not show that the effect of the mother's genotype depends on the litter's.
When the types agree
In the balanced layout of Two-Way Analysis of Variance the order of fitting does not matter. The exact condition is weaker than balance.
Theorem 3 (proportional cell counts). With no empty cell, $R(A) = R(A \mid B)$ for every data vector $\mathbf{y}$ if and only if the cell counts are proportional:
$$n_{ij} = \frac{n_{i\cdot}\,n_{\cdot j}}{n} \quad\text{for all } i, j.$$
The same condition gives $R(B) = R(B \mid A)$.
The idea: the order does not matter exactly when the part of $V_A$ perpendicular to $\mathbf{1}$ is perpendicular to all of $V_B$. The inner products that decide this are $n_{ij} - n_{i\cdot}n_{\cdot j}/n$.
Proof. Write $\mathbf{u}_i = \mathbf{l}_i - \frac{n_{i\cdot}}{n}\mathbf{1}$, which span $V_A \ominus V_1$ (proof of Theorem 1). Then
$$\mathbf{u}_i\cdot\mathbf{m}_j = n_{ij} - \frac{n_{i\cdot}\,n_{\cdot j}}{n}.$$
If the counts are proportional, every $\mathbf{u}_i$ is perpendicular to every $\mathbf{m}_j$, hence to $V_B$, so $P_{V_B}\mathbf{u}_i = \mathbf{0}$. Since $(I - P_{V_B})\mathbf{1} = \mathbf{0}$, $(I - P_{V_B})\mathbf{l}_i = (I - P_{V_B})\mathbf{u}_i = \mathbf{u}_i$. So the vectors spanning $V_{A+B} \ominus V_B$ are the vectors spanning $V_A \ominus V_1$; the two subspaces, hence their projections, are equal, and $R(A \mid B) = R(A)$ for every $\mathbf{y}$.
Conversely, suppose $R(A) = R(A \mid B)$ for every $\mathbf{y}$. Then $\mathbf{y}^{\mathsf{T}}(P_1 - P_2)\mathbf{y} = 0$ for all $\mathbf{y}$, where $P_1$ and $P_2$ are the two projections. A symmetric matrix $D$ with $\mathbf{y}^{\mathsf{T}}D\mathbf{y} = 0$ for every $\mathbf{y}$ is zero, because $2\mathbf{x}^{\mathsf{T}}D\mathbf{z} = (\mathbf{x} + \mathbf{z})^{\mathsf{T}}D(\mathbf{x} + \mathbf{z}) - \mathbf{x}^{\mathsf{T}}D\mathbf{x} - \mathbf{z}^{\mathsf{T}}D\mathbf{z} = 0$ for all $\mathbf{x}$, $\mathbf{z}$. So $P_1 = P_2$ and $V_A \ominus V_1 = V_{A+B} \ominus V_B$, which is perpendicular to $V_B$. Each $\mathbf{u}_i$ is therefore perpendicular to each $\mathbf{m}_j$, and the displayed inner product vanishes. For B, exchange the names of the factors: $R(B)$ and $R(B \mid A)$ become $R(A)$ and $R(A \mid B)$ of the transposed layout, whose counts $n'_{ji} = n_{ij}$ are proportional exactly when the original counts are, so the statement for B is the one just proved, applied to the transposed layout. $\square$
For the rat data the largest $\lvert n_{ij} - n_{i\cdot}n_{\cdot j}/n\rvert$ is 1.69, and the two orders differ, as the table showed. A design such as $n_{ij} = (2, 1)^{\mathsf{T}}(1, 2, 3)$ has rows of 2, 4, 6 and 1, 2, 3 observations, 18 observations in all. It is unbalanced but proportional, and the script confirms that its two projections coincide.
Corollary (the balanced two-way layout). With $r \ge 2$ observations in every cell, the three $F$ tests of Two-Way Analysis of Variance have $F$ distributions with $a - 1$, $b - 1$ and $(a - 1)(b - 1)$ numerator degrees of freedom and $ab(r - 1)$ denominator degrees of freedom. The test for A has the central distribution exactly when the row means $\bar\mu_{i\cdot}$ are equal, whether or not there is an interaction. Likewise for B, and the interaction test is central exactly when the means are additive.
Proof. Equal counts $n_{ij} = r$ are proportional ($n_{i\cdot}n_{\cdot j}/n = br \cdot ar/(abr) = r$), so the order of fitting does not matter and $SS_A = R(A)$. The projection onto $V_A$ replaces each observation by its row mean $\bar{x}_{i\cdot}$ and the projection onto $V_1$ by the grand mean, so $R(A) = \lVert(P_{V_A} - P_{V_1})\mathbf{y}\rVert^2 = br\sum_i(\bar{x}_{i\cdot} - \bar{x})^2$, the course page's $SS_A$ (the course page writes $x$ for the data); likewise $SS_B = R(B)$. By Lemma 1 the test of A is central exactly when the weighted row means are equal. With equal counts the weights are $n_{ij}/n_{i\cdot} = 1/b$, so the weighted means are the unweighted means $\bar\mu_{i\cdot}$. The interaction is Theorem 2. The course page's $SS_{AB}$ is $R(AB \mid A, B)$: its partition shows $SS_A + SS_B + SS_{AB}$ equals the between-cells sum of squares $\mathrm{RSS}(V_1) - \mathrm{RSS}(V_{AB})$, which is $R(A) + R(B \mid A) + R(AB \mid A, B)$, and $R(B \mid A) = R(B) = SS_B$. $\square$
In a balanced design the three types give the same sum of squares for every data vector, not merely the same hypothesis. The step from one to the other is a lemma.
Lemma 2 (the hypothesis determines the sum of squares). Let $V$ and $W$ be subspaces of $V_{AB}$ such that the cell-mean vectors $\mathbf{m} \in V_{AB}$ perpendicular to $V$ are exactly those perpendicular to $W$. Then $V = W$, so $\lVert P_V\mathbf{y}\rVert^2 = \lVert P_W\mathbf{y}\rVert^2$ for every $\mathbf{y}$.
Proof. The set of $\mathbf{m} \in V_{AB}$ perpendicular to $V$ is $V_{AB} \ominus V$, the part of $V_{AB}$ perpendicular to $V$. A subspace $V$ of $V_{AB}$ is the part of $V_{AB}$ perpendicular to $V_{AB} \ominus V$: every vector of $V_{AB}$ is the sum of its projections onto $V$ and onto $V_{AB} \ominus V$ (The Geometry of Least Squares, Theorem 1), and it is perpendicular to $V_{AB} \ominus V$ exactly when the second projection is zero. So $V_{AB} \ominus V = V_{AB} \ominus W$ gives $V = W$. $\square$
Each Type III sum of squares is of the form $\lVert P_V\mathbf{y}\rVert^2$ with $V \subset V_{AB}$: by the proof of Theorem 2 of The General Linear Hypothesis (with $\mathbf{d} = \mathbf{0}$), $V$ is the span of the columns of $X(X^{\mathsf{T}}X)^{-1}C^{\mathsf{T}}$, which lie in $V_{AB}$. Under balance the Type I, Type II and Type III hypotheses for A are the same set of cell-mean vectors (equal unweighted row means, by the corollary and by Theorem 3), so by Lemma 2 the three sums of squares for A are equal for every $\mathbf{y}$; likewise for B. The script confirms it on the course's balanced sample (second worked problem).
Additive models: Types II and III ask the same question
If there is no interaction, "the effect of A" has an unambiguous meaning: the $\alpha_i$ of the additive model, the same difference at every level of B. Type III tests it, since $\bar\mu_{i\cdot} = \theta + \alpha_i + \bar\beta$ is equal for all $i$ exactly when the $\alpha_i$ are. So does Type II, although its hypothesis looked different.
Theorem 4 (Type II under additivity). With no empty cell, if $\mu_{ij} = \theta + \alpha_i + \beta_j$, then the Type II hypothesis for A holds if and only if $\alpha_1 = \cdots = \alpha_a$.
Proof. In column $j$ the weighted mean is $\tilde\mu_{\cdot j} = \theta + \beta_j + \tilde\alpha_j$ with $\tilde\alpha_j = \sum_{i'}n_{i'j}\alpha_{i'}/n_{\cdot j}$, because the weights $n_{i'j}/n_{\cdot j}$ sum to 1. So $\mu_{ij} - \tilde\mu_{\cdot j} = \alpha_i - \tilde\alpha_j$, and the hypothesis $\sum_j n_{ij}(\alpha_i - \tilde\alpha_j) = 0$ reads
$$\alpha_i = \sum_{i'} w_{ii'}\,\alpha_{i'}, \qquad w_{ii'} = \frac{1}{n_{i\cdot}}\sum_j \frac{n_{ij}\,n_{i'j}}{n_{\cdot j}}.$$
The weights $w_{ii'}$ are positive, because every cell is nonempty, and they sum to 1 over $i'$, because $\sum_{i'}n_{i'j} = n_{\cdot j}$ and $\sum_j n_{ij} = n_{i\cdot}$. If all $\alpha_i$ are equal the equations hold. Conversely, take $i^*$ with $\alpha_{i^*}$ largest. Then $\alpha_{i^*} = \sum_{i'}w_{i^*i'}\alpha_{i'}$ is an average, with positive weights, of numbers none larger than $\alpha_{i^*}$, so every $\alpha_{i'}$ equals $\alpha_{i^*}$. $\square$
Type I with A first does not have this property. Its hypothesis, equal weighted row means $\tilde\mu_{i\cdot} = \theta + \alpha_i + \sum_j n_{ij}\beta_j/n_{i\cdot}$, involves the $\beta_j$ through the counts. So a factor B can make the test of A reject when A has no effect at all. Take the rat design with no litter effect and mother effects of 0, 3, $-1$ and $-4$ g, and set $\sigma^2$ to the residual mean square 54.24. The noncentrality of $R(L)$ is then 0.231, and the "5%" test of the litter's genotype entered first rejects with probability 0.063. $R(L \mid M)$ has noncentrality exactly 0 there.
Example (an interaction that Type III cannot see as a main effect). The converse failure belongs to Types I and II. Take cell means that are zero except for a pure interaction in the corner of litters A and B and mothers A and B. Put $+10$ g in cells (A, A) and (B, B), $-10$ g in (A, B) and (B, A). Every unweighted row mean is 0, so the Type III hypothesis holds and its test has level exactly 0.05. But the weighted row mean of litter A is $10 \times (5 - 3)/17 = 1.18$ g and that of litter B is $10 \times (5 - 4)/15 = 0.67$ g. So the Type I hypothesis fails, with noncentrality 0.28, and the Type II noncentrality is also 0.28. The Type I and Type II tests "of the litter's genotype" have power 0.066 here, although in this configuration the litter's genotype, averaged over mothers with equal weight, has no effect. Which answer is right depends on which question was meant, and Theorem 1 states the question each test answers.
Power. When the model is additive and both tests are valid, which is more powerful? Take the rat design with litter effects $-3$, 0, 0 and 3 g, mother effects 0, 3, $-1$ and $-4$ g, and normal errors with the residual standard deviation $\sigma = 7.36$ g. The noncentralities are 5.29 for Type II and 5.07 for Type III. The 5% tests (critical value 2.8115 on 3 and 45 degrees of freedom) have power 0.429 and 0.412. An experiment confirms the calculation. For each of 30 seeds (numpy.random.default_rng(3000 + seed)), 2,000 data sets were generated from this model on the rat design. The Type II test rejected in a share 0.429 of them (median over seeds; interquartile range 0.422 to 0.438), the Type III test in 0.412 (0.408 to 0.418). The advantage of Type II here is small. A plausible source, not established here, is the weighting. The unweighted means give the sparse cells, such as the 2 litters of genotype B raised by mothers J, as much weight as the full ones, while Type II weights cells by their counts.
The parametrization that produces Type III
Type III sums of squares are usually computed by fitting the full model in an effects parametrization and deleting the columns of one main effect. Whether that is meaningful depends on the coding.
Write $\mu_{ij} = \theta + \alpha_i + \beta_j + \gamma_{ij}$ with the sum-to-zero constraints $\sum_i\alpha_i = \sum_j\beta_j = 0$ and $\sum_i\gamma_{ij} = \sum_j\gamma_{ij} = 0$ for every $j$ and $i$. Every set of cell means has exactly one such representation: averaging over $j$ and then over $i$ forces
$$\theta = \bar\mu_{\cdot\cdot}, \qquad \alpha_i = \bar\mu_{i\cdot} - \bar\mu_{\cdot\cdot}, \qquad \beta_j = \bar\mu_{\cdot j} - \bar\mu_{\cdot\cdot}, \qquad \gamma_{ij} = \mu_{ij} - \bar\mu_{i\cdot} - \bar\mu_{\cdot j} + \bar\mu_{\cdot\cdot},$$
with unweighted averages, and these values satisfy the constraints and add back to $\mu_{ij}$. So $\alpha = \mathbf{0}$ is the hypothesis that the unweighted row means are equal. Deleting the $a - 1$ columns that carry $\alpha$ (with the constraints built into the coding) fits the model restricted by $\alpha = \mathbf{0}$. By Theorem 1 of The General Linear Hypothesis the increase in residual sum of squares is the extra sum of squares of the hypothesis $\alpha = \mathbf{0}$, that is, of equal unweighted row means. Since that increase is the Type III sum of squares by the working definition, this proves that Type III tests equality of the unweighted row means. This is the computation Venables (2000, §5.2) reports with "a contrast matrix with zero-sum columns", and the script confirms that it gives 27.66 for the litter's genotype.
The treatment coding makes the first level of each factor the baseline ($\alpha_1 = \beta_1 = 0$, $\gamma_{1j} = \gamma_{i1} = 0$). Its main-effect parameters are $\alpha_i = \mu_{i1} - \mu_{11}$: differences between litters among the litters raised by mothers of genotype A only. Deleting those columns tests that hypothesis, a sum of squares of 591.69 for the rat data, which the script confirms is the general linear hypothesis $\mu_{11} = \mu_{21} = \mu_{31} = \mu_{41}$. Venables (2000) calls the result of this coding "nonsense". The reason is now plain: the hypothesis concerns one column of the table, chosen by the alphabetical order of the labels.
The arguments for each type
The theorems settle what each type tests. Which to report is a judgment about which hypothesis is of interest, and published opinions differ.
- Type I is the right analysis when the order of the terms is itself the question, as in a polynomial regression or a nested design. It is the only one whose sums of squares add up to the total. The SAS documentation lists balanced models specified in proper sequence, purely nested models and polynomial regression models as those for which Type I hypotheses "are considered appropriate". For two crossed factors in an unbalanced design the first factor's test is contaminated by the second (Theorem 4 and the example after it).
- Type II tests each main effect adjusted for the other main effects. The SAS documentation notes that "the Type II hypotheses for A and B are dependent on the cell frequencies in the design". For effects contained in other effects it calls them "of questionable value" when the cell frequencies are not proportional to those of a background population. Under an additive model it tests exactly the main effect (Theorem 4). Langsrud (2003), in his abstract, argues that "by considering the power of the two methods, it is clear that Type II is preferable". The power comparison above agrees in direction for this design, by a small margin.
- Type III tests a hypothesis that does not depend on the cell counts, equality of the unweighted marginal means. The SAS documentation gives its rationale as "the hypotheses tested for any given effect should be the same for all designs with the same general form of estimable functions". Its critics answer that when there is an interaction the unweighted marginal mean is an average over levels of the other factor with weights nobody chose. Venables (2000, §5.1) writes that "testing main effects in the presence of an interaction is a violation of the marginality principle". He adds that when the factors interact, "it is difficult to see why the main effects for either factor can be of any interest". Langsrud's abstract, citing Nelder and Lane (1995), describes the Type III analysis as "founded on unrealistic models", namely "models with interactions, but without all corresponding main effects".
What the theorems add is that none of the three is a test "of the main effect" in general. Each tests a stated hypothesis about weighted or unweighted means. They agree when the counts are proportional (Theorem 3), and Types II and III agree on the hypothesis, though not on power, when there is no interaction (Theorem 4). When there is an interaction, the question "does A matter?" has no single meaning. The interaction itself, which every type tests the same way, is usually the finding to report.
What is proved, what is cited, what the experiment shows
Proved here. The dimension of the additive model; the lemma identifying the hypothesis behind a sum of squares inside the model; the hypotheses of Types I, II and III (Theorem 1), and the identification of Types I and II as general linear hypotheses; the equivalence of additivity with the vanishing of the interaction contrasts, and the interaction sum of squares as their extra sum of squares (Theorem 2); the proportional-counts condition for order-independence (Theorem 3) and the proof of the $F$ tests of the balanced two-way layout (corollary); the equality of the three types under balance (Lemma 2); the equivalence of Types II and III under additivity and the contamination of Type I (Theorem 4); the hypotheses tested by deleting main-effect columns under the sum-to-zero and treatment codings.
Taken from other articles. The projection theorem (Sampling Distributions from the Normal); nested projections (The Geometry of Least Squares); restricted least squares and the $F$ test of a general linear hypothesis (The General Linear Hypothesis).
Cited. The definitions of the types and the reduction notation, and the statements quoted from the SAS/STAT documentation (SAS Institute, 2015); the analyses of the rat data and the quotations from Venables (2000); the abstract of Langsrud (2003). Yates (1934), Nelder and Lane (1995) and Scheffé (1959) are cited through those sources, not read. That Type III coincides with Yates's method of weighted squares of means is the SAS documentation's and Langsrud's statement, and the data's origin in Scheffé (1959) and Bailey (1953) is the R documentation's.
The experiment shows rejection rates of the Type II and Type III tests for one additive configuration on the rat design under normal errors, matching the exact noncentral $F$ calculation.
Worked problems
These problems were written for this article, on a layout from the SAS documentation and on the course's NHANES samples. The script tools/experiments/statistics/unbalanced-designs-and-types-of-sums-of-squares.py recomputes every number.
Problem 1 (SAS's unbalanced 2 x 2 layout). The SAS documentation's example of Type II estimable functions is a 2 x 2 layout with two observations in every cell except cell (2, 2), which has one. It reports that the Type II hypothesis for A puts weights 0.6 and 0.4 on the two cells of each row. Derive these weights from Theorem 1, and give the Type III weights.
Solution. The Type II hypothesis for A is $0.6(\mu_{11} - \mu_{21}) + 0.4(\mu_{12} - \mu_{22}) = 0$; the Type III hypothesis is $0.5(\mu_{11} - \mu_{21}) + 0.5(\mu_{12} - \mu_{22}) = 0$.
The counts are $n_{11} = n_{12} = n_{21} = 2$ and $n_{22} = 1$, so the column totals are $n_{\cdot 1} = 4$ and $n_{\cdot 2} = 3$. The weighted column means are $\tilde\mu_{\cdot 1} = (2\mu_{11} + 2\mu_{21})/4$ and $\tilde\mu_{\cdot 2} = (2\mu_{12} + \mu_{22})/3$. Theorem 1's condition for row 1 is
$$2\Big(\mu_{11} - \frac{\mu_{11} + \mu_{21}}{2}\Big) + 2\Big(\mu_{12} - \frac{2\mu_{12} + \mu_{22}}{3}\Big) = (\mu_{11} - \mu_{21}) + \tfrac23(\mu_{12} - \mu_{22}) = 0.$$
Dividing by $1 + \frac23 = \frac53$ gives weights $\frac35 = 0.6$ and $\frac25 = 0.4$, as in the documentation's table of Type II estimable functions for this layout. The condition for row 2 is minus the one for row 1 (the conditions for all rows sum to zero), so the hypothesis has one degree of freedom, as A has two levels. The Type III hypothesis is equality of the unweighted row means, $\frac12(\mu_{11} + \mu_{12}) = \frac12(\mu_{21} + \mu_{22})$, with weights 0.5 and 0.5 whatever the counts. The documentation gives exactly these as the Type II weights of the balanced layout, and states that the Type III hypotheses for every 2 x 2 layout with no empty cell are the balanced ones. The missing observation shifts Type II's weight toward the column with more observations.
Problem 2 (the course's balanced sample). The course page's two-way example has 5 men and 5 women from each of four age groups. Show that the order of fitting does not matter, and state what its test for age group tests.
Solution. The sums of squares are the same in every order and type: 4,066.76 for age group, 205.51 for sex, 223.02 for the interaction and 5,541.87 within cells, as on the course page.
The counts are all 5, hence proportional, so by Theorem 3, $R(\text{age}) = R(\text{age} \mid \text{sex})$ and $R(\text{sex}) = R(\text{sex} \mid \text{age})$ for these and any other responses; the script confirms both equalities. By the corollary, the test for age group refers $(4{,}066.76/3)/(5{,}541.87/32)$ to the $F$ distribution with 3 and 32 degrees of freedom. It is central exactly when the four age groups have equal means averaged with equal weight over the two sexes. That holds whether or not the gap between the sexes varies with age, a question the interaction test answers separately.
Problem 3 (an unbalanced NHANES sample). Draw 40 NHANES adults at random from the 5,247 adults of the course pages (no body mass index is needed here, so the unit's 5,176 are not used; numpy.random.default_rng(7)), classify them by age group and sex, and test sex by Types I (both orders), II and III.
Solution. The cells hold, for men and women:
| Age group | Men | Women |
|---|---|---|
| 18–29 | 3 | 7 |
| 30–44 | 2 | 4 |
| 45–59 | 6 | 4 |
| 60 and over | 4 | 10 |
The full model has 8 cells, so the residual has $40 - 8 = 32$ degrees of freedom, with mean square 217.34. For sex, each on 1 and 32 degrees of freedom, the sums of squares are:
- $R(\text{sex}) = 55.21$ (Type I, sex first), with $F = 0.254$ and $p = 0.618$;
- $R(\text{sex} \mid \text{age}) = 73.38$ (Type I with age first, and Type II), with $F = 0.338$ and $p = 0.565$;
- 113.26 for Type III, with $F = 0.521$ and $p = 0.476$.
For age group, $R(\text{age}) = 12{,}373.14$ and $R(\text{age} \mid \text{sex}) = 12{,}391.31$, with $F = 18.977$ and 19.004 on 3 and 32 degrees of freedom and p-values below 0.001. The interaction has sum of squares 462.78, $F = 0.710$, $p = 0.553$.
The three sums for sex differ because they test different weighted comparisons of men with women. Type I with sex first compares all men with all women. The men and women of this sample are distributed differently over the age groups (the oldest group is 4 men and 10 women), so it mixes the age difference into the sex comparison. Type II compares them within age groups, pooled with the cell counts as weights. Type III compares the unweighted average over the four age groups. None of the three detects a difference. If the model is additive, as the interaction test gives no reason to doubt, Theorem 4 makes the Type II and Type III tests both tests of sex alone; they differ only in power.
Summary
| Sum of squares for A | Reduction | Hypothesis tested (no empty cell) |
|---|---|---|
| Type I, A first | $R(A)$ | weighted row means $\tilde\mu_{i\cdot} = \sum_j n_{ij}\mu_{ij}/n_{i\cdot}$ equal |
| Type I, A second, and Type II | $R(A \mid B)$ | $\sum_j n_{ij}(\mu_{ij} - \tilde\mu_{\cdot j}) = 0$ for every $i$ |
| Type III | sum-to-zero coding, drop A | unweighted row means $\bar\mu_{i\cdot}$ equal |
| Interaction, every type | $R(AB \mid A, B)$ | $\mu_{ij} - \mu_{ij'} - \mu_{i'j} + \mu_{i'j'} = 0$ |
| Condition | Consequence |
|---|---|
| $n_{ij} = n_{i\cdot}n_{\cdot j}/n$ | $R(A) = R(A \mid B)$, $R(B) = R(B \mid A)$ |
| $n_{ij} = r$ | all three types coincide; the balanced $F$ tests are exact |
| additive model | Types II and III both test $\alpha_1 = \cdots = \alpha_a$; Type I does not |
What we have learned
In an unbalanced two-way layout, a "sum of squares for factor A" is the squared length of the data's projection onto some subspace inside the full model. The hypothesis it tests is that the cell means are perpendicular to that subspace. Written out, the sequential sum for the first factor tests equality of row means weighted by the cell counts. The Type II sum tests a balance of each row against its columns' weighted means, and the Type III sum tests equality of unweighted row means. All three test the interaction identically. They coincide exactly when the counts are proportional, which is why the balanced layout of the course page has one analysis, and its $F$ tests are now proved. Under additivity Types II and III ask the same question, and on the rat design Type II answered it with slightly more power. For Scheffé's 61 litters, all three agree that the mother's genotype matters ($p$ from 0.0057 to 0.0114) and the litter's does not.
The tests in this unit rest on the assumption that the errors are independent with a common variance, and a single observation can move a fit far. Leverage, Influence and Residual Diagnostics shows how to find such observations and what deleting one does to the coefficients. Venables (2000) notes at least one large outlier in the rat data that much of the analysis hangs on.
References
- Bailey, D. W. (1953). The Inheritance of Maternal Influences on the Growth of the Rat. Unpublished Ph.D. thesis, University of California. Table B of the Appendix (as recorded in the documentation of the R data set
genotype). record - Langsrud, Ø. (2003). ANOVA for unbalanced data: Use Type II instead of Type III sums of squares. Statistics and Computing, 13(2), 163–167. https://doi.org/10.1023/A:1023260610025 record
- Nelder, J. A., and Lane, P. W. (1995). The computer analysis of factorial experiments: in memoriam—Frank Yates. The American Statistician, 49(4), 382–385. https://doi.org/10.1080/00031305.1995.10476189 record
- SAS Institute Inc. (2015). SAS/STAT 14.1 User's Guide, Chapter 15, "The Four Types of Estimable Functions". SAS Institute Inc., Cary, NC. https://support.sas.com/documentation/onlinedoc/stat/141/introglmest.pdf record
- Scheffé, H. (1959). The Analysis of Variance. Wiley. record · 2 pages
- Venables, W. N. (2000). Exegeses on Linear Models. Paper presented to the S-PLUS User's Conference, Washington, DC, 8–9 October 1998 (version of 13 May 2000). https://www.stats.ox.ac.uk/pub/MASS3/Exegeses.pdf record
- Venables, W. N., and Ripley, B. D. (2002). Modern Applied Statistics with S (4th ed.). Springer. Data set
genotypeof the R package MASS, as distributed by the Rdatasets archive: https://vincentarelbundock.github.io/Rdatasets/ record - Yates, F. (1934). The analysis of multiple classifications with unequal numbers in the different classes. Journal of the American Statistical Association, 29(185), 51–66. https://doi.org/10.1080/01621459.1934.10502686 record