Two-Way Analysis of Variance

The analysis of variance as a regression on indicator variables, and its extension to two factors and their interaction. The regression identities for the one-way analysis, the partition of the total sum of squares of a two-way layout with equal cell sizes proved, the three F tests stated, and an example on NHANES blood pressures by age group and sex.

One-Way Analysis of Variance compared the systolic blood pressures of four age groups in the NHANES survey of 2017 to 2018. Each of those adults is also a man or a woman, so the same blood pressures can be classified two ways at once. Does the gap between men and women depend on the age group? In a sample of 5 men and 5 women from each age group, the test built in this article finds no evidence that it does: the interaction of age group and sex has $F = 0.429$ on 3 and 32 degrees of freedom, $p = 0.733$.

A two-way analysis of variance asks three questions of such data: whether the first factor matters, whether the second does, and whether the effect of one depends on the level of the other. It answers each with its own $F$ ratio, as the one-way analysis answers its single question.

First we show that the one-way analysis of variance is a regression on indicator variables, which is the view that extends to several factors. Then we prove the partition of the sum of squares for two factors with equal cell sizes, state the three $F$ tests, and work the example on age group and sex. The article assumes One-Way Analysis of Variance: its notation, its worked sample of 10 adults from each of four age groups, the sums of squares $SSB$, $SSW$ and $SST$, the share $\eta^2 = SSB/SST$, and the theorem on the $F$ distribution.

ANOVA as regression

The method grows beyond one factor through another view of the same analysis: the one-way analysis of variance is a special case of linear regression.

Code the age group with three indicator variables, one each for 30–44, 45–59 and 60 and over, with 18–29 as the baseline. Then fit a multiple linear regression of blood pressure on them, as described there for categorical inputs. For the worked sample of One-Way Analysis of Variance the fitted intercept is 108.267, the mean of the baseline group. The three coefficients are 3.300, 6.967 and 22.067, the differences between each group's mean and the baseline's. Every quantity of the analysis of variance reappears:

  • the fitted value for every person is their group's mean, so the residual sum of squares is $SSW$ and the residual mean square is $s_W^2 = 166.20$. The reason: with an intercept and one indicator for each other group, the model can assign any four values $m_1, \ldots, m_4$ to the four groups. Least squares minimizes $\sum_i\sum_j (x_{ij} - m_i)^2$, and each inner sum is smallest at $m_i = \bar{x}_i$, because the mean minimizes the sum of squared deviations (Measures of Center and Spread);
  • the regression's $R^2$, one minus the residual share of the total sum of squares, is $1 - SSW/SST = SSB/SST = 0.3227$, which is $\eta^2$;
  • the regression's overall $F$ statistic is the ANOVA $F$. Dividing the numerator and denominator of $F$ by $SST$, $$F = \frac{SSB/3}{SSW/36} = \frac{R^2/3}{(1 - R^2)/36} = 5.716.$$

The null hypothesis that all four means are equal is the hypothesis that all three indicator coefficients are zero.

The regression view explains why the analysis of variance extends so readily. Other factors, numerical covariates and interactions enter as more columns of the same design matrix, and the $F$ test compares the fit with and without them.

More than one factor

The groups of the one-way analysis are defined by one factor, age. Often observations are classified in two ways at once: by age group and by sex, or by drug and by dose. Fisher (1925, §42) described this case: each observation "belongs to one class of type A and to a different class of type B", the variance can be found separately between classes of each type, and what remains may include "an interaction of causes, so that a change in class of type A does not have the same effect in all B classes".

A two-way analysis of variance splits the total sum of squares into four parts: one for each factor, one for their interaction and a within-cells part. It tests each of the first three against the within-cells mean square with its own $F$ ratio. An interaction means that the effect of one factor depends on the level of the other: for example, the gap in blood pressure between men and women might be different at different ages. When the interaction is present, the effect of either factor alone is an average over levels of the other, and that average may describe no actual group.

The partition for equal cell sizes. Let factor A have $a$ levels and factor B have $b$, with $r$ observations $x_{ijl}$ in each of the $ab$ cells. Write $\bar{x}_{ij}$ for a cell mean, $\bar{x}_{i\cdot}$ for the mean of all $br$ observations at level $i$ of A, $\bar{x}_{\cdot j}$ for the mean of all $ar$ observations at level $j$ of B, and $\bar{x}$ for the grand mean. Each deviation from the grand mean is a sum of four parts,

$$x_{ijl} - \bar{x} = (\bar{x}_{i\cdot} - \bar{x}) + (\bar{x}_{\cdot j} - \bar{x}) + (\bar{x}_{ij} - \bar{x}_{i\cdot} - \bar{x}_{\cdot j} + \bar{x}) + (x_{ijl} - \bar{x}_{ij}),$$

and squaring and summing over all observations gives

$$SST = \underbrace{br\sum_i(\bar{x}_{i\cdot} - \bar{x})^2}_{SS_A} + \underbrace{ar\sum_j(\bar{x}_{\cdot j} - \bar{x})^2}_{SS_B} + \underbrace{r\sum_i\sum_j(\bar{x}_{ij} - \bar{x}_{i\cdot} - \bar{x}_{\cdot j} + \bar{x})^2}_{SS_{AB}} + \underbrace{\sum_i\sum_j\sum_l(x_{ijl} - \bar{x}_{ij})^2}_{SS_E},$$

because all six cross terms vanish. Write $A_i$, $B_j$, $C_{ij}$ and $E_{ijl}$ for the four parts in the order above. With equal cell sizes, $\bar{x}_{i\cdot}$ is the plain average of the cell means in row $i$, $\bar{x}_{\cdot j}$ of those in column $j$, and $\bar{x}$ of all the row means or all the column means. So:

  • $\sum_l E_{ijl} = 0$ in every cell, since deviations from a cell's own mean sum to zero. The other three parts are constant within a cell, so their cross terms with $E$ vanish.
  • $\sum_j C_{ij} = b\bar{x}_{i\cdot} - b\bar{x}_{i\cdot} - b\bar{x} + b\bar{x} = 0$ for every $i$, and $\sum_i C_{ij} = a\bar{x}_{\cdot j} - a\bar{x} - a\bar{x}_{\cdot j} + a\bar{x} = 0$ for every $j$. Since $A_i$ does not depend on $j$ and $B_j$ does not depend on $i$, the cross terms of $C$ with $A$ and with $B$ vanish.
  • $\sum_i A_i = 0$, so $\sum_i\sum_j A_i B_j = (\sum_i A_i)(\sum_j B_j) = 0$.

The first three parts add up to the between-cells sum of squares $r\sum_i\sum_j(\bar{x}_{ij} - \bar{x})^2$, the $SSB$ of a one-way analysis that treats the $ab$ cells as groups, and $SS_E$ is that analysis's $SSW$. The degrees of freedom split the same way: $a - 1$ for A, $b - 1$ for B, $(a - 1)(b - 1)$ for the interaction and $ab(r - 1)$ within cells, which add up to $abr - 1$.

The tests. For independent normal observations with a common variance, each of the three mean squares, divided by the within-cells mean square $SS_E/[ab(r - 1)]$, has an $F$ distribution under its own null hypothesis (no effect of A, no effect of B, no interaction), with the degrees of freedom just listed. This article states those three distributions and does not prove them. They follow from the general normal theory of tests of linear hypotheses, which Scheffé (1959) develops and applies to the two-way layout.

Example (age group and sex). The script drew 5 men and 5 women at random from each of the four NHANES age groups (seed 1), so $a = 4$, $b = 2$ and $r = 5$. The cell means, in mm Hg:

Age group Men Women Both
18–29 110.93 111.33 111.13
30–44 122.27 110.13 116.20
45–59 126.13 121.60 123.87
60 and over 138.80 136.93 137.87
All ages 124.53 120.00 122.27
Source Sum of squares df Mean square $F$ $p$
Age group 4066.76 3 1355.59 7.827 0.0005
Sex 205.51 1 205.51 1.187 0.284
Interaction 223.02 3 74.34 0.429 0.733
Within cells 5541.87 32 173.18
Total 10037.16 39

The four sums of squares add up to the total, as the partition requires, and the first three add up to the between-cells sum of squares, 4495.29. Age group is significant, as in the one-way analysis. The men's mean is 4.53 mm Hg above the women's, but with 20 of each that difference is well within chance ($p = 0.284$). The interaction is not significant either. In this sample the gap between men and women runs from $-0.40$ mm Hg in the youngest group to 12.13 in the 30–44 group, and the test says that differences of that size between the four gaps are what samples of five would show by chance.

In regression terms, a two-way analysis with interaction is a regression on the indicators for both factors and their products, the interaction terms of multiple regression. With unequal cell sizes the cross terms above no longer vanish, so the sums of squares for the two factors and the interaction no longer add up to the between-cells sum, and the share credited to each factor depends on the order in which the terms are fitted (Scheffé, 1959, discusses the two-way layout with unequal numbers in the cells). The analysis is then done as a comparison of regression models.

What we have learned

A one-way analysis of variance is a regression on indicator variables for the groups: the fitted values are the group means, $R^2$ is $\eta^2$, and the overall $F$ of the regression is the ANOVA $F$. With two factors and equal cell sizes, the total sum of squares splits into parts for each factor, for their interaction and within cells, and each of the first three is tested against the within-cells mean square. On the 40 blood pressures classified by age group and sex, age group was significant and neither sex nor the interaction was. With unequal cell sizes the parts no longer add up, and the analysis becomes a comparison of regression models, the subject of the library's Multiple Linear Regression.

Whichever test rejects, it says only that some means differ, not which. Multiple Comparisons and the Bonferroni Correction takes up that question for the four age groups. Every number in this article is computed by tools/experiments/applied-statistics/one-way-anova.py in the site's repository, the script it shares with One-Way Analysis of Variance.

References