Systolic blood pressure rises with age. Among the adults examined in the NHANES survey of 2017 to 2018, those aged 18 to 29 averaged 113.18 mm Hg and those aged 60 and over 136.85 mm Hg. A study that measures ten people from each of four age groups sees four sample means that differ, as any four sample means will. The two-sample $t$-test compares two of them; four groups make six pairs, and six $t$-tests at the 0.05 level give a false alarm well over 5% of the time. Is there a single test of whether the four population means are all equal?
Analysis of variance is that test. It estimates the population variance in two ways. One estimate uses the spread within each group, which does not depend on whether the group means differ. The other uses the spread of the group means, which is inflated when they do. If the means are equal, the two estimates agree apart from sampling error; if not, the second is larger. Under the null hypothesis their ratio has Fisher's $F$ distribution, and that gives the test.
First we split the total variation into a part between groups and a part within them, and show what each estimates. Then we derive the distribution of their ratio and work the test by hand on the age groups. Finally we show that for two groups it is the pooled $t$-test, and examine the assumptions, in particular the common variance that Welch's version drops. The article assumes The Two-Sample $t$-Test. It is also the course's home of the $F$ distribution, which Multiple Comparisons and the Bonferroni Correction and the variance test of the two-sample article build on.
Notation and hypotheses
The notation follows the course's formula sheet. There are $k$ groups. Group $i$ is a random sample of size $n_i$ from a population with mean $\mu_i$, and its observations are $x_{i1}, \ldots, x_{in_i}$, with sample mean $\bar{x}_i$ and sample variance $s_i^2$. The groups are independent of one another, and $n = n_1 + \cdots + n_k$ is the total number of observations. The grand mean is the mean of all $n$ observations,
$$\bar{x} = \frac{1}{n}\sum_{i=1}^{k}\sum_{j=1}^{n_i} x_{ij} = \frac{\sum_{i} n_i \bar{x}_i}{\sum_i n_i}.$$
It is the average of the group means weighted by the group sizes, and it equals the plain average of the group means only when the groups are the same size. With unequal groups, averaging the group means directly is a common slip.
The hypotheses are
$$H_0: \mu_1 = \mu_2 = \cdots = \mu_k, \qquad H_A: \text{at least one pair of means differs}.$$
The alternative does not say which means differ, or how many. Rejecting $H_0$ says only that the means are not all equal. Finding which ones differ is the job of multiple comparison procedures.
The model behind the test writes each observation as its group's mean plus an error,
$$x_{ij} = \mu_i + \varepsilon_{ij},$$
with the errors independent and normally distributed with mean 0 and a common variance $\sigma^2$. The NIST/SEMATECH e-Handbook of Statistical Methods (§7.4.3.2) states the same model in the form $\mu + \tau_i$, splitting each group mean into an overall level and a group effect. Both forms make the same three assumptions: independent observations, normal errors, and one variance for all groups.
So the model says the variation in the data comes partly from the group means and partly from the errors. The next step measures each part.
The partition of the sum of squares
Here is the idea in one sentence. Each observation's distance from the grand mean is two distances added: its distance from its own group's mean, and that group mean's distance from the grand mean. Squared and summed over all observations, the two parts still add, with no cross term.
The total variation in the data is measured by the total sum of squares, the squared deviations of all observations from the grand mean:
$$SST = \sum_{i=1}^{k}\sum_{j=1}^{n_i}(x_{ij} - \bar{x})^2.$$
Each deviation splits into a part within the group and a part between groups:
$$x_{ij} - \bar{x} = (x_{ij} - \bar{x}_i) + (\bar{x}_i - \bar{x}).$$
Squaring and summing over all observations,
$$\sum_{i}\sum_{j}(x_{ij} - \bar{x})^2 = \sum_{i}\sum_{j}(x_{ij} - \bar{x}_i)^2 + \sum_{i}\sum_{j}(\bar{x}_i - \bar{x})^2 + 2\sum_{i}(\bar{x}_i - \bar{x})\sum_{j}(x_{ij} - \bar{x}_i).$$
The cross term vanishes, because for each group the inner sum $\sum_j (x_{ij} - \bar{x}_i)$ is the sum of deviations from that group's own mean, which is zero. In the second term, $(\bar{x}_i - \bar{x})^2$ does not depend on $j$, so summing over the $n_i$ members of group $i$ multiplies it by $n_i$. The result is the identity
$$\underbrace{\sum_{i}\sum_{j}(x_{ij} - \bar{x})^2}_{SST} = \underbrace{\sum_{i} n_i(\bar{x}_i - \bar{x})^2}_{SSB} + \underbrace{\sum_{i}\sum_{j}(x_{ij} - \bar{x}_i)^2}_{SSW}.$$
The two parts are:
- the between-groups sum of squares $SSB$, which measures how far the group means are from the grand mean;
- the within-groups sum of squares $SSW$, which measures how far the observations are from their own group's mean.
The NIST handbook (§7.4.3.1) calls this decomposition the guiding principle of the analysis of variance. It names the two parts the treatment and error sums of squares, abbreviated $SST$ and $SSE$, so the handbook's $SST$ is this article's $SSB$. The within-groups sum can be computed from the group variances, since $\sum_j (x_{ij} - \bar{x}_i)^2 = (n_i - 1)s_i^2$:
$$SSW = (n_1 - 1)s_1^2 + (n_2 - 1)s_2^2 + \cdots + (n_k - 1)s_k^2.$$
The degrees of freedom split the same way.
- The total sum of squares has $n - 1$, since the $n$ deviations from the grand mean sum to zero.
- The within-groups sum has $n - k$, since each group contributes $n_i - 1$, having lost one to its own mean.
- The between-groups sum has $k - 1$, since it is built from $k$ group means whose weighted deviations from the grand mean sum to zero.
Indeed $(k - 1) + (n - k) = n - 1$, and the worked example below checks the identity on 40 blood pressures.
Two estimates of the variance
The partition splits the variation, but a sum of squares grows with the number of terms in it. To compare the two parts on a common scale, we divide each by its degrees of freedom. The result is a mean square, and the formula sheet names the two that matter:
$$s_W^2 = \frac{(n_1 - 1)s_1^2 + \cdots + (n_k - 1)s_k^2}{n_1 + n_2 + \cdots + n_k - k}, \qquad s_B^2 = \frac{n_1(\bar{x}_1 - \bar{x})^2 + \cdots + n_k(\bar{x}_k - \bar{x})^2}{k - 1}.$$
The within-groups variance $s_W^2$ is the pooled variance $s_p^2$ of the two-sample $t$-test, extended to $k$ groups. It is a weighted average of the group variances, each weighted by its degrees of freedom. The between-groups variance $s_B^2$ measures the spread of the group means.
What does each one estimate? Taking expected values under the model will show that $s_W^2$ estimates $\sigma^2$ whatever the means are, and $s_B^2$ estimates $\sigma^2$ plus a term that is zero only when the means are equal.
The within-groups variance. Each $S_i^2$ is an unbiased estimate of $\sigma^2$, so $E[(n_i - 1)S_i^2] = (n_i - 1)\sigma^2$. Summing over groups gives $E[SSW] = (n - k)\sigma^2$. Therefore
$$E[s_W^2] = \sigma^2,$$
whether or not the group means are equal. The within-groups variance never sees the differences between groups, because every observation is measured from its own group's mean.
The between-groups variance. Write $\bar{\mu} = \sum_i n_i\mu_i / n$ for the weighted average of the population means, the quantity the grand mean estimates. Expanding the square, $SSB = \sum_i n_i\bar{X}_i^2 - n\bar{X}^2$. Each group mean has mean $\mu_i$ and variance $\sigma^2/n_i$, so $E[\bar{X}_i^2] = \mu_i^2 + \sigma^2/n_i$. The grand mean is $\bar{X} = \sum_i n_i\bar{X}_i/n$. Its mean is $\bar{\mu}$ and its variance is $\sum_i n_i^2(\sigma^2/n_i)/n^2 = \sigma^2/n$, so $E[\bar{X}^2] = \bar{\mu}^2 + \sigma^2/n$. Then
$$E[SSB] = \sum_i n_i\mu_i^2 + k\sigma^2 - n\bar{\mu}^2 - \sigma^2 = (k - 1)\sigma^2 + \sum_i n_i(\mu_i - \bar{\mu})^2,$$
using $\sum_i n_i\mu_i^2 - n\bar{\mu}^2 = \sum_i n_i(\mu_i - \bar{\mu})^2$, the same identity as for any weighted variance. Dividing by $k - 1$,
$$E[s_B^2] = \sigma^2 + \frac{\sum_i n_i(\mu_i - \bar{\mu})^2}{k - 1}.$$
When $H_0$ is true, every $\mu_i$ equals $\bar{\mu}$ and the second term is zero, so $s_B^2$ is a second unbiased estimate of $\sigma^2$. When $H_0$ is false, $s_B^2$ estimates $\sigma^2$ plus a positive amount, which grows with the differences between the means and with the group sizes. This is the whole logic of the test: compare the two estimates, and reject $H_0$ when $s_B^2$ is too much larger than $s_W^2$ to be chance. The NIST handbook (§7.4.3.3) puts it the same way: under the null hypothesis the two mean squares estimate the same quantity, and their ratio should be close to 1.
To check the two expectations, the script drew 300,000 sets of four normal samples of 10. The population means were the four NHANES age-group means, and the common standard deviation was 17.057 mm Hg (the square root of the average of the four population variances). So $\sigma^2 = 290.94$ and $\sum_i n_i(\mu_i - \bar{\mu})^2/(k - 1) = 1074.96$, and the theory predicts $E[s_W^2] = 290.94$ and $E[s_B^2] = 1365.90$. The simulated averages, medians over 30 seeds of 10,000 sets each, were 290.63 and 1367.39.
Check your understanding. Suppose the script had instead given all four populations the same mean, keeping $\sigma^2 = 290.94$. What would the average $s_B^2$ have been?
About 290.94, the same as $s_W^2$. With equal means the extra term $\sum_i n_i(\mu_i - \bar{\mu})^2/(k - 1)$ is zero, so both mean squares estimate $\sigma^2$. The script ran exactly this case, and the median of the simulated averages was 290.49.
The F distribution under the null hypothesis
The test statistic is the ratio on the formula sheet,
$$F = \frac{s_B^2}{s_W^2}.$$
Under $H_0$ it should be near 1. How far above 1 can it go by chance? Here is the plan of the derivation. Each sum of squares, divided by $\sigma^2$, is a chi-squared variable, and the two are independent. The $F$ distribution is defined as a ratio of such variables, each over its degrees of freedom, and $\sigma^2$ cancels from it. For normal populations with a common variance, three facts supply the ingredients.
1. $SSW/\sigma^2$ is chi-squared with $n - k$ degrees of freedom. For a normal sample, $(n_i - 1)S_i^2/\sigma^2$ is chi-squared with $n_i - 1$ degrees of freedom, and independent chi-squared variables add. Confidence Intervals for a Mean defines the chi-squared distribution and states both facts. The groups are independent, so $SSW/\sigma^2 = \sum_i (n_i - 1)S_i^2/\sigma^2$ has $\sum_i (n_i - 1) = n - k$ degrees of freedom. This holds whether or not $H_0$ is true.
2. Under $H_0$, $SSB/\sigma^2$ is chi-squared with $k - 1$ degrees of freedom. This is the one step that needs new work, and the idea is a rotation. When all means equal a common $\mu$, each $\bar{X}_i$ is normal with mean $\mu$ and variance $\sigma^2/n_i$, so the standardized means
$$Z_i = \frac{\sqrt{n_i}\,(\bar{X}_i - \mu)}{\sigma}, \qquad i = 1, \ldots, k,$$
are independent standard normal variables. Let $u$ be the unit vector $(\sqrt{n_1}, \ldots, \sqrt{n_k})/\sqrt{n}$. The grand mean satisfies $\bar{X} - \mu = \sum_i n_i(\bar{X}_i - \mu)/n = (\sigma/\sqrt{n})\,(u \cdot Z)$. The shortcut $\sum_i n_i(a_i - \bar{a})^2 = \sum_i n_i a_i^2 - n\bar{a}^2$, applied to $a_i = \bar{X}_i - \mu$, then gives
$$\frac{SSB}{\sigma^2} = \frac{\sum_i n_i(\bar{X}_i - \mu)^2 - n(\bar{X} - \mu)^2}{\sigma^2} = \sum_{i=1}^{k} Z_i^2 - (u \cdot Z)^2.$$
This is the squared length of $Z$ minus the squared length of its component along $u$. In other words, it is the squared length of the part of $Z$ perpendicular to $u$. Now rotate the coordinates so that $u$ becomes the first axis. Let $Q$ be an orthogonal matrix whose first row is $u$, and set $W = QZ$. Then:
- $W$ has covariance matrix $QQ^{\mathsf{T}} = I$, so its components are again independent standard normal variables;
- $W_1 = u \cdot Z$;
- $\lVert W\rVert^2 = \lVert Z\rVert^2$, since a rotation preserves length.
Hence $SSB/\sigma^2 = W_2^2 + \cdots + W_k^2$, a sum of $k - 1$ independent squared standard normals: chi-squared with $k - 1$ degrees of freedom. One degree of freedom is lost to the grand mean, just as one is lost to the sample mean in an ordinary sample variance.
3. $SSB$ and $SSW$ are independent. $SSB$ is a function of the group means only, and $SSW$ of the group variances only. Within a normal sample the mean and the variance are independent (the third fact in Confidence Intervals for a Mean). The groups are independent of each other, so the collection of means is independent of the collection of variances.
The $F$ distribution. Let $U$ and $V$ be independent chi-squared variables on $d_1$ and $d_2$ degrees of freedom. The random variable $(U/d_1)/(V/d_2)$ has, by definition, the $F$ distribution with $d_1$ and $d_2$ degrees of freedom. Facts 1 to 3 supply exactly such a ratio. Dividing the two chi-squared variables by their degrees of freedom,
$$F = \frac{\dfrac{SSB/\sigma^2}{k - 1}}{\dfrac{SSW/\sigma^2}{n - k}} = \frac{s_B^2}{s_W^2}.$$
The unknown $\sigma^2$ cancels, so under $H_0$ the statistic has the $F$ distribution with $k - 1$ and $n - k$ degrees of freedom, written $F_{k-1,\,n-k}$ on the formula sheet.
Check your understanding. The alternative allows the means to differ in any direction. Should the test then reject in both tails of $F$, as a two-sided $t$-test does?
No. A false null hypothesis inflates only the numerator, whichever direction the means differ in. A small $F$ says the group means are unusually close together, which is no evidence that they differ. So the test is one-sided in $F$. At level $\alpha$ it rejects when $F > F_{k-1,\,n-k,\,1-\alpha}$, the point with probability $1 - \alpha$ below it (the formula sheet calls it $F^*$), and the $p$-value is $P(F_{k-1,\,n-k} \ge F)$.
When $H_0$ is false, the ratio has a noncentral $F$ distribution. Its noncentrality, $\sum_i n_i(\mu_i - \bar{\mu})^2/\sigma^2$, is the same quantity that appeared in $E[s_B^2]$, and the larger it is, the more power the test has.
Fisher's Statistical Methods for Research Workers (1925) presented the analysis of variance as a general method. In it (§41) he reduced all of its tests to "the single problem of testing whether one estimate of variance derived from $n_1$ degrees of freedom is significantly greater than a second such estimate derived from $n_2$ degrees of freedom". He tabulated $z$, half the natural logarithm of the ratio of the two estimates, whereas modern tables and software give the ratio itself.
The simulation behind the figure checks the derivation. For four normal samples of 10 with equal means, the upper 5% point of the simulated $F$ was 2.8516 (median over 30 seeds; interquartile range 2.8397 to 2.8786), against $F_{3,\,36,\,0.95} = 2.8663$. The test rejected in 4.93% of samples.
A worked example: blood pressure by age
Back to the four age groups. The National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018 measured the blood pressure of 5,247 adults, each person's value being the mean of up to three systolic readings. As elsewhere in this course, these adults are treated as a population to sample from:
| Age group (NHANES 2017–2018 adults) | Size | Mean (mm Hg) | Standard deviation (mm Hg) |
|---|---|---|---|
| 18–29 | 986 | 113.18 | 11.32 |
| 30–44 | 1,132 | 118.75 | 15.36 |
| 45–59 | 1,206 | 127.56 | 18.91 |
| 60 and over | 1,923 | 136.85 | 21.02 |
The survey records ages of 80 and over as 80, so the oldest group is open-ended. Mean blood pressure rises by about 24 mm Hg from the youngest group to the oldest, and the spread nearly doubles. The oldest group's standard deviation is 1.86 times the youngest's, a variance ratio of 3.45, and that second fact matters below.
The script drew 10 adults at random from each age group (seed 1):
| Age group | $n_i$ | $\bar{x}_i$ | $s_i$ | $s_i^2$ |
|---|---|---|---|---|
| 18–29 | 10 | 108.267 | 8.387 | 70.34 |
| 30–44 | 10 | 111.567 | 14.234 | 202.62 |
| 45–59 | 10 | 115.233 | 11.257 | 126.72 |
| 60 and over | 10 | 130.333 | 16.282 | 265.11 |
Check your understanding. Before computing, look at the four sample means. Which group will contribute most to the between-groups sum of squares, and roughly what share?
The oldest. Its mean is 14 mm Hg above the grand mean and the others at most 8.1 mm Hg from it, and contributions grow with the squared distance. Step 2 finds that it supplies 1955.34 of the 2850.01, about two thirds.
Step 1, the grand mean. With equal group sizes it is the plain average of the group means: $\bar{x} = (108.267 + 111.567 + 115.233 + 130.333)/4 = 116.350$ mm Hg.
Step 2, the between-groups sum of squares. Each group contributes $n_i(\bar{x}_i - \bar{x})^2$: $10(108.267 - 116.350)^2 = 653.40$, then 228.80, 12.47 and 1955.34. Their sum is $SSB = 2850.01$.
Step 3, the within-groups sum of squares. Each group contributes $(n_i - 1)s_i^2$: $9 \times 70.34 = 633.07$, then 1823.57, 1140.46 and 2386.00, for $SSW = 5983.09$ (the parts are rounded). Computing $SST$ directly from the 40 observations gives 8833.10, which is $SSB + SSW$, as the partition requires.
Step 4, the two variance estimates.
$$s_B^2 = \frac{2850.01}{4 - 1} = 950.00, \qquad s_W^2 = \frac{5983.09}{40 - 4} = 166.20.$$
With equal group sizes, $s_W^2$ is the plain average of the four sample variances. Its square root, $s_W = 12.89$ mm Hg, is the estimated common standard deviation.
Step 5, the test.
$$F = \frac{950.00}{166.20} = 5.716$$
on 3 and 36 degrees of freedom. The critical value at $\alpha = 0.05$ is $F_{3,\,36,\,0.95} = 2.8663$, and at $\alpha = 0.01$ it is 4.3771, so $H_0$ is rejected at both levels, with $p = P(F_{3,36} \ge 5.716) = 0.0026$.
The results are usually set out in an ANOVA table, in the layout of the NIST handbook (§7.4.3.3):
| Source | Sum of squares | df | Mean square | $F$ | $p$ |
|---|---|---|---|---|---|
| Between groups | 2850.01 | 3 | $s_B^2 = 950.00$ | 5.716 | 0.0026 |
| Within groups | 5983.09 | 36 | $s_W^2 = 166.20$ | ||
| Total | 8833.10 | 39 |
The data are strong evidence that mean blood pressure is not the same in all four age groups, which is true of the populations. A tempting reading goes further: "so every group differs from every other." The test does not say that, only that the means are not all equal. The sample means suggest that the oldest group stands apart and the three younger ones are close together. Yet in the population the 30–44 and 45–59 groups differ by 8.8 mm Hg. Multiple Comparisons and the Bonferroni Correction takes these samples and asks which pairwise differences they can establish.
A useful companion to the test is the share of the total variation that lies between groups,
$$\eta^2 = \frac{SSB}{SST} = \frac{2850.01}{8833.10} = 0.3227.$$
Age group accounts for about a third of the variation in these 40 blood pressures. Unlike $F$ and its $p$-value, $\eta^2$ does not grow with the sample size for a fixed pattern of means, so it describes the size of the effect, not the strength of the evidence. In the four populations the between-groups share of the total sum of squares is 0.2119, smaller than in this sample, because the population means differ by less, relative to their spread, than this sample's do.
Two groups: F is t squared
For two groups the previous unit already had a test, the pooled two-sample $t$-test. Can the two disagree? No: with $k = 2$ they are the same test, and $F$ is the square of $t$.
Write $d = \bar{x}_1 - \bar{x}_2$. The grand mean is $\bar{x} = (n_1\bar{x}_1 + n_2\bar{x}_2)/n$, so $\bar{x}_1 - \bar{x} = n_2 d/n$ and $\bar{x}_2 - \bar{x} = -n_1 d/n$. Then
$$SSB = n_1\left(\frac{n_2 d}{n}\right)^2 + n_2\left(\frac{n_1 d}{n}\right)^2 = \frac{n_1 n_2 (n_2 + n_1)}{n^2}\,d^2 = \frac{n_1 n_2}{n}\,d^2 = \frac{d^2}{1/n_1 + 1/n_2},$$
and with $k - 1 = 1$, $s_B^2 = SSB$. The within-groups variance for two groups is exactly the pooled variance, $s_W^2 = s_p^2$, on $n - 2 = n_1 + n_2 - 2$ degrees of freedom. So
$$F = \frac{s_B^2}{s_W^2} = \frac{d^2}{s_p^2\,(1/n_1 + 1/n_2)} = t^2,$$
the square of the pooled $t$ statistic.
The distributions match too: if $T$ has a $t$ distribution with $\nu$ degrees of freedom, then $T = Z/\sqrt{V/\nu}$ with $Z$ standard normal and $V$ chi-squared. So $T^2 = (Z^2/1)/(V/\nu)$, and $Z^2$ is chi-squared with 1 degree of freedom: $T^2$ has the $F_{1,\,\nu}$ distribution. Since $P(T^2 > c^2) = P(|T| > c)$, the $F$ test's $p$-value equals the two-sided $t$-test's, and $F_{1,\,\nu,\,1-\alpha} = t_{\nu,\,1-\alpha/2}^2$. Fisher (1925, §41) noted this equivalence: with only two classes, his test "would have been equivalent to testing the significance of $t$", and his table of $z$ for one degree of freedom in the numerator is just a transformed table of $t$.
Check your understanding. For the 20 men and 20 women of the two-sample $t$-test article, the pooled test gave $t = 0.6740$ with $p = 0.5044$. What $F$ and $p$-value will the analysis of variance give on the same sample?
$F = 0.6740^2 = 0.4543$ on 1 and 38 degrees of freedom, with the same $p$-value, 0.5044. The script confirms it on that sample (seed 1): $s_W^2 = 376.06 = s_p^2$, $s_B^2 = SSB = 170.844$, and $F = 0.4543$. The critical values agree as well: $F_{1,\,38,\,0.95} = 4.0982 = 2.0244^2$.
One consequence: the one-way analysis of variance extends the pooled two-sample $t$-test and inherits that test's assumption of equal variances, with the same consequences when the assumption fails. A one-sided comparison of two groups ($H_A: \mu_1 > \mu_2$) has no counterpart in $F$, which squares away the sign; for that, the $t$-test is used.
Assumptions
The derivation of the $F$ distribution used three assumptions: independent observations, normal populations, and a common variance. The age groups already break the third. How much does each matter? The three are far from equal in importance, as the rest of this section shows.
Independence is the most important, and no formula repairs a failure of it. Observations must be independent within each group, and the groups independent of each other. A common violation measures the same subjects under every condition, for example blood pressure on the same patients under three drugs. The measurements on one patient are correlated, so the groups are not independent, and a one-way analysis overstates the variation that the comparison has to overcome. Such data call for a repeated-measures analysis, the many-condition counterpart of the paired $t$-test, which is outside this article.
Normality matters less, especially with equal group sizes of moderate size. The reason is the same as for the $t$-test: the statistic depends on the data through group means, which are close to normal by the central limit theorem. The two-sample $t$-test article shows how much more robust tests on means are than tests on variances. The simulations below agree: the actual NHANES age groups are right-skewed (skewness 0.75 to 1.10), yet drawing from them gave the classical test the same error rates as drawing from normal populations with the same standard deviations. For strongly skewed data or outliers, the Kruskal–Wallis test compares the groups by ranks.
Equal variances matter a great deal when the group sizes differ. The reason is visible in the expected mean squares. If the population variances $\sigma_i^2$ differ, the same algebra as above gives, under $H_0$,
$$E[SSW] = \sum_i (n_i - 1)\sigma_i^2, \qquad E[SSB] = \sum_i \left(1 - \frac{n_i}{n}\right)\sigma_i^2.$$
The two weightings of the group variances differ. With equal group sizes they agree (both are proportional to the plain average of the $\sigma_i^2$), so $s_B^2$ and $s_W^2$ still estimate the same quantity under $H_0$. With unequal sizes, $s_W^2$ gives the large groups the most weight, while $s_B^2$ gives them the least.
Check your understanding. Suppose the largest groups have the smallest variances. Will the classical test reject a true null hypothesis too often or too rarely?
Too often. The small variances dominate $s_W^2$, so the denominator is too small relative to $s_B^2$, and $F$ runs large. When the large groups have the large variances, the reverse happens and the test becomes too cautious.
The script measured the Type I error rate at $\alpha = 0.05$ for four populations with equal means. Their standard deviations were those of the NHANES age groups (11.32, 15.36, 18.91 and 21.02 mm Hg). There were three designs:
- equal groups of 10;
- group sizes 20, 15, 10 and 5, so that the largest group has the smallest spread;
- sizes 5, 10, 15 and 20, the reverse.
Each entry is the median over 30 seeds of 10,000 samples. The last column is the ratio $E[s_B^2]/E[s_W^2]$ under $H_0$, which is 1 when the test is fair.
| Group sizes (SDs increasing) | Classical $F$ | Welch's ANOVA | $E[s_B^2] / E[s_W^2]$ |
|---|---|---|---|
| 10, 10, 10, 10 | 0.056 | 0.051 | 1.000 |
| 20, 15, 10, 5 | 0.117 | 0.058 | 1.324 |
| 5, 10, 15, 20 | 0.027 | 0.053 | 0.783 |
| 20, 15, 10, 5, with equal SDs | 0.050 | 0.056 | 1.000 |
Sampling from the actual NHANES age groups gave nearly the same rates. These populations were shifted to a common mean, so that $H_0$ holds but their skewness and unequal spreads remain. The classical test's error rates were 0.056, 0.116 and 0.026, and Welch's 0.052, 0.067 and 0.051.
The pattern is the one the two-sample $t$-test showed. With equal group sizes the classical test is close to its nominal level, even with a variance ratio of 3.45. With unequal sizes it fails in the direction the expected mean squares predict: when the small groups are the variable ones, it rejects a true null hypothesis more than twice as often as it should. Unequal group sizes on their own do no harm (the last row); it is the combination with unequal variances that does. A common rule of thumb calls for caution when the largest group is more than twice the smallest. It can be stated more precisely: unequal sizes matter only in combination with unequal variances, and then even modest imbalance changes the error rate.
Why not test the variances first and choose the analysis accordingly? Because a preliminary test is a poor guide.
- The Levene test (NIST handbook §1.3.5.10) is itself a one-way analysis of variance, applied to each observation's absolute deviation from its group's median. The median version, due to Brown and Forsythe, is the one NIST recommends for its robustness. On the worked sample, the mean absolute deviations were 6.133, 10.967, 7.500 and 13.800 mm Hg, and the test gave $W = 2.031$ on 3 and 36 degrees of freedom, $p = 0.127$: it did not detect a variance ratio of 3.45 between the populations.
- Bartlett's test, the classical alternative, is sensitive to non-normality. With non-normal data it "may simply be testing for non-normality" (NIST handbook §1.3.5.7).
The two-sample $t$-test article made the case against a preliminary test of variances for two groups, and it applies unchanged. The better course is a test that does not need the assumption.
Welch's analysis of variance
Welch (1951) extended his unequal-variance $t$-test to $k$ groups. Here is the idea in one sentence: compare each group mean with the others in units of its own standard error, so that a noisy group counts for less. The statistic weights each group by the precision of its mean, $w_i = n_i/s_i^2$, the reciprocal of the estimated variance of $\bar{x}_i$. With $W = \sum_i w_i$ and the precision-weighted mean $\tilde{x} = \sum_i w_i\bar{x}_i / W$,
$$F_W = \frac{\dfrac{1}{k - 1}\displaystyle\sum_i w_i(\bar{x}_i - \tilde{x})^2}{1 + \dfrac{2(k - 2)}{k^2 - 1}\,\Lambda}, \qquad \Lambda = \sum_i \frac{(1 - w_i/W)^2}{n_i - 1},$$
referred to an $F$ distribution with $k - 1$ and
$$\nu_2 = \frac{k^2 - 1}{3\Lambda}$$
degrees of freedom. In the numerator, a noisy group cannot dominate the comparison as it can in $s_W^2$. The denominator and $\nu_2$ come from an approximation of the same kind as the Satterthwaite degrees of freedom, and $\Lambda$ plays the same role: it is small when every group has many observations, and grows when the comparison leans on a small, imprecise group. In scipy, f_oneway with equal_var=False computes this statistic, and the script's implementation agrees with it to the printed digits.
Two groups. With $k = 2$ the correction term vanishes ($k - 2 = 0$), and the statistic reduces to Welch's $t$-test. Write $v_i = s_i^2/n_i = 1/w_i$. Then $1 - w_1/W = w_2/W = v_1/(v_1 + v_2)$, and likewise for group 2, so
$$\Lambda = \frac{v_1^2/(n_1 - 1) + v_2^2/(n_2 - 1)}{(v_1 + v_2)^2}, \qquad \nu_2 = \frac{1}{\Lambda} = \frac{(v_1 + v_2)^2}{v_1^2/(n_1 - 1) + v_2^2/(n_2 - 1)},$$
the Welch–Satterthwaite degrees of freedom. The numerator is $\sum_i w_i(\bar{x}_i - \tilde{x})^2 = \dfrac{w_1 w_2}{w_1 + w_2}\,d^2 = \dfrac{d^2}{v_1 + v_2}$, the square of Welch's $t$. For the 10 older and 30 younger adults of the two-sample article, Welch's ANOVA gives $F_W = 11.4332$ on 1 and 10.1881 degrees of freedom. Welch's $t$ there was 3.3813 on 10.1881 degrees of freedom: $3.3813^2 = 11.4332$, and both $p$-values are 0.0068.
The worked example. For the four age groups the weights are $w_i = 10/s_i^2$: 0.14217, 0.04935, 0.07892 and 0.03772, summing to $W = 0.30815$. The youngest group, the least variable, gets nearly four times the weight of the oldest. The weighted mean, $\tilde{x} = 113.280$ mm Hg, is below the grand mean of 116.350 because the precise groups are the younger ones. The numerator is 4.996, and the terms of $\Lambda$ are 0.03224, 0.07837, 0.06149 and 0.08557, so $\Lambda = 0.25767$ and the denominator is $1 + (4/15)(0.25767) = 1.06871$. So
$$F_W = \frac{4.996}{1.06871} = 4.675 \quad \text{on 3 and } \nu_2 = \frac{15}{3 \times 0.25767} = 19.40 \text{ degrees of freedom},$$
with $p = 0.0128$. The conclusion is the same as the classical test's at the 0.05 level, with a larger $p$-value (0.0128 against 0.0026). Part of the difference is the statistic: the oldest group, whose mean stands farthest from the others, is also the most variable and gets the least weight. The rest is the degrees of freedom: referred to 36 instead of 19.40, the same $F_W$ would give $p = 0.0074$.
Its level and its cost. In the simulations above, Welch's ANOVA stayed within 0.008 of the 0.05 level with one exception: with the skewed NHANES populations and the most variable group the smallest (5), it reached 0.067. With equal variances it gives up a little power. The script tested four normal groups with a common standard deviation and means 0, 0.3, 0.6 and 0.9 standard deviations apart. With 10 per group, the classical test rejected in 36.3% of samples and Welch's in 33.6%; with 5 per group, 17.5% against 15.1%. As for two groups, the choice is between a small, predictable loss of power and an error rate that depends on unknown variances. With equal group sizes the classical test is nearly safe; with unequal sizes and any reason to expect unequal spreads, Welch's version is the better default.
ANOVA as regression
How does the method grow beyond one factor? The answer starts from another view of the same analysis: the one-way analysis of variance is a special case of linear regression.
Code the age group with three indicator variables, one each for 30–44, 45–59 and 60 and over, with 18–29 as the baseline. Then fit a multiple linear regression of blood pressure on them, as described there for categorical inputs. For the worked sample the fitted intercept is 108.267, the mean of the baseline group. The three coefficients are 3.300, 6.967 and 22.067, the differences between each group's mean and the baseline's. Every quantity of the analysis of variance reappears:
- the fitted value for every person is their group's mean, so the residual sum of squares is $SSW$ and the residual mean square is $s_W^2 = 166.20$;
- the regression's $R^2$ is 0.3227, which is $\eta^2$;
- the regression's overall $F$ statistic, $(R^2/3)/((1 - R^2)/36) = 5.716$, is the ANOVA $F$.
The null hypothesis that all four means are equal is the hypothesis that all three indicator coefficients are zero.
The regression view explains why the analysis of variance extends so readily. Other factors, numerical covariates and interactions enter as more columns of the same design matrix, and the $F$ test compares the fit with and without them.
More than one factor
The groups here are defined by one factor, age. Often observations are classified in two ways at once: by age group and by sex, or by drug and by dose. Fisher (1925, §42) described this case: each observation "belongs to one class of type A and to a different class of type B", the variance can be found separately between classes of each type, and what remains may include "an interaction of causes, so that a change in class of type A does not have the same effect in all B classes".
A two-way analysis of variance splits the total sum of squares into four parts: one for each factor, one for their interaction and a within-cells part. It tests each of the first three against the within-cells mean square with its own $F$ ratio. An interaction means that the effect of one factor depends on the level of the other: for example, the gap in blood pressure between men and women might be different at different ages. When the interaction is present, the effect of either factor alone is an average over levels of the other, and that average may describe no actual group.
In regression terms, a two-way analysis with interaction is a regression on the indicators for both factors and their products, the interaction terms of multiple regression. With the same number of observations in every cell, the sums of squares for the two factors and the interaction add up exactly to the between-cells sum of squares, as in the one-way partition. With unequal cell sizes they no longer separate uniquely, and the analysis is best done as a comparison of regression models.
What we have learned
The analysis of variance compares several means by comparing two estimates of one variance. The within-groups mean square estimates $\sigma^2$ whatever the means are, and the between-groups mean square is inflated when they differ. For normal populations with a common variance, their ratio has the $F$ distribution with $k - 1$ and $n - k$ degrees of freedom. For two groups it is the square of the pooled $t$ statistic. On the 40 blood pressures, $F = 5.716$ and $p = 0.0026$. The test is robust to moderate skewness, but not to unequal variances combined with unequal group sizes, where Welch's version holds its level.
| Classical one-way ANOVA | Welch's ANOVA | |
|---|---|---|
| Hypotheses | $H_0: \mu_1 = \cdots = \mu_k$; $H_A$: at least one pair differs | Same |
| Statistic | $F = s_B^2/s_W^2$ | $F_W$, means weighted by $n_i/s_i^2$ |
| Null distribution | $F_{k-1,\,n-k}$, exact for normal populations with equal variances | Approximately $F_{k-1,\,\nu_2}$ |
| Two groups | Pooled $t$ squared | Welch's $t$ squared |
| When variances differ | Fine with equal $n_i$; wrong error rates with unequal $n_i$ | Holds its level, slightly liberal with very small groups |
A significant $F$ leaves the obvious next question open: which of the four age groups differ from which? Answering it means returning to the six pairwise comparisons the analysis of variance was meant to replace, without letting their false alarms multiply. That is the subject of Multiple Comparisons and the Bonferroni Correction.
In Python
import numpy as np
from scipy import stats
# the worked sample: 10 adults from each NHANES age group (mm Hg)
g1 = [114.7, 110.7, 110.0, 114.0, 118.7, 114.0, 96.7, 100.0, 110.0, 94.0] # 18-29
g2 = [116.0, 116.7, 131.3, 110.7, 112.7, 88.7, 96.0, 121.3, 95.0, 127.3] # 30-44
g3 = [108.0, 112.7, 137.0, 103.3, 116.7, 114.0, 108.7, 112.7, 106.0, 133.3] # 45-59
g4 = [144.7, 156.0, 108.7, 134.0, 110.0, 122.7, 144.0, 126.0, 142.0, 115.3] # 60+
print(stats.f_oneway(g1, g2, g3, g4)) # F = 5.719, p = 0.0026
print(stats.f_oneway(g1, g2, g3, g4, equal_var=False)) # Welch: F = 4.672, p = 0.0128
print(stats.levene(g1, g2, g3, g4, center='median')) # W = 2.028, p = 0.127
The values are rounded to 0.1 mm Hg, so the statistics differ slightly from the article's, which use the unrounded means of each person's readings ($F = 5.716$, $F_W = 4.675$, $W = 2.031$). The equal_var argument of f_oneway was added in scipy 1.16.0. In R, oneway.test performs Welch's analysis by default (its documentation describes it as generalizing "the commonly known 2-sample Welch test to the case of arbitrarily many samples"), and its argument var.equal = TRUE gives the classical $F$ test. Every number in this article is computed by tools/experiments/applied-statistics/one-way-anova.py in the site's repository, which implements the $F$ distribution directly and was checked against scipy 1.17.1.
References
- Fisher, R. A. (1925). Statistical Methods for Research Workers. Oliver and Boyd. Chapter VII, §§41–42. Online edition: Classics in the History of Psychology, York University. https://psychclassics.yorku.ca/Fisher/Methods/ record · 6 pages
- National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018: Demographic Variables and Sample Weights (DEMO_J) and Blood Pressure (BPX_J) data files. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 45 pages
- NIST/SEMATECH. e-Handbook of Statistical Methods, §1.3.5.7 (Bartlett's test), §1.3.5.10 (Levene test for equality of variances), §7.4.3.1 (One-way ANOVA overview), §7.4.3.2 (The one-way ANOVA model and assumptions) and §7.4.3.3 (The ANOVA table and tests of hypotheses about means). National Institute of Standards and Technology. https://www.itl.nist.gov/div898/handbook/ record · 30 pages
- Pagano, M., Gauvreau, K., and Mattie, H. (2022). Principles of Biostatistics (3rd ed.). CRC Press. The course text. record · 35 pages
- R Core Team.
oneway.test: Test for Equal Means in a One-Way Layout. Documentation for package stats, R. https://stat.ethz.ch/R-manual/R-devel/library/stats/html/oneway.test.html record - SciPy.
scipy.stats.f_oneway. SciPy reference documentation. https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.f_oneway.html record - Welch, B. L. (1951). On the comparison of several mean values: an alternative approach. Biometrika, 38(3–4), 330–336. record · 2 pages