Analysis of Variance

Comparing the means of more than two groups. One-way analysis of variance derived from the partition of the sum of squares, with the F test, its relation to the pooled t-test, its assumptions and Welch's version for unequal variances; and multiple comparison procedures that find which groups differ while controlling the family-wise error rate: Bonferroni, Holm and Tukey. Worked on NHANES blood pressures in four age groups.

Ten adults from each of four age groups have their blood pressure measured, and the four sample means differ, as any four sample means will. The oldest group averages 130 mm Hg and the youngest 108. The previous unit compared two groups; with four there are six pairs, and six $t$-tests at the 0.05 level give a false alarm far more often than 5% of the time. Is there one test of whether the four population means differ at all?

The analysis of variance is that test. It estimates the population variance twice. The first estimate comes from the variation within the groups, and the second from the variation of the group means around their overall mean. If the population means are equal, the two estimates agree; if they are not, the second is inflated. Their ratio has Fisher's $F$ distribution when the null hypothesis holds, and a large ratio is evidence against it.

The unit answers two questions in turn.

  1. Do the group means differ at all? The total sum of squares splits into a part between groups and a part within them, and the ratio of the two variance estimates gives the $F$ test. For two groups it is exactly the pooled $t$-test. When the groups do not share a variance, Welch's version takes over.
  2. Which groups differ? A significant $F$ test says only that the means are not all equal. Finding which ones differ brings back the pairwise comparisons. This time they come with procedures that keep the family-wise error rate, the chance of any false alarm, at the chosen level.

Both articles work one example: systolic blood pressures of 10 adults from each of four age groups in the National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018. In these data both the mean and the spread of blood pressure rise with age. The notation follows the course's formula sheets:

  • $k$ groups of sizes $n_i$, with $n = \sum n_i$;
  • group means $\bar{x}_i$ and grand mean $\bar{x}$;
  • the within-groups variance $s_W^2$ and the between-groups variance $s_B^2$;
  • the test statistic $F = s_B^2/s_W^2$, with $k - 1$ and $n - k$ degrees of freedom.

Every number comes from a script in the site's repository.

Articles

  • One-Way Analysis of Variance: how can one test compare several means? The partition $SST = SSB + SSW$ and its degrees of freedom, the expected mean squares, and the $F$ distribution of $s_B^2/s_W^2$ under the null hypothesis, derived. The worked example and its ANOVA table ($F = 5.716$ on 3 and 36 degrees of freedom, $p = 0.0026$). The proof that $F = t^2$ for two groups. The assumptions, with simulations of what unequal variances and unequal group sizes do together, and Levene's test. Welch's analysis of variance (Welch, 1951) and its reduction to Welch's $t$-test, the analysis of variance as a regression on indicator variables, and a note on two-way designs and interaction. This article is the course's home of the $F$ distribution.
  • Multiple Comparisons and the Bonferroni Correction: once the $F$ test rejects, which groups differ? The family-wise error rate of all pairwise $t$-tests, measured by simulation. The Bonferroni correction proved from the union bound, and Holm's step-down procedure (Holm, 1979) with its proof. Tukey's honestly significant difference from the studentized range, simultaneous confidence intervals and Scheffé's method for contrasts. Then why a significant $F$ test does not protect the comparisons that follow it, and the power each procedure gives up.

The unit builds on Comparison of Two Means. The within-groups variance generalizes the pooled variance of the two-sample $t$-test, the two-group analysis of variance is that test, and Welch's analysis of variance extends Welch's test in the same way. The chi-squared distribution, from which the $F$ distribution is built, is defined in Confidence Intervals for a Mean.

The library's Multiple Linear Regression covers indicator variables and interaction terms. These turn the analysis of variance into a regression and extend it to several factors. In information retrieval, Significance Testing meets the multiple comparisons problem when many search engine variants are compared with one baseline.

The next unit, Nonparametric Methods, drops the assumption of normal populations. For skewed data or outliers, the Kruskal–Wallis test compares several groups by ranks, as the Wilcoxon rank-sum test does for two.

References

  • Fisher, R. A. (1925). Statistical Methods for Research Workers. Oliver and Boyd. Chapter VII. record · 6 pages
  • Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2), 65–70. record · 2 pages
  • NIST/SEMATECH. e-Handbook of Statistical Methods, §7.4.3 (Are the means equal?) and §7.4.7 (How can we make multiple comparisons?). National Institute of Standards and Technology. https://www.itl.nist.gov/div898/handbook/ record · 30 pages
  • Pagano, M., Gauvreau, K., and Mattie, H. (2022). Principles of Biostatistics (3rd ed.). CRC Press. The course text. record · 35 pages
  • Welch, B. L. (1951). On the comparison of several mean values: an alternative approach. Biometrika, 38(3–4), 330–336. record · 2 pages