Multivariate Statistics

Statistics for several measurements on each subject at once, with every result proved or cited. The multivariate normal distribution with its conditional distributions and the Schur complement; covariance matrices, the Mahalanobis distance and the exact theory of quadratic forms in normal vectors, with the sample covariance matrix and the Wishart distribution; Hotelling's T-squared test for a mean vector; linear and quadratic discriminant analysis; Gaussian mixture models fitted by EM, with k-means as a limit; and the singular value decomposition beneath principal components and latent semantic indexing. Worked on NHANES blood pressures and body measurements and on the Cranfield abstracts.

The course measured one variable at a time: a mean blood pressure, a proportion with hypertension, a regression of one response on its predictors. Most data come with several measurements to a subject. NHANES records each adult's systolic and diastolic pressure, height and weight together, and the measurements are correlated. Multivariate statistics treats such a vector of measurements as one object. Its mean is a vector, its spread is a covariance matrix, and its questions are about the whole vector: how far an observation lies from the center when the measurements are correlated, whether a mean vector differs from a target, which group a subject belongs to, and how much of the variation a few combinations can carry.

The site already uses this material in places. Principal Component Analysis and Factor Analysis model a covariance matrix, Detecting Outliers uses the Mahalanobis distance, Latent Semantic Indexing truncates a singular value decomposition, k-means clusters by distance to a centroid, Model-Based Clustering and the EM Algorithm fits normal mixtures, and CMA-ES adapts the covariance matrix of a multivariate normal search distribution. This unit is the theory beneath them.

What it assumes. The unit on Probability Theory, especially Expectation, Variance and Covariance (covariance matrices and the variance of a linear combination), Moment Generating and Characteristic Functions, Conditional Expectation and Sampling Distributions from the Normal (projections and Cochran's theorem). Linear algebra at the level of eigenvalues, orthogonal matrices and determinants. The Linear Models unit is used for comparison in places, and the EM algorithm of The EM Algorithm for mixtures.

What is proved and what is cited. Proved across the unit: that a mean vector and covariance matrix determine a multivariate normal distribution, that a vector is multivariate normal exactly when its linear combinations are normal, its density, the independence of uncorrelated blocks, and its conditional distributions with the Schur complement as conditional covariance; which matrices are covariance matrices, the square root and the extreme variances of a covariance matrix, the chi-squared distribution of the Mahalanobis distance, the exact condition for a quadratic form in a normal vector to be chi-squared and for two forms to be independent, the independence of the sample mean and sample covariance matrix of a normal sample, and the moments of the Wishart distribution; the exact F distribution of Hotelling's statistic, its invariance and its equality with the largest squared t statistic over linear combinations; the Bayes rule for normal classes and its linear and quadratic boundaries, and the equivalence of Fisher's criterion with it; the unbounded likelihood of a normal mixture, the EM updates and k-means as their small-variance limit; the existence of the singular value decomposition and the best low-rank approximation theorem. Cited rather than proved, among the results the proofs rest on: the spectral theorem for symmetric matrices and the facts about determinants, from linear algebra; the uniqueness theorem for characteristic functions of random vectors; the linear change of variables for densities and the existence of independent vectors with given distributions; the density of the Wishart distribution; in one converse, the Craig–Sakamoto theorem on determinants; the uniqueness of the generalized inverse (Penrose, 1955); the Bayesian information criterion through Laplace's approximation, from Bayes Factors and Posterior Model Probabilities, the monotonicity of EM, from The EM Algorithm; the spectral-norm best low-rank approximation, from Latent Semantic Indexing; and Wilks's approximation, from Likelihood Ratio, Wald and Score Tests. The comparisons of discriminant analysis with logistic regression report published asymptotic results as their authors state them.

Notation. Vectors are bold columns and $^{\mathsf T}$ is the transpose. A random vector $\mathbf{X}$ has $p$ coordinates, mean vector $\boldsymbol\mu$ and covariance matrix $\Sigma$, with entries $\sigma_{jk}$. The multivariate normal distribution is $N_p(\boldsymbol\mu, \Sigma)$, with the covariance matrix second; for $p = 1$ that is the variance, while elsewhere on the site a one-dimensional normal is written $N(\mu, \sigma)$ with the standard deviation second, so one-dimensional normals are described in words. Partitioned vectors have blocks $\mathbf{X}_1, \mathbf{X}_2$ and the covariance blocks $\Sigma_{11}, \Sigma_{12}, \Sigma_{22}$, with the Schur complement $\Sigma_{11\cdot 2} = \Sigma_{11} - \Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}$. The spectral decomposition is $\Sigma = Q\Lambda Q^{\mathsf T}$ with eigenvalues $\lambda_1 \ge \cdots \ge \lambda_p$. The squared Mahalanobis distance is $\Delta^2(\mathbf{x}) = (\mathbf{x} - \boldsymbol\mu)^{\mathsf T}\Sigma^{-1}(\mathbf{x} - \boldsymbol\mu)$, and $d_i^2$ for an observation of a sample under the sample mean $\bar{\mathbf{x}}$ and the sample covariance matrix $S$, which has divisor $n - 1$. The Wishart distribution with scale $\Sigma$ and $m$ degrees of freedom is $W_p(\Sigma, m)$.

Data. The running data are from the National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018: the systolic and diastolic pressures of 5,232 adults, the three systolic readings of 4,730 adults, those pressures with height and weight for 5,161 adults, and further body measurements for discriminant analysis, mixtures and principal components. The last article also uses the term–document matrix of the 1,400 Cranfield abstracts. As in the other units, the adults are treated as a finite population and the survey weights are ignored. Every number is computed by a script in tools/experiments/statistics/.

Articles

The normal model and its algebra

  • The Multivariate Normal Distribution: the distribution defined as an affine image of independent standard normal variables, determined by its mean vector and covariance matrix through the characteristic function, and characterized by normal linear combinations; affine maps, marginals and sums; the density; independence of uncorrelated blocks, with pairs of normal variables that are uncorrelated but dependent; conditional distributions with a linear mean and the Schur complement as covariance, partial correlations from the inverse covariance matrix, and the degenerate case; checked on the blood pressures, whose conditional means stay within 4 mm Hg of the normal line, though bending away from it by more than normal samples do, while their conditional spread doubles.
  • Covariance Matrices and Quadratic Forms: which matrices are covariance matrices, and what the spectral decomposition gives; the Mahalanobis distance, its invariance and its chi-squared distribution; the exact condition for a quadratic form in a normal vector to be chi-squared, for any covariance matrix, and for a linear and a quadratic form, or two quadratic forms, to be independent; the unbiased sample covariance matrix, its independence from the sample mean under normality, and the Wishart distribution with its moments; Mahalanobis outliers among four NHANES measurements, and what normality adds to the sampling behavior of the covariance matrix.

Inference and classification

  • Hotelling's T-Squared Test: the $t$-test for a mean vector, defined as a squared Mahalanobis distance; its exact F distribution, noncentral under the alternative, proved by a rotation and a residual sum of squares; its invariance, its equality with the largest squared $t$ statistic over linear combinations and the simultaneous intervals that follow, and its derivation as a likelihood ratio test; the two-sample version; and when it is more or less powerful than separate $t$-tests with a Bonferroni correction.
  • Linear Discriminant Analysis: the Bayes rule for normal classes with a shared covariance matrix, proved linear, with its exact error rate in terms of the Mahalanobis distance between class means; quadratic boundaries when the covariance matrices differ; Fisher's criterion and least squares on a class code giving the same direction; the plug-in estimates as maximum likelihood estimates; and a cross-validated comparison with logistic regression on predicting sex from NHANES body measurements.

Structure without labels

  • Gaussian Mixture Models: soft clustering with a mixture of multivariate normal distributions fitted by the EM algorithm; the likelihood proved unbounded and the EM step proved to avoid the collapse; the covariance update; affine equivariance, which k-means lacks; k-means and its Mahalanobis variants as small-variance limits of EM; label switching; the number of components by BIC; and a two-component fit to NHANES heights and weights that splits the adults by build rather than by sex.
  • The Singular Value Decomposition in Statistics: the decomposition proved to exist without the spectral theorem; the best low-rank approximation proved with a uniqueness condition, with the perturbation and interlacing inequalities for singular values; principal components as the decomposition of centered data on seven NHANES body measurements; latent semantic indexing as the uncentered decomposition of the Cranfield term–document matrix; and the pseudo-inverse and ridge regression as filters on singular values.

Where the unit leads

The multivariate normal distribution is the model behind the normal linear model, whose response vector is $N_n(X\boldsymbol\beta, \sigma^2I)$, and behind the asymptotic normality of estimators of several parameters in Asymptotics of the Maximum Likelihood Estimator. The covariance matrix and its eigenvectors return in Principal Component Analysis and Factor Analysis, and the adapted search distribution of CMA-ES is a multivariate normal. The next unit of the collection is Prediction Error and Model Choice.

References

Each article lists its own references. The sources on which the unit draws throughout: record · 2 pages

  • Axler, S. (2024). Linear Algebra Done Right (4th ed.). Springer (open access). Chapter 7 (the spectral theorem, positive operators) and Section 9C (determinants). record · 4 pages
  • Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer. Section 2.3 (the Gaussian distribution) and Chapter 9 (mixture models and EM). record · 10 pages
  • Durrett, R. (2019). Probability: Theory and Examples (5th ed.). Cambridge University Press. Section 3.10 (limit theorems in $\mathbb{R}^d$ and the multivariate normal distribution). record · 10 pages
  • Penrose, R. (1955). A generalized inverse for matrices. Mathematical Proceedings of the Cambridge Philosophical Society, 51(3), 406–413. https://doi.org/10.1017/S0305004100030401 (cited for the uniqueness of the generalized inverse, through The Singular Value Decomposition in Statistics). record · 3 pages