The National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018 measured the blood pressure of 5,232 adults who have both a systolic and a diastolic value. The two pressures rise together: their correlation is 0.453. Suppose an adult's systolic pressure is 150 mm Hg. What is the distribution of the diastolic pressure of adults like this one, and how far does knowing the systolic pressure narrow it? If the pair has a joint normal distribution, the answer is a normal distribution whose mean is a straight line in the systolic pressure and whose spread is the same whatever the systolic pressure is. This article says what "joint normal" means, proves that answer and the other properties of the distribution, and checks them against the 5,232 adults.
A random vector has a multivariate normal distribution if it is a fixed vector plus a matrix times a vector of independent standard normal variables. This article is the home of that distribution for the Multivariate Statistics unit: later articles cite its theorems from here. The algebra of covariance matrices that it uses, the square root of a matrix and the chi-squared distribution of the Mahalanobis distance, is proved in Covariance Matrices and Quadratic Forms.
First we set out the tools: mean vectors, covariance matrices and characteristic functions of random vectors. Then we define the distribution and prove that its mean vector and covariance matrix determine it, and that a vector is multivariate normal exactly when all its linear combinations are normal. Next come affine maps, the density and the link between zero covariance and independence, with two cautionary pairs. Then we derive conditional distributions, the Schur complement and partial correlations, and check them on the blood pressures. Finally we treat the degenerate case. The article assumes Expectation, Variance and Covariance, Moment Generating and Characteristic Functions and Conditional Expectation and the Laws of Total Expectation and Variance.
Random vectors and their first two moments
Vectors are columns, written in bold; matrices are capitals; $^{\mathsf T}$ is the transpose. A random vector $\mathbf{X} = (X_1, \ldots, X_p)^{\mathsf T}$ is a list of $p$ random variables on the same experiment. When each coordinate has a finite variance, its mean vector is $\boldsymbol\mu = E[\mathbf{X}]$, the vector of the means $\mu_j = E[X_j]$, and its covariance matrix is the $p \times p$ matrix
$$\Sigma = \operatorname{Cov}(\mathbf{X}) = E\big[(\mathbf{X} - \boldsymbol\mu)(\mathbf{X} - \boldsymbol\mu)^{\mathsf T}\big], \qquad \sigma_{jk} = \operatorname{Cov}(X_j, X_k),$$
with the variances $\sigma_{jj} = \sigma_j^2$ on its diagonal. The expected value of a random matrix is taken entry by entry. Two facts are proved in Expectation, Variance and Covariance (Theorem 6 and the paragraph after it): for a fixed $q \times p$ matrix $B$ and vector $\mathbf{c}$,
$$E[B\mathbf{X} + \mathbf{c}] = B\boldsymbol\mu + \mathbf{c}, \qquad \operatorname{Cov}(B\mathbf{X} + \mathbf{c}) = B\Sigma B^{\mathsf T},$$
and, with $B$ a single row $\mathbf{a}^{\mathsf T}$, $\operatorname{Var}(\mathbf{a}^{\mathsf T}\mathbf{X}) = \mathbf{a}^{\mathsf T}\Sigma\mathbf{a} \ge 0$: a covariance matrix is symmetric and positive semidefinite. For two random vectors the cross-covariance $\operatorname{Cov}(\mathbf{X}, \mathbf{Y}) = E[(\mathbf{X} - E\mathbf{X})(\mathbf{Y} - E\mathbf{Y})^{\mathsf T}]$ collects the covariances of every coordinate of one with every coordinate of the other, and $\operatorname{Cov}(B\mathbf{X}, C\mathbf{Y}) = B\operatorname{Cov}(\mathbf{X}, \mathbf{Y})C^{\mathsf T}$ by the same theorem.
For the 5,232 adults, treated as a finite population as in the other units, the mean vector of (systolic, diastolic) is $(126.31, 72.49)$ mm Hg and the covariance matrix, with divisor $N$, is
$$\Sigma = \begin{pmatrix} 402.65 & 109.05 \\ 109.05 & 144.14 \end{pmatrix},$$
so the standard deviations are 20.07 and 12.01 and the correlation is $109.05/(20.07 \times 12.01) = 0.453$. Each adult's pressure is the mean of up to three readings, and a diastolic reading of 0 is treated as missing. Standard deviations within a subgroup (a band of systolic pressure, an age group) use the divisor $n - 1$ of the subgroup.
Characteristic functions of vectors. The tool that makes the multivariate normal distribution easy to handle is the characteristic function of a vector,
$$\varphi_{\mathbf{X}}(\mathbf{t}) = E\big[e^{i\mathbf{t}^{\mathsf T}\mathbf{X}}\big] = E[\cos(\mathbf{t}^{\mathsf T}\mathbf{X})] + i\,E[\sin(\mathbf{t}^{\mathsf T}\mathbf{X})], \qquad \mathbf{t} \in \mathbb{R}^p,$$
defined for every random vector because the cosine and sine are bounded. It is the characteristic function of the single variable $\mathbf{t}^{\mathsf T}\mathbf{X}$ at the point 1, $\varphi_{\mathbf{X}}(\mathbf{t}) = \varphi_{\mathbf{t}^{\mathsf T}\mathbf{X}}(1)$, so everything the article on characteristic functions proves about one variable applies along each direction $\mathbf{t}$. One theorem is cited.
Uniqueness (cited). If two random vectors have the same characteristic function at every $\mathbf{t}$, they have the same distribution. Durrett (2019, Theorem 3.10.4) proves the inversion formula for random vectors, which recovers the probability of every box whose boundary carries no probability from the characteristic function; uniqueness follows from it in two steps. For each coordinate, only countably many values $a$ give the hyperplane $\{x_j = a\}$ positive probability, so boxes whose corners avoid them determine the joint distribution function, as in the one-dimensional corollary of Moment Generating and Characteristic Functions; and a joint distribution function determines the joint distribution, a fact Random Variables and Distribution Functions takes as given.
From uniqueness comes a test for independence.
Lemma 1 (independence and characteristic functions; for random variables, Durrett, 2019, Exercise 3.10.6). Let $\mathbf{X} = (\mathbf{X}_1^{\mathsf T}, \mathbf{X}_2^{\mathsf T})^{\mathsf T}$, with $\mathbf{t} = (\mathbf{t}_1^{\mathsf T}, \mathbf{t}_2^{\mathsf T})^{\mathsf T}$ split in the same way. The vectors $\mathbf{X}_1$ and $\mathbf{X}_2$ are independent if and only if $\varphi_{\mathbf{X}}(\mathbf{t}) = \varphi_{\mathbf{X}_1}(\mathbf{t}_1)\,\varphi_{\mathbf{X}_2}(\mathbf{t}_2)$ for every $\mathbf{t}$.
Proof. Suppose $\mathbf{X}_1$ and $\mathbf{X}_2$ are independent, and write $a = \mathbf{t}_1^{\mathsf T}\mathbf{X}_1$ and $b = \mathbf{t}_2^{\mathsf T}\mathbf{X}_2$. Then $e^{i(a + b)} = (\cos a\cos b - \sin a\sin b) + i(\sin a\cos b + \cos a\sin b)$. Each of the four products is a bounded function of $\mathbf{X}_1$ times a bounded function of $\mathbf{X}_2$. Such functions are independent (Random Variables and Distribution Functions, Theorem 6, part 3), so the expected value of each product is the product of the expected values (Expectation, Variance and Covariance, Theorem 5). Recombining the four terms gives $E[e^{ia}]\,E[e^{ib}]$.
Conversely, suppose the characteristic function factors. Let $\mathbf{Y}_1$ and $\mathbf{Y}_2$ be independent random vectors with the distributions of $\mathbf{X}_1$ and $\mathbf{X}_2$. Such a pair exists: it is the product of the two distributions, a construction the probability unit takes for granted. By the first part, $\varphi_{(\mathbf{Y}_1, \mathbf{Y}_2)}(\mathbf{t}) = \varphi_{\mathbf{X}_1}(\mathbf{t}_1)\varphi_{\mathbf{X}_2}(\mathbf{t}_2) = \varphi_{\mathbf{X}}(\mathbf{t})$, so by uniqueness $(\mathbf{Y}_1, \mathbf{Y}_2)$ has the distribution of $\mathbf{X}$. Then for any sets $A$ and $B$, $P(\mathbf{X}_1 \in A, \mathbf{X}_2 \in B) = P(\mathbf{Y}_1 \in A)\,P(\mathbf{Y}_2 \in B) = P(\mathbf{X}_1 \in A)\,P(\mathbf{X}_2 \in B)$. $\square$
The same argument with more than two blocks shows that $\mathbf{X}_1, \ldots, \mathbf{X}_k$ are mutually independent exactly when the characteristic function is the product of the $k$ characteristic functions of the blocks.
The definition, and what determines the distribution
Here is the idea of the definition. The one-dimensional normal family is the set of variables $\mu + \sigma Z$ obtained from one standard normal $Z$ by a shift and a stretch. In $p$ dimensions, start from several independent standard normal variables and apply any linear map and any shift. The map can mix the variables, which is how correlation arises.
A standard normal vector $\mathbf{Z} = (Z_1, \ldots, Z_k)^{\mathsf T}$ has independent coordinates, each normal with mean 0 and variance 1. Its mean vector is $\mathbf{0}$ and its covariance matrix is the identity $I_k$.
Definition. A random vector $\mathbf{X}$ with $p$ coordinates is multivariate normal if $\mathbf{X} = \boldsymbol\mu + A\mathbf{Z}$ for some vector $\boldsymbol\mu$, some $p \times k$ matrix $A$ and some standard normal vector $\mathbf{Z}$ with $k$ coordinates.
Its mean vector is $\boldsymbol\mu$, and its covariance matrix is $\Sigma = AI_kA^{\mathsf T} = AA^{\mathsf T}$, by the rules above. Many matrices $A$ give the same $\Sigma$. Does the choice matter?
Theorem 1 (the mean vector and covariance matrix determine the distribution). If $\mathbf{X} = \boldsymbol\mu + A\mathbf{Z}$ as in the definition and $\Sigma = AA^{\mathsf T}$, then for every $\mathbf{t}$ in $\mathbb{R}^p$
$$\varphi_{\mathbf{X}}(\mathbf{t}) = \exp\big(i\,\mathbf{t}^{\mathsf T}\boldsymbol\mu - \tfrac12\,\mathbf{t}^{\mathsf T}\Sigma\,\mathbf{t}\big), \qquad M_{\mathbf{X}}(\mathbf{t}) = E\big[e^{\mathbf{t}^{\mathsf T}\mathbf{X}}\big] = \exp\big(\mathbf{t}^{\mathsf T}\boldsymbol\mu + \tfrac12\,\mathbf{t}^{\mathsf T}\Sigma\,\mathbf{t}\big).$$
Consequently the distribution of $\mathbf{X}$ depends on $A$ only through $AA^{\mathsf T}$. For every vector $\boldsymbol\mu$ and every symmetric positive semidefinite matrix $\Sigma$ there is a multivariate normal vector with mean $\boldsymbol\mu$ and covariance matrix $\Sigma$. Its distribution is written $N_p(\boldsymbol\mu, \Sigma)$.
Proof. Let $\mathbf{c} = A^{\mathsf T}\mathbf{t}$, a vector with $k$ entries. Then $\mathbf{t}^{\mathsf T}\mathbf{X} = \mathbf{t}^{\mathsf T}\boldsymbol\mu + \sum_j c_jZ_j$. The standard normal characteristic function is $e^{-s^2/2}$, so $Z_j$ contributes $E[e^{ic_jZ_j}] = e^{-c_j^2/2}$, and by independence, as in the first part of Lemma 1, the expected value of the product $\prod_j e^{ic_jZ_j}$ is the product of the expected values. Hence
$$\varphi_{\mathbf{X}}(\mathbf{t}) = e^{i\mathbf{t}^{\mathsf T}\boldsymbol\mu}\prod_{j=1}^k e^{-c_j^2/2} = \exp\big(i\,\mathbf{t}^{\mathsf T}\boldsymbol\mu - \tfrac12\lVert\mathbf{c}\rVert^2\big),$$
and $\lVert\mathbf{c}\rVert^2 = \mathbf{t}^{\mathsf T}AA^{\mathsf T}\mathbf{t} = \mathbf{t}^{\mathsf T}\Sigma\mathbf{t}$. The moment generating function is the same computation with the standard normal moment generating function $e^{s^2/2}$, which is finite for every $s$, and with the product rule applied to the positive variables $e^{c_jZ_j}$. The characteristic function depends on $A$ only through $\Sigma$, so by uniqueness so does the distribution. For existence, a symmetric positive semidefinite $\Sigma$ has a symmetric square root $\Sigma^{1/2}$ with $\Sigma^{1/2}\Sigma^{1/2} = \Sigma$ (Covariance Matrices and Quadratic Forms, Theorem 3), and $\boldsymbol\mu + \Sigma^{1/2}\mathbf{Z}$ with $k = p$ has mean $\boldsymbol\mu$ and covariance matrix $\Sigma^{1/2}(\Sigma^{1/2})^{\mathsf T} = \Sigma$. $\square$
A note on notation. For $p = 1$, $N_1(\mu, \sigma^2)$ is the normal distribution with mean $\mu$ and variance $\sigma^2$. Elsewhere on the site a one-dimensional normal is written $N(\mu, \sigma)$ with the standard deviation second (The Normal Distribution). In this unit the subscript marks the multivariate convention, with the covariance matrix second, and one-dimensional normals are described in words.
The definition allows $\Sigma$ to be singular, for instance $A$ with a column of zeros or with fewer columns than rows. Such distributions put all their probability on a lower-dimensional flat set; the last section treats them. A single constant is the case $\Sigma = 0$: a normal distribution with variance 0. Allowing it keeps the family closed under every linear map.
The next theorem is the most useful test for membership of the family, and it is the reason the family behaves so well.
Theorem 2 (normal linear combinations). A random vector $\mathbf{X}$ is multivariate normal if and only if every linear combination $\mathbf{a}^{\mathsf T}\mathbf{X}$ has a normal distribution (variance 0 allowed). Then $\mathbf{a}^{\mathsf T}\mathbf{X}$ has mean $\mathbf{a}^{\mathsf T}\boldsymbol\mu$ and variance $\mathbf{a}^{\mathsf T}\Sigma\mathbf{a}$.
Durrett (2019, Exercise 3.10.8) poses the equivalence as an exercise; here is the solution.
Proof. If $\mathbf{X} = \boldsymbol\mu + A\mathbf{Z}$, then by Theorem 1 with $\mathbf{t} = s\mathbf{a}$ the variable $\mathbf{a}^{\mathsf T}\mathbf{X}$ has characteristic function $\exp(is\,\mathbf{a}^{\mathsf T}\boldsymbol\mu - \tfrac12 s^2\,\mathbf{a}^{\mathsf T}\Sigma\mathbf{a})$, which is that of a normal variable with mean $\mathbf{a}^{\mathsf T}\boldsymbol\mu$ and variance $\mathbf{a}^{\mathsf T}\Sigma\mathbf{a}$. By the uniqueness theorem for one variable, that is its distribution.
Conversely, suppose every linear combination is normal. Each coordinate $X_j$ is then normal, so it has a finite variance, and $\mathbf{X}$ has a mean vector $\boldsymbol\mu$ and a covariance matrix $\Sigma$, positive semidefinite. For each $\mathbf{t}$, the normal variable $\mathbf{t}^{\mathsf T}\mathbf{X}$ has mean $\mathbf{t}^{\mathsf T}\boldsymbol\mu$ and variance $\mathbf{t}^{\mathsf T}\Sigma\mathbf{t}$, so
$$\varphi_{\mathbf{X}}(\mathbf{t}) = \varphi_{\mathbf{t}^{\mathsf T}\mathbf{X}}(1) = \exp\big(i\,\mathbf{t}^{\mathsf T}\boldsymbol\mu - \tfrac12\,\mathbf{t}^{\mathsf T}\Sigma\mathbf{t}\big).$$
This is the characteristic function of $N_p(\boldsymbol\mu, \Sigma)$, which exists by Theorem 1, and uniqueness finishes the proof. $\square$
The theorem gives a way to look at data for joint normality: every direction must look normal. For the blood pressures, the systolic pressure alone has skewness 1.02 and the diastolic 0.18. Over twelve directions spaced 15 degrees apart in standardized coordinates, the most skewed projection has skewness $-1.25$, at 150 degrees; reversed in sign, that direction is close to the pulse pressure, systolic minus diastolic, whose own skewness is 1.25. A normal variable has skewness 0. In 30 samples of 5,232 drawn from the normal distribution with the adults' mean vector and covariance matrix, the largest absolute skewness over the same twelve directions had a median of 0.050 (interquartile range 0.044 to 0.068). The adults' pressures are far from jointly normal in this respect.
Example (normal coordinates are not enough). "Each coordinate is normal, so the vector is multivariate normal." Theorem 2 asks for every combination, not only the coordinates, and the difference is real. The section on independence below gives a pair of standard normal variables whose sum is 0 with probability one half, so the sum is not normal and the pair is not multivariate normal.
Affine maps, marginals and sums
The definition is closed under the operations of statistics: transforming, selecting and adding.
Theorem 3 (affine maps). If $\mathbf{X} \sim N_p(\boldsymbol\mu, \Sigma)$, $B$ is a $q \times p$ matrix and $\mathbf{c}$ a vector with $q$ entries, then $B\mathbf{X} + \mathbf{c} \sim N_q(B\boldsymbol\mu + \mathbf{c},\ B\Sigma B^{\mathsf T})$. In particular:
- Marginals. Any subvector of $\mathbf{X}$ is multivariate normal, with the corresponding entries of $\boldsymbol\mu$ and the corresponding rows and columns of $\Sigma$.
- Sums. If $\mathbf{X}_1 \sim N_p(\boldsymbol\mu_1, \Sigma_1)$ and $\mathbf{X}_2 \sim N_p(\boldsymbol\mu_2, \Sigma_2)$ are independent, then $\mathbf{X}_1 + \mathbf{X}_2 \sim N_p(\boldsymbol\mu_1 + \boldsymbol\mu_2, \Sigma_1 + \Sigma_2)$, and the stacked vector $(\mathbf{X}_1^{\mathsf T}, \mathbf{X}_2^{\mathsf T})^{\mathsf T}$ is multivariate normal with the two covariance matrices as diagonal blocks and zeros elsewhere.
Proof. By Theorem 1, $\mathbf{X}$ has the distribution of $\boldsymbol\mu + A\mathbf{Z}$ with $AA^{\mathsf T} = \Sigma$, so $B\mathbf{X} + \mathbf{c}$ has the distribution of $(B\boldsymbol\mu + \mathbf{c}) + (BA)\mathbf{Z}$, which is multivariate normal with covariance $BAA^{\mathsf T}B^{\mathsf T} = B\Sigma B^{\mathsf T}$. A subvector is $B\mathbf{X}$ for the matrix $B$ whose rows are the corresponding rows of the identity. For the stacked vector, Lemma 1 and Theorem 1 give
$$\varphi(\mathbf{t}_1, \mathbf{t}_2) = \varphi_{\mathbf{X}_1}(\mathbf{t}_1)\varphi_{\mathbf{X}_2}(\mathbf{t}_2) = \exp\big(i\,\mathbf{t}_1^{\mathsf T}\boldsymbol\mu_1 + i\,\mathbf{t}_2^{\mathsf T}\boldsymbol\mu_2 - \tfrac12\mathbf{t}_1^{\mathsf T}\Sigma_1\mathbf{t}_1 - \tfrac12\mathbf{t}_2^{\mathsf T}\Sigma_2\mathbf{t}_2\big),$$
the characteristic function of the normal distribution with the block-diagonal covariance matrix, and uniqueness applies. The sum is the stacked vector times $(I_p\ \ I_p)$. $\square$
The stacking rule is what makes a sample usable: if $\mathbf{x}_1, \ldots, \mathbf{x}_n$ are independent $N_p(\boldsymbol\mu, \Sigma)$ vectors, the $np$ numbers together form one multivariate normal vector, so every linear function of the whole sample is multivariate normal. Covariance Matrices and Quadratic Forms uses this for the sample mean and the sample covariance matrix.
Example (two summaries of blood pressure). The pulse pressure, systolic minus diastolic, has coefficients $\mathbf{a} = (1, -1)$; the average of the two pressures has $\mathbf{a} = (\tfrac12, \tfrac12)$. Their means are $\mathbf{a}^{\mathsf T}\boldsymbol\mu = 53.82$ and $99.40$, and their variances $\mathbf{a}^{\mathsf T}\Sigma\mathbf{a}$ are $402.65 + 144.14 - 2(109.05) = 328.68$ and $\tfrac14(402.65 + 144.14 + 2 \times 109.05) = 191.22$ (from unrounded entries). If the pair were jointly normal, each summary would be exactly normal with these moments. The skewness of 1.25 of the pulse pressure, found above, says that it is not.
The density
When $\Sigma$ is nonsingular the distribution has a density, and its form explains the elliptical contours of normal scatter plots.
Theorem 4 (the density). If $\Sigma$ is nonsingular, $N_p(\boldsymbol\mu, \Sigma)$ has the density
$$f(\mathbf{x}) = \frac{1}{(2\pi)^{p/2}(\det\Sigma)^{1/2}}\exp\Big(-\tfrac12(\mathbf{x} - \boldsymbol\mu)^{\mathsf T}\Sigma^{-1}(\mathbf{x} - \boldsymbol\mu)\Big), \qquad \mathbf{x} \in \mathbb{R}^p.$$
Proof. By Theorem 1 the distribution is that of $\boldsymbol\mu + \Sigma^{1/2}\mathbf{Z}$ with $\mathbf{Z}$ standard normal with $p$ coordinates. The independent coordinates of $\mathbf{Z}$ have the joint density $(2\pi)^{-p/2}e^{-\lVert\mathbf{z}\rVert^2/2}$, the product of their densities. Because $\Sigma$ is nonsingular, so is $\Sigma^{1/2}$, with inverse $\Sigma^{-1/2}$ satisfying $\Sigma^{-1/2}\Sigma^{-1/2} = \Sigma^{-1}$, and $\det\Sigma^{1/2} = (\det\Sigma)^{1/2} > 0$ (Theorem 3 of the companion article). The linear change of variables cited on The Chi-Squared and t Distributions says that $\Sigma^{1/2}\mathbf{Z}$ has density $(2\pi)^{-p/2}e^{-\lVert\Sigma^{-1/2}\mathbf{w}\rVert^2/2}/\det\Sigma^{1/2}$ at $\mathbf{w}$. Adding the constant vector $\boldsymbol\mu$ moves the density to $\mathbf{w} = \mathbf{x} - \boldsymbol\mu$. Finally $\lVert\Sigma^{-1/2}\mathbf{w}\rVert^2 = \mathbf{w}^{\mathsf T}\Sigma^{-1/2}\Sigma^{-1/2}\mathbf{w} = \mathbf{w}^{\mathsf T}\Sigma^{-1}\mathbf{w}$. $\square$
Durrett (2019, Section 3.10) gives this density for mean 0, and Bishop (2006, eq. 2.43) in this form. The quantity in the exponent, $\Delta^2(\mathbf{x}) = (\mathbf{x} - \boldsymbol\mu)^{\mathsf T}\Sigma^{-1}(\mathbf{x} - \boldsymbol\mu)$, is the squared Mahalanobis distance of $\mathbf{x}$ from the mean. The density is constant where $\Delta^2$ is constant, on ellipsoids centered at $\boldsymbol\mu$, and Covariance Matrices and Quadratic Forms proves (its Theorem 6) that $\Delta^2(\mathbf{X})$ has the chi-squared distribution with $p$ degrees of freedom. The ellipsoid $\Delta^2 \le c$ therefore holds probability $P(\chi^2_p \le c)$.
Two variables. Write $\Sigma$ with standard deviations $\sigma_1, \sigma_2$ and correlation $\rho$, $|\rho| < 1$. Then $\det\Sigma = \sigma_1^2\sigma_2^2(1 - \rho^2)$ and
$$\Sigma^{-1} = \frac{1}{\sigma_1^2\sigma_2^2(1 - \rho^2)}\begin{pmatrix} \sigma_2^2 & -\rho\sigma_1\sigma_2 \\ -\rho\sigma_1\sigma_2 & \sigma_1^2 \end{pmatrix},$$
as multiplying out confirms. With standardized coordinates $z_j = (x_j - \mu_j)/\sigma_j$, the theorem becomes the familiar bivariate density
$$f(x_1, x_2) = \frac{1}{2\pi\sigma_1\sigma_2\sqrt{1 - \rho^2}}\exp\Big(-\frac{z_1^2 - 2\rho z_1z_2 + z_2^2}{2(1 - \rho^2)}\Big).$$
For two variables the chi-squared distribution with 2 degrees of freedom has the tail $P(\chi^2_2 > c) = e^{-c/2}$, since its density is $\tfrac12e^{-x/2}$ (The Chi-Squared and t Distributions). So the ellipse holding probability $q$ is $\Delta^2 \le -2\ln(1 - q)$: 1.386 for one half, 4.605 for 90%, 9.210 for 99%.
The adults fill these ellipses unevenly: 56.0% of them lie inside the 50% ellipse, 90.5% inside the 90% ellipse and 97.5% inside the 99% ellipse. In the 30 normal samples the share inside the 99% ellipse had a median of 99.0% (interquartile range 98.9% to 99.1%). Too many adults near the center and too many far out is the signature of tails heavier than the normal's; Covariance Matrices and Quadratic Forms makes the comparison formal with the fourth moment of the Mahalanobis distance.
Zero covariance and independence
For general random variables, zero covariance does not imply independence (Expectation, Variance and Covariance gives $X$ and $X^2$). Inside the multivariate normal family it does.
Theorem 5 (uncorrelated blocks of a normal vector are independent). Let $\mathbf{X} = (\mathbf{X}_1^{\mathsf T}, \mathbf{X}_2^{\mathsf T})^{\mathsf T}$ be multivariate normal, with covariance matrix partitioned as
$$\Sigma = \begin{pmatrix} \Sigma_{11} & \Sigma_{12} \\ \Sigma_{21} & \Sigma_{22}\end{pmatrix}, \qquad \Sigma_{21} = \Sigma_{12}^{\mathsf T}.$$
Then $\mathbf{X}_1$ and $\mathbf{X}_2$ are independent if and only if $\Sigma_{12} = 0$. In particular the coordinates of a multivariate normal vector are mutually independent if and only if $\Sigma$ is diagonal.
Proof. Independent coordinates have covariance 0 (Expectation, Variance and Covariance, Theorem 6, part 5), so independence gives $\Sigma_{12} = 0$. Conversely, if $\Sigma_{12} = 0$, the quadratic form in Theorem 1 splits: $\mathbf{t}^{\mathsf T}\Sigma\mathbf{t} = \mathbf{t}_1^{\mathsf T}\Sigma_{11}\mathbf{t}_1 + 2\,\mathbf{t}_1^{\mathsf T}\Sigma_{12}\mathbf{t}_2 + \mathbf{t}_2^{\mathsf T}\Sigma_{22}\mathbf{t}_2$ with the middle term 0. So $\varphi_{\mathbf{X}}(\mathbf{t})$ is the product of $\exp(i\,\mathbf{t}_1^{\mathsf T}\boldsymbol\mu_1 - \tfrac12\mathbf{t}_1^{\mathsf T}\Sigma_{11}\mathbf{t}_1)$ and the same expression for block 2. By Theorem 3 these are the characteristic functions of $\mathbf{X}_1$ and $\mathbf{X}_2$, and Lemma 1 gives independence. For the coordinates, apply the same argument to $p$ blocks of size 1. $\square$
Durrett (2019, Exercise 3.10.7) states the coordinate version. The theorem needs the whole vector to be multivariate normal. Normal coordinates with zero correlation are not enough, and the next example shows how badly it can fail.
Example (uncorrelated standard normal variables that are dependent). Let $X$ be standard normal and let $W$ be independent of it, equal to $+1$ or $-1$ with probability one half each. Put $Y = WX$.
- $Y$ is standard normal: $P(Y \le y) = \tfrac12P(X \le y) + \tfrac12P(-X \le y) = P(X \le y)$, since $-X$ has the distribution of $X$.
- $X$ and $Y$ are uncorrelated: $E[XY] = E[W]\,E[X^2] = 0 \times 1 = 0$, by independence of $W$ and $X$.
- They are dependent: $|Y| = |X|$ always, so $P(|X| \le 1,\ |Y| > 1) = 0$, while $P(|X| \le 1)\,P(|Y| > 1) = 0.683 \times 0.317 > 0$.
- The pair is not multivariate normal: $X + Y = (1 + W)X$ equals 0 whenever $W = -1$, so $P(X + Y = 0) \ge \tfrac12$. A normal variable with variance $\operatorname{Var}(X + Y) = 2$ gives probability 0 to every single point, so $X + Y$ is not normal, and Theorem 2 rules the pair out.
So two variables can each be exactly normal and uncorrelated, and yet one determines the other up to sign. The fourth worked problem gives a second pair of this kind, in which $Y$ is a function of $X$.
Conditional distributions
Back to the adult with a systolic pressure of 150. Here is the plan. Subtract from $\mathbf{X}_1$ the multiple of $\mathbf{X}_2$ that removes all covariance with $\mathbf{X}_2$. The remainder is uncorrelated with $\mathbf{X}_2$, hence independent of it by Theorem 5. Knowing $\mathbf{X}_2$ therefore tells us the subtracted part exactly and nothing about the remainder.
Theorem 6 (conditional distributions). Let $\mathbf{X} \sim N_p(\boldsymbol\mu, \Sigma)$ be partitioned into $\mathbf{X}_1$ with $q$ coordinates and $\mathbf{X}_2$ with $p - q$, with $\boldsymbol\mu$ and $\Sigma$ partitioned to match, and suppose $\Sigma_{22}$ is nonsingular. Let
$$B = \Sigma_{12}\Sigma_{22}^{-1}, \qquad \mathbf{W} = \mathbf{X}_1 - B\mathbf{X}_2, \qquad \Sigma_{11\cdot 2} = \Sigma_{11} - \Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}.$$
- $\mathbf{W} \sim N_q(\boldsymbol\mu_1 - B\boldsymbol\mu_2,\ \Sigma_{11\cdot 2})$, and $\mathbf{W}$ is independent of $\mathbf{X}_2$.
- Hence $\mathbf{X}_1 = \mathbf{W} + B\mathbf{X}_2$ is, given $\mathbf{X}_2 = \mathbf{x}_2$, distributed as $N_q\big(\boldsymbol\mu_1 + B(\mathbf{x}_2 - \boldsymbol\mu_2),\ \Sigma_{11\cdot 2}\big)$. When $\Sigma$ is nonsingular this is a statement about densities: $f(\mathbf{x}_1, \mathbf{x}_2)/f_2(\mathbf{x}_2)$, the conditional density of $\mathbf{X}_1$ given $\mathbf{X}_2 = \mathbf{x}_2$, is the density of that normal distribution.
- The conditional mean $E[\mathbf{X}_1 \mid \mathbf{X}_2 = \mathbf{x}_2] = \boldsymbol\mu_1 + B(\mathbf{x}_2 - \boldsymbol\mu_2)$ is linear in $\mathbf{x}_2$, and the conditional covariance matrix $\Sigma_{11\cdot 2}$ does not depend on $\mathbf{x}_2$. It is never larger than $\Sigma_{11}$: $\Sigma_{11} - \Sigma_{11\cdot 2} = B\Sigma_{22}B^{\mathsf T}$ is positive semidefinite.
Proof. (1) The pair $(\mathbf{W}, \mathbf{X}_2)$ is an affine image of $\mathbf{X}$, so it is multivariate normal (Theorem 3). Its cross-covariance is
$$\operatorname{Cov}(\mathbf{W}, \mathbf{X}_2) = \operatorname{Cov}(\mathbf{X}_1, \mathbf{X}_2) - B\operatorname{Cov}(\mathbf{X}_2, \mathbf{X}_2) = \Sigma_{12} - \Sigma_{12}\Sigma_{22}^{-1}\Sigma_{22} = 0,$$
so $\mathbf{W}$ and $\mathbf{X}_2$ are independent by Theorem 5. The mean of $\mathbf{W}$ is $\boldsymbol\mu_1 - B\boldsymbol\mu_2$, and its covariance matrix is
$$\Sigma_{11} - B\Sigma_{21} - \Sigma_{12}B^{\mathsf T} + B\Sigma_{22}B^{\mathsf T} = \Sigma_{11} - \Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21} - \Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21} + \Sigma_{12}\Sigma_{22}^{-1}\Sigma_{22}\Sigma_{22}^{-1}\Sigma_{21} = \Sigma_{11\cdot 2},$$
using $B^{\mathsf T} = \Sigma_{22}^{-1}\Sigma_{21}$ (the inverse of a symmetric matrix is symmetric).
(2) Because $\mathbf{W}$ is independent of $\mathbf{X}_2$, fixing $\mathbf{X}_2 = \mathbf{x}_2$ leaves the distribution of $\mathbf{W}$ unchanged and makes $B\mathbf{X}_2$ the constant $B\mathbf{x}_2$. For the density statement, suppose $\Sigma$ is nonsingular. Then $\Sigma_{11\cdot 2}$ is nonsingular (Theorem 7 below), so $\mathbf{W}$ has a normal density $f_{\mathbf{W}}$ by Theorem 4, and so has $\mathbf{X}_2$. Independent vectors with densities have the product of the densities as their joint density (the product rule cited on The Chi-Squared and t Distributions). The map $(\mathbf{w}, \mathbf{x}_2) \mapsto (\mathbf{w} + B\mathbf{x}_2, \mathbf{x}_2)$ is linear with the matrix $\begin{pmatrix} I & B \\ 0 & I\end{pmatrix}$, which is upper triangular with ones on its diagonal and so has determinant 1 (Axler, 2024, 9.48). By the linear change of variables, $\mathbf{X} = (\mathbf{W} + B\mathbf{X}_2, \mathbf{X}_2)$ has the density
$$f(\mathbf{x}_1, \mathbf{x}_2) = f_{\mathbf{W}}(\mathbf{x}_1 - B\mathbf{x}_2)\,f_2(\mathbf{x}_2).$$
Dividing by $f_2(\mathbf{x}_2) > 0$ gives $f_{\mathbf{W}}(\mathbf{x}_1 - B\mathbf{x}_2)$, which is the density of $N_q(\boldsymbol\mu_1 - B\boldsymbol\mu_2 + B\mathbf{x}_2,\ \Sigma_{11\cdot 2})$ in $\mathbf{x}_1$.
(3) The conditional mean is the mean of the distribution in (2). Finally $\Sigma_{11} - \Sigma_{11\cdot 2} = \Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21} = B\Sigma_{22}B^{\mathsf T}$, which is the covariance matrix of $B\mathbf{X}_2$ and therefore positive semidefinite. $\square$
These are equations (2.81) and (2.82) of Bishop (2006), which obtains them from the quadratic form in the exponent of the density. When $\Sigma$ is singular, statement 2 is to be read through the representation $\mathbf{X}_1 = \mathbf{W} + B\mathbf{X}_2$, which describes the vector for every value of $\mathbf{X}_2$ at once. Conditional Expectation explains, in its section on conditioning on an event of probability zero, why such statements are about the variable $\mathbf{X}_2$ and not about a single event.
Three readings of the theorem are worth stating.
- The regression is linear. By Conditional Expectation (Theorem 3, part 4), the conditional mean is the best predictor of each coordinate of $\mathbf{X}_1$ among all functions of $\mathbf{X}_2$. For a normal vector it is linear, so it coincides with the best linear predictor of Expectation, Variance and Covariance (Theorem 8): no curve predicts better than the straight line.
- The spread is the same everywhere. The conditional covariance $\Sigma_{11\cdot 2}$ is the same for every $\mathbf{x}_2$. A normal model cannot describe data whose scatter around the regression line widens.
- The law of total variance in matrix form. $\Sigma_{11} = \Sigma_{11\cdot 2} + B\Sigma_{22}B^{\mathsf T}$: the expected conditional covariance plus the covariance of the conditional mean.
Two variables. With one variable on each side, $B = \sigma_{12}/\sigma_2^2 = \rho\sigma_1/\sigma_2$ and $\Sigma_{11\cdot 2} = \sigma_1^2 - \sigma_{12}^2/\sigma_2^2 = \sigma_1^2(1 - \rho^2)$. In standardized units the predicted $z$ is $\rho$ times the given $z$. Since $|\rho| < 1$ the prediction is closer to the mean than the condition was. This is regression toward the mean (Simple Linear Regression works it on data, and Introduction to Regression tells how Galton named it): it is a property of the conditional mean, not a tendency of individuals to become average.
The blood pressures. For diastolic given systolic pressure, the slope is $109.05/402.65 = 0.2708$ mm Hg per mm Hg and the intercept $72.49 - 0.2708 \times 126.31 = 38.28$, so the line is $38.28 + 0.2708\,x$. The conditional variance is $144.14 \times (1 - 0.453^2) = 114.60$, a conditional standard deviation of 10.71 mm Hg (from unrounded values). Systolic pressure accounts for $\rho^2 = 20.5\%$ of the variance of diastolic pressure. At a systolic pressure of 150 the normal model predicts a diastolic pressure normal with mean 78.90 and standard deviation 10.71, so a probability of 15.0% that it is 90 or more.
The NHANES data contain 374 adults with systolic pressure from 145 to just under 155. Their mean systolic is 149.77 and the line there gives 78.84. Their mean diastolic is 77.44, close to the line. Their standard deviation is 14.20, not 10.71, and 20.9% of them have a diastolic pressure of 90 or more, not 15.0%. The model has the center right and the spread wrong.
The same pattern holds across the range. The table groups the adults by systolic pressure in bands of 10 mm Hg. It compares the mean diastolic pressure with the line at the band's mean systolic pressure, and the standard deviation of the diastolic pressure with what the normal model gives in 30 samples of 5,232 from the fitted normal distribution (median and interquartile range of the band's standard deviation over the 30 samples).
| Systolic (mm Hg) | Adults | Mean diastolic | Line | SD of diastolic | SD in normal samples |
|---|---|---|---|---|---|
| 90 to 99 | 226 | 60.5 | 64.3 | 7.3 | 10.8 (10.6 to 11.0) |
| 100 to 109 | 809 | 65.8 | 66.8 | 7.9 | 10.7 (10.5 to 11.0) |
| 110 to 119 | 1,153 | 69.4 | 69.4 | 8.7 | 10.7 (10.5 to 10.9) |
| 120 to 129 | 1,137 | 73.6 | 72.0 | 9.6 | 10.7 (10.6 to 10.9) |
| 130 to 139 | 785 | 76.3 | 74.7 | 11.8 | 10.7 (10.5 to 10.8) |
| 140 to 149 | 444 | 77.2 | 77.3 | 12.6 | 10.7 (10.5 to 10.9) |
| 150 to 159 | 313 | 78.9 | 80.0 | 14.6 | 10.5 (10.3 to 10.9) |
| 160 to 169 | 155 | 81.0 | 82.8 | 15.6 | 10.7 (10.2 to 11.2) |
| 170 to 179 | 98 | 81.8 | 85.4 | 15.3 | 10.9 (10.3 to 11.4) |
The means stay within 4 mm Hg of the line, but they bend below it at both ends by more than sampling variation explains (through the 160 band; the top band, with 98 adults, varies as much in the normal samples). In six of the nine bands the adults' mean lies outside the range of all 30 normal samples: in the lowest band the adults' mean is 3.8 mm Hg below the line, while every normal sample was within 1.1 mm Hg of it. The standard deviations double, from 7.3 to about 15, where the normal model holds them near 10.7 in every band. As one number: the standard deviation of the deviations from the line among adults with systolic pressure of 150 or more, divided by the same among adults below 110, is 1.97 for the data, against a median of 0.98 (interquartile range 0.97 to 1.02) in the normal samples.
Example (a correct line is not a correct model). "The conditional means follow the line, so the normal model fits." Linear conditional means are what Theorem 6 predicts, but many pairs that are not normal have them too: if $Y = a + bX + \varepsilon$ with $\varepsilon$ independent of $X$ and of mean 0, then $E[Y \mid X] = a + bX$ whatever the distributions of $X$ and $\varepsilon$. A linear conditional mean is necessary for joint normality, far from sufficient, so a good line says little about normality. What normality adds is the shape and the constant spread around the line, and those are what the data contradict.
Example (a population mixed over age). Part of the widening comes from age. The correlation of the two pressures is 0.621 among the 1,742 adults aged 18 to 39, 0.637 among the 1,578 aged 40 to 59, and 0.377 among the 1,912 aged 60 and over. If (systolic, diastolic, age) were jointly normal, Theorem 6 with age as $\mathbf{X}_2$ would give one conditional correlation at every age, the partial correlation defined below, here 0.498. A band of ages is not a single age, so compare like with like. Under the fitted normal model of the triple, the correlation within each of the three bands would be 0.493 and the diastolic standard deviation 12.0 (computed exactly: within a band the pair has covariance $\Sigma_{11\cdot 2} + \mathbf{b}\mathbf{b}^{\mathsf T}v$, where $\mathbf{b}$ holds the two slopes on age and $v$ is the variance of the fitted normal age distribution cut to the band); the adults show 0.621, 0.637 and 0.377. The older adults, whose systolic pressures are highest, have the weakest link between the two pressures and the widest spread of diastolic pressure: a mean systolic pressure of 136.76 and a diastolic standard deviation of 12.26, against 114.84 and 11.08 among the youngest group. Age itself is not normal, so the triple cannot be jointly normal, and the figures show that the departure matters.
The Schur complement and the inverse covariance matrix
The matrix $\Sigma_{11\cdot 2}$ has a name and a second life inside $\Sigma^{-1}$.
Theorem 7 (block factorization). Let $\Sigma$ be a symmetric matrix partitioned as above, with $\Sigma_{22}$ nonsingular, and $B = \Sigma_{12}\Sigma_{22}^{-1}$. Then
$$\Sigma = \begin{pmatrix} I & B \\ 0 & I\end{pmatrix}\begin{pmatrix} \Sigma_{11\cdot 2} & 0 \\ 0 & \Sigma_{22}\end{pmatrix}\begin{pmatrix} I & 0 \\ B^{\mathsf T} & I\end{pmatrix}.$$
Consequently:
- $\det\Sigma = \det\Sigma_{11\cdot 2}\cdot\det\Sigma_{22}$, so $\Sigma$ is nonsingular exactly when $\Sigma_{11\cdot 2}$ is.
- When $\Sigma$ is nonsingular, the upper-left block of $\Sigma^{-1}$ is $\Sigma_{11\cdot 2}^{-1}$, and its upper-right block is $-\Sigma_{11\cdot 2}^{-1}B$.
- For every $\mathbf{x}$, with $\mathbf{m}(\mathbf{x}_2) = \boldsymbol\mu_1 + B(\mathbf{x}_2 - \boldsymbol\mu_2)$,
$$(\mathbf{x} - \boldsymbol\mu)^{\mathsf T}\Sigma^{-1}(\mathbf{x} - \boldsymbol\mu) = (\mathbf{x}_2 - \boldsymbol\mu_2)^{\mathsf T}\Sigma_{22}^{-1}(\mathbf{x}_2 - \boldsymbol\mu_2) + (\mathbf{x}_1 - \mathbf{m}(\mathbf{x}_2))^{\mathsf T}\Sigma_{11\cdot 2}^{-1}(\mathbf{x}_1 - \mathbf{m}(\mathbf{x}_2)).$$
$\Sigma_{11\cdot 2}$ is called the Schur complement of $\Sigma_{22}$ in $\Sigma$, the name Bishop (2006, eqs. 2.76 and 2.77) uses for the same matrix in the formula for a partitioned inverse.
Proof. Multiply the last two factors: $\begin{pmatrix} \Sigma_{11\cdot 2} & 0 \\ \Sigma_{22}B^{\mathsf T} & \Sigma_{22}\end{pmatrix}$. Multiplying by the first factor gives the blocks $\Sigma_{11\cdot 2} + B\Sigma_{22}B^{\mathsf T} = \Sigma_{11}$, $B\Sigma_{22} = \Sigma_{12}$, $\Sigma_{22}B^{\mathsf T} = \Sigma_{21}$ and $\Sigma_{22}$, which is $\Sigma$.
(1) The first factor is upper triangular with ones on the diagonal and has determinant 1 (Axler, 2024, 9.48); the third is its transpose, which has the same determinant (9.56(a)); the determinant is multiplicative (9.49). The middle factor is block diagonal. Writing each block through its eigenvectors (the companion article, Theorem 2), $\Sigma_{11\cdot 2} = Q_1\Lambda_1Q_1^{\mathsf T}$ and $\Sigma_{22} = Q_2\Lambda_2Q_2^{\mathsf T}$, the middle factor is $Q\Lambda Q^{\mathsf T}$ with $Q$ the orthogonal block-diagonal matrix of $Q_1, Q_2$ and $\Lambda$ the diagonal matrix of both sets of eigenvalues. By 9.49 twice, $\det(Q\Lambda Q^{\mathsf T}) = \det Q\,\det\Lambda\,\det Q^{\mathsf T} = \det\Lambda\,\det(QQ^{\mathsf T}) = \det\Lambda$, the product of all the eigenvalues, which is $\det\Sigma_{11\cdot 2}\det\Sigma_{22}$. A matrix is nonsingular exactly when its determinant is not 0, and $\det\Sigma_{22} \ne 0$.
(2) The inverse of $\begin{pmatrix} I & B \\ 0 & I\end{pmatrix}$ is $\begin{pmatrix} I & -B \\ 0 & I\end{pmatrix}$, as multiplying out shows, and the inverse of the transpose is the transpose of the inverse. Inverting the factorization in reverse order,
$$\Sigma^{-1} = \begin{pmatrix} I & 0 \\ -B^{\mathsf T} & I\end{pmatrix}\begin{pmatrix} \Sigma_{11\cdot 2}^{-1} & 0 \\ 0 & \Sigma_{22}^{-1}\end{pmatrix}\begin{pmatrix} I & -B \\ 0 & I\end{pmatrix} = \begin{pmatrix} \Sigma_{11\cdot 2}^{-1} & -\Sigma_{11\cdot 2}^{-1}B \\ -B^{\mathsf T}\Sigma_{11\cdot 2}^{-1} & \Sigma_{22}^{-1} + B^{\mathsf T}\Sigma_{11\cdot 2}^{-1}B\end{pmatrix}.$$
(3) Let $\mathbf{y} = \mathbf{x} - \boldsymbol\mu$. By the middle expression in (2), $\mathbf{y}^{\mathsf T}\Sigma^{-1}\mathbf{y} = \mathbf{u}^{\mathsf T}\operatorname{diag}(\Sigma_{11\cdot 2}^{-1}, \Sigma_{22}^{-1})\mathbf{u}$ with $\mathbf{u} = \begin{pmatrix} I & -B \\ 0 & I\end{pmatrix}\mathbf{y} = (\mathbf{y}_1 - B\mathbf{y}_2,\ \mathbf{y}_2)$, and $\mathbf{y}_1 - B\mathbf{y}_2 = \mathbf{x}_1 - \mathbf{m}(\mathbf{x}_2)$. $\square$
Part 3 is Theorem 6 in another form. Put it in the exponent of the density of Theorem 4 and use part 1 for the constant: the joint density is the density of $\mathbf{X}_2$ times a normal density in $\mathbf{x}_1$ centered at $\mathbf{m}(\mathbf{x}_2)$ with covariance $\Sigma_{11\cdot 2}$. It also splits the Mahalanobis distance into the distance of $\mathbf{x}_2$ from its mean and the distance of $\mathbf{x}_1$ from its prediction.
Partial correlation and the precision matrix. Let $K = \Sigma^{-1}$, the precision matrix, and take $\mathbf{X}_1 = (X_j, X_k)$ for two coordinates, with all the others in $\mathbf{X}_2$. By part 2, the $2 \times 2$ conditional covariance matrix is the inverse of the corresponding block of $K$:
$$\Sigma_{11\cdot 2} = \begin{pmatrix} K_{jj} & K_{jk} \\ K_{jk} & K_{kk}\end{pmatrix}^{-1} = \frac{1}{K_{jj}K_{kk} - K_{jk}^2}\begin{pmatrix} K_{kk} & -K_{jk} \\ -K_{jk} & K_{jj}\end{pmatrix}.$$
The correlation computed from this matrix, the partial correlation of $X_j$ and $X_k$ given the rest, is therefore
$$\rho_{jk\cdot\text{rest}} = \frac{-K_{jk}}{\sqrt{K_{jj}K_{kk}}}.$$
For a normal vector the conditional distribution of $(X_j, X_k)$ is normal (Theorem 6), so by Theorem 5 applied to it: $K_{jk} = 0$ exactly when $X_j$ and $X_k$ are independent given all the other coordinates. Zeros in the inverse covariance matrix, not in the covariance matrix, describe conditional independence. For systolic pressure, diastolic pressure and age, the partial correlation of the two pressures is 0.498, against the raw correlation 0.453. Systolic pressure is higher in older adults (slope 0.5246 mm Hg per year of age), diastolic barely (0.0228), so removing age removes variation from the systolic pressure that the diastolic does not share.
Degenerate normal distributions
When $\Sigma$ is singular there is no density, but the distribution is still normal on a smaller flat set.
Theorem 8 (the degenerate case). Let $\mathbf{X} \sim N_p(\boldsymbol\mu, \Sigma)$ with $\Sigma$ of rank $r < p$.
- If $\Sigma\mathbf{a} = \mathbf{0}$, then $\mathbf{a}^{\mathsf T}\mathbf{X} = \mathbf{a}^{\mathsf T}\boldsymbol\mu$ with probability 1. Hence $\mathbf{X} - \boldsymbol\mu$ lies in the column space of $\Sigma$, of dimension $r$, with probability 1.
- $\mathbf{X}$ has no density on $\mathbb{R}^p$.
- If $\mathbf{q}_1, \ldots, \mathbf{q}_r$ are orthonormal eigenvectors of $\Sigma$ with positive eigenvalues $\lambda_1, \ldots, \lambda_r$, then $\mathbf{X}$ has the distribution of $\boldsymbol\mu + \sum_{j=1}^{r}\sqrt{\lambda_j}\,U_j\mathbf{q}_j$ with $U_1, \ldots, U_r$ independent standard normal: a nondegenerate normal distribution in $r$ coordinates, carried into $\mathbb{R}^p$.
- If $\Sigma_{22}$ is singular in Theorem 6, choose coordinates of $\mathbf{X}_2$ whose covariance matrix is nonsingular, as many as the rank of $\Sigma_{22}$. Each remaining coordinate of $\mathbf{X}_2$ equals, with probability 1, a constant plus a linear combination of the chosen ones, so conditioning on $\mathbf{X}_2$ is conditioning on the chosen coordinates, and Theorem 6 applies to them.
Proof. (1) $\operatorname{Var}(\mathbf{a}^{\mathsf T}\mathbf{X}) = \mathbf{a}^{\mathsf T}\Sigma\mathbf{a} = 0$, so the nonnegative variable $(\mathbf{a}^{\mathsf T}\mathbf{X} - \mathbf{a}^{\mathsf T}\boldsymbol\mu)^2$ has mean 0 and is 0 with probability 1 (the lemma on a nonnegative variable with mean zero in Probability Inequalities). Take for $\mathbf{a}$ each vector of a basis of the null space of $\Sigma$, finitely many, so all these events hold together with probability 1. Then $\mathbf{X} - \boldsymbol\mu$ is perpendicular to the null space of $\Sigma$. For a symmetric matrix the null space is the set of vectors perpendicular to every row, which are the columns, so the vectors perpendicular to the null space form the column space.
(2) Take $\mathbf{a} \ne \mathbf{0}$ with $\Sigma\mathbf{a} = \mathbf{0}$. By (1), $\mathbf{X}$ lies with probability 1 in the hyperplane $\mathbf{a}^{\mathsf T}\mathbf{x} = \mathbf{a}^{\mathsf T}\boldsymbol\mu$. A hyperplane has zero volume, so it has probability 0 under any density.
(3) By the spectral decomposition (the companion article, Theorem 2), $\Sigma = \sum_{j \le r}\lambda_j\mathbf{q}_j\mathbf{q}_j^{\mathsf T} = AA^{\mathsf T}$ for the $p \times r$ matrix $A$ with columns $\sqrt{\lambda_j}\mathbf{q}_j$. By Theorem 1 the distribution of $\mathbf{X}$ is that of $\boldsymbol\mu + A\mathbf{U}$.
(4) Write $\Sigma_{22} = G^{\mathsf T}G$ with $G = \Sigma_{22}^{1/2}$, so $\sigma_{jk} = \mathbf{g}_j^{\mathsf T}\mathbf{g}_k$ for the columns $\mathbf{g}_j$ of $G$. The rank of $\Sigma_{22}$, say $s$, is the rank of $G$ (their null spaces agree, because $G^{\mathsf T}G\mathbf{v} = \mathbf{0}$ gives $\lVert G\mathbf{v}\rVert^2 = 0$). So some $s$ columns of $G$ are linearly independent, and their matrix of inner products, the covariance matrix of the corresponding coordinates, is nonsingular: if $\sum_k c_k\mathbf{g}_j^{\mathsf T}\mathbf{g}_k = 0$ for each chosen $j$, then $\lVert\sum_k c_k\mathbf{g}_k\rVert^2 = 0$ and every $c_k = 0$. Every other column $\mathbf{g}_l$ is a combination $\sum_k c_k\mathbf{g}_k$ of the chosen ones, so the variance of $X_l - \sum_k c_kX_k$ is $\lVert\mathbf{g}_l - \sum_k c_k\mathbf{g}_k\rVert^2 = 0$, and by the argument of (1) that difference is constant with probability 1. $\square$
Durrett (2019, Section 3.10) makes the same point: the distribution in degenerate cases is computed by a linear transformation to the nondegenerate case.
Example (pulse pressure). Add the pulse pressure to the two pressures: $\mathbf{T} = (\text{systolic}, \text{diastolic}, \text{systolic} - \text{diastolic})$. Its covariance matrix is $3 \times 3$ with the pulse pressure's variance 328.68 in the corner, but its rank is 2: its eigenvalues are 668.33, 207.13 and 0, and the eigenvector of 0 is $(1, -1, -1)/\sqrt{3}$, the combination that is identically 0. A normal model for the triple can only be a degenerate one, living on the plane where the third coordinate is the difference of the first two. The same happens wherever variables are constrained to add up: the deviations $X_i - \bar{X}$ of a normal sample sum to zero (Sampling Distributions from the Normal), and the limiting normal distribution of multinomial counts lives on the plane where the deviations sum to zero (The Chi-Square Goodness-of-Fit Test).
What is proved here, what is cited, and what the experiment shows
Proved here. Lemma 1 (independence by characteristic functions), given uniqueness; Theorem 1 (characteristic and moment generating functions, and that $\boldsymbol\mu$ and $\Sigma$ determine the distribution); Theorem 2 (the characterization by normal linear combinations); Theorem 3 (affine maps, marginals, sums and stacking); Theorem 4 (the density), given the linear change of variables; Theorem 5 (uncorrelated blocks are independent) and the counterexamples; Theorem 6 (conditional distributions); Theorem 7 (block factorization, determinant, partitioned inverse, the split of the quadratic form) and the partial correlation formula; Theorem 8 (degenerate cases).
Cited. The uniqueness theorem for characteristic functions of random vectors, from the inversion formula of Durrett (2019, Theorem 3.10.4); the linear change of variables for densities and the product rule for independent densities, as cited on The Chi-Squared and t Distributions; the existence of a pair of independent vectors with given distributions; from linear algebra, the spectral theorem and the determinant facts of Axler (2024), through Covariance Matrices and Quadratic Forms, and the fact that a hyperplane has zero volume. As in the rest of the Probability Theory unit, measure theory is avoided: Random Variables and Distribution Functions says what is left out.
What the experiment shows. On the 5,232 adults, the conditional means of diastolic pressure stay within 4 mm Hg of the normal line over the bands of the table, though outside the range of the normal samples in six of the nine bands, while the conditional standard deviation doubles; the projections are skewed by as much as 1.25 in absolute value; the normal ellipses hold too many adults at the center (56.0% inside the 50% ellipse) and too many beyond the 99% ellipse (2.5% against 1%), and so too few in between. Thirty samples of the same size from the fitted normal distribution (seeds 1 to 30) show what sampling variation alone produces: in the middle half of samples, a spread ratio between 0.97 and 1.02 and a largest skewness between 0.044 and 0.068. These are observations about this population. They say that the normal model describes the center and the slope of the regression of these two pressures roughly, within 4 mm Hg, though the band means bend away from the line by more than normal samples do, and that it describes their spread and tails poorly.
Worked problems
These problems were written for this article; none comes from a textbook. They use the NHANES readings and small exact cases, and every number is computed and asserted by the script named at the end.
Problem 1 (predicting a third reading). NHANES takes three consecutive systolic readings. For the 4,730 adults with all three, the mean vector is $(125.74, 125.40, 124.96)$ and the covariance matrix (divisor $N$) has entries $\sigma_{11} = 387.28$, $\sigma_{12} = 373.17$, $\sigma_{13} = 365.34$, $\sigma_{22} = 385.21$, $\sigma_{23} = 368.64$ and $\sigma_{33} = 374.94$, as in Expectation, Variance and Covariance. Treating the readings as multivariate normal, find the distribution of the third reading of an adult whose first two readings are 150 and 146. Compare with conditioning on the second reading alone, and with the data.
Solution. Take $\mathbf{X}_1 = X_3$ and $\mathbf{X}_2 = (X_1, X_2)$. Theorem 6 needs $B = \Sigma_{12}\Sigma_{22}^{-1}$, here the row $(365.34, 368.64)$ times the inverse of $\begin{pmatrix} 387.28 & 373.17 \\ 373.17 & 385.21\end{pmatrix}$. That matrix has determinant $387.28 \times 385.21 - 373.17^2 = 9{,}926.1$, so
$$B = \frac{1}{9{,}926.1}\big(365.34 \times 385.21 - 368.64 \times 373.17,\ \ 368.64 \times 387.28 - 365.34 \times 373.17\big) = (0.3191,\ 0.6479),$$
computed from unrounded entries. The conditional mean is $124.96 + 0.3191(150 - 125.74) + 0.6479(146 - 125.40) = 146.05$. The conditional variance is $\sigma_{33} - B(\sigma_{13}, \sigma_{23})^{\mathsf T} = 374.94 - (0.3191 \times 365.34 + 0.6479 \times 368.64) = 19.54$, a standard deviation of 4.42, against the third reading's own standard deviation of 19.36. Knowing two readings removes about 95% of the variance of the third, because most of the variance is between people. The later reading gets about twice the weight (0.6479 against 0.3191), although its correlation with the third reading is only slightly higher (0.970 against 0.959): the first two readings nearly duplicate each other (correlation 0.966), so a small edge decides how the weight is shared.
With the second reading alone, the slope is $368.64/385.21 = 0.9570$, the conditional mean $124.96 + 0.9570(146 - 125.40) = 144.67$ and the standard deviation $\sqrt{374.94 - 368.64^2/385.21} = 4.71$. Adding the first reading moves the prediction up by 1.4 mm Hg, because 150 is above 146, and narrows it a little.
In the data, 160 adults have a first reading within 4 mm Hg of 150 and a second within 4 of 146. Their mean first and second readings are 149.41 and 146.21, where the plane gives 146.00. Their third readings average 146.38 with a standard deviation of 5.41: the center is predicted to within half a millimeter, the spread is somewhat wider than 4.42, partly because the band itself has width.
Problem 2 (the two regression lines). For the 5,232 adults, find the line for systolic given diastolic pressure and compare it with the line for diastolic given systolic. Why are they not the same line? What does each predict, starting from a systolic pressure of 150?
Solution. By Theorem 6 with the roles exchanged, the slope of systolic on diastolic is $\sigma_{12}/\sigma_2^2 = 109.05/144.14 = 0.7566$ and the intercept $126.31 - 0.7566 \times 72.49 = 71.47$. The conditional standard deviation is $20.07\sqrt{1 - 0.453^2} = 17.89$. Drawn in one picture with systolic pressure on the horizontal axis, the line for systolic given diastolic has slope $1/0.7566 = 1.32$, while the line for diastolic given systolic has slope 0.2708. The lines would coincide only if the two slopes were reciprocals; instead their product is $\sigma_{12}^2/(\sigma_1^2\sigma_2^2) = \rho^2 = 0.2049$. Each line minimizes errors in a different direction, vertical for one and horizontal for the other, and they coincide only when $|\rho| = 1$.
From a systolic pressure of 150, whose standardized value is $(150 - 126.31)/20.07 = 1.181$, the first line predicts a diastolic of 78.90, whose standardized value is $(78.90 - 72.49)/12.01 = 0.534$: $\rho$ times 1.181. From a diastolic of 78.90, the second line predicts a systolic pressure of $71.47 + 0.7566 \times 78.90 = 131.16$ (from unrounded coefficients), not 150. Each prediction moves toward the mean, by the factor $\rho$ in standardized units. Going there and back does not return to the start.
Problem 3 (normal margins, positive correlation, not jointly normal). Let $(X, Y)$ have the density $2\phi(x)\phi(y)$ where $x$ and $y$ have the same sign, and 0 where they have opposite signs, with $\phi$ the standard normal density. Show that $X$ and $Y$ are standard normal, find their correlation, and show that the pair is not multivariate normal.
Solution. The function is nonnegative, and it integrates to 1 because the quadrants $\{x > 0, y > 0\}$ and $\{x < 0, y < 0\}$ each carry $\tfrac14$ of the product density. For $x > 0$, $f_X(x) = 2\phi(x)\int_0^\infty\phi(y)\,dy = 2\phi(x) \cdot \tfrac12 = \phi(x)$, and the same holds for $x < 0$, so $X$ is standard normal; by symmetry so is $Y$. Both have mean 0, so
$$\operatorname{Cov}(X, Y) = E[XY] = 2\Big(\int_0^\infty\!\!\int_0^\infty xy\,\phi(x)\phi(y)\,dy\,dx + \int_{-\infty}^0\!\!\int_{-\infty}^0 xy\,\phi(x)\phi(y)\,dy\,dx\Big) = 4\Big(\int_0^\infty x\phi(x)\,dx\Big)^2 = 4 \cdot \frac{1}{2\pi} = \frac{2}{\pi},$$
because $\int_0^\infty x\phi(x)\,dx = [-\phi(x)]_0^\infty = \phi(0) = 1/\sqrt{2\pi}$. The correlation is $2/\pi = 0.6366$. If the pair were multivariate normal, its covariance matrix $\begin{pmatrix} 1 & 2/\pi \\ 2/\pi & 1\end{pmatrix}$ would be nonsingular, because $|2/\pi| < 1$. By Theorem 4 its density would be positive everywhere, so $P(XY < 0) > 0$. But here $P(XY < 0) = 0$. So the pair is not multivariate normal, although both margins are normal and the correlation is an ordinary 0.64.
Problem 4 (a dependent, uncorrelated pair in which $Y$ is a function of $X$). Let $X$ be standard normal, let $c > 0$, and put $Y = X$ when $|X| \le c$ and $Y = -X$ when $|X| > c$. Show that $Y$ is standard normal, find the $c$ that makes $X$ and $Y$ uncorrelated, and show that the pair is then not multivariate normal.
Solution. For any $y$, $P(Y \le y) = P(X \le y,\ |X| \le c) + P(-X \le y,\ |X| > c)$. Since $-X$ has the distribution of $X$ and the event $|X| > c$ is the same for $X$ and $-X$, the second term equals $P(X \le y,\ |X| > c)$, and the sum is $P(X \le y)$. So $Y$ is standard normal. Next, $XY = X^2$ when $|X| \le c$ and $-X^2$ otherwise, so
$$E[XY] = E[X^2;\ |X| \le c] - E[X^2;\ |X| > c] = 2E[X^2;\ |X| \le c] - 1,$$
using $E[X^2] = 1$, where $E[X^2;\ |X| \le c]$ means $E[X^2\,\mathbf{1}\{|X| \le c\}]$. Integrating by parts with $(-\phi)' = x\phi$,
$$E[X^2;\ |X| \le c] = 2\int_0^c x \cdot x\phi(x)\,dx = 2\Big([-x\phi(x)]_0^c + \int_0^c\phi(x)\,dx\Big) = 2\big(\Phi(c) - \tfrac12 - c\,\phi(c)\big).$$
The covariance is 0 when this equals $\tfrac12$, that is, when $\Phi(c) - c\,\phi(c) = \tfrac34$. The left side increases with $c$ (its derivative is $c^2\phi(c) > 0$), from $\tfrac12$ at 0 toward 1, so there is exactly one root, $c = 1.5382$, and at it $E[X^2;\ |X| \le c] = 0.5000$. Then $X$ and $Y$ are uncorrelated standard normal variables, and $Y$ is a function of $X$, so they are as dependent as two variables can be. The sum $X + Y$ is $2X$ when $|X| \le c$ and 0 otherwise, so $P(X + Y = 0) = P(|X| > c) = 2(1 - \Phi(c)) = 0.1240$. A normal variable with a positive variance gives no single point positive probability, and $X + Y$ is not constant, so it is not normal, and by Theorem 2 the pair is not multivariate normal.
Summary
- $\mathbf{X}$ is multivariate normal if $\mathbf{X} = \boldsymbol\mu + A\mathbf{Z}$ with $\mathbf{Z}$ standard normal; then $\varphi_{\mathbf{X}}(\mathbf{t}) = \exp(i\mathbf{t}^{\mathsf T}\boldsymbol\mu - \tfrac12\mathbf{t}^{\mathsf T}\Sigma\mathbf{t})$ with $\Sigma = AA^{\mathsf T}$, and $\boldsymbol\mu$ and $\Sigma$ determine the distribution $N_p(\boldsymbol\mu, \Sigma)$, for any positive semidefinite $\Sigma$ (Theorem 1).
- A vector is multivariate normal exactly when every linear combination is normal (Theorem 2); normal coordinates are not enough.
- Affine maps, subvectors, sums of independent normal vectors and stacks of them are multivariate normal (Theorem 3).
- For nonsingular $\Sigma$ the density is $(2\pi)^{-p/2}(\det\Sigma)^{-1/2}e^{-\Delta^2(\mathbf{x})/2}$ with the squared Mahalanobis distance $\Delta^2$; its contours are ellipsoids (Theorem 4).
- Blocks of a normal vector are independent exactly when uncorrelated (Theorem 5); uncorrelated normal variables that are not jointly normal can be completely dependent.
- Given $\mathbf{X}_2$, $\mathbf{X}_1$ is normal with mean $\boldsymbol\mu_1 + \Sigma_{12}\Sigma_{22}^{-1}(\mathbf{x}_2 - \boldsymbol\mu_2)$ and covariance the Schur complement $\Sigma_{11\cdot 2}$, which does not depend on $\mathbf{x}_2$ (Theorem 6).
- $\det\Sigma = \det\Sigma_{11\cdot 2}\det\Sigma_{22}$, the upper-left block of $\Sigma^{-1}$ is $\Sigma_{11\cdot 2}^{-1}$, and partial correlations are $-K_{jk}/\sqrt{K_{jj}K_{kk}}$ from the precision matrix (Theorem 7).
- A singular $\Sigma$ puts the distribution on a flat set of dimension equal to its rank, with no density on the whole space (Theorem 8).
- On NHANES blood pressures the conditional means stay within 4 mm Hg of the normal line, though they bend away from it by more than normal samples do, while the conditional spread doubles and the projections are skewed: the pair is not jointly normal.
What we have learned
The multivariate normal distribution is the image of independent standard normal variables under an affine map. Everything else follows from that and from the characteristic function: two numbers per pair of variables determine the distribution, linear maps stay in the family, zero covariance means independence, and conditioning gives a normal distribution with a linear mean and a fixed spread. These are strong properties, and real data need not have them: for the blood pressures of the adults the first test, a linear conditional mean, is met roughly, and the second, a constant conditional spread, fails clearly.
The proofs leaned on facts about covariance matrices that this article took from its companion: the square root of a positive semidefinite matrix, the spectral decomposition, and the chi-squared distribution of the Mahalanobis distance that measures how far an observation lies from the center. Covariance Matrices and Quadratic Forms proves them, decides when a quadratic form in normal variables is chi-squared, and finds the sampling behavior of the sample mean vector and sample covariance matrix.
References
- Axler, S. (2024). Linear Algebra Done Right (4th ed.). Springer (open access). Results 9.48 (determinant of an upper-triangular matrix), 9.49 (the determinant is multiplicative) and 9.56(a) (determinant of the transpose); the spectral theorem and square roots through Covariance Matrices and Quadratic Forms. record · 4 pages
- Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer. Section 2.3, eq. 2.43 (the density), eqs. 2.76 and 2.77 (partitioned inverse and the Schur complement) and eqs. 2.81 and 2.82 (conditional mean and covariance). record · 10 pages
- Durrett, R. (2019). Probability: Theory and Examples (5th ed.). Cambridge University Press. Section 3.10: Theorem 3.10.4 (inversion formula in $\mathbb{R}^d$), the discussion of the multivariate normal distribution and its density, and Exercises 3.10.6 to 3.10.8; numbers of the author's version of January 11, 2019. record · 10 pages
- National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018 data files and documentation (DEMO_J, BPX_J). Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 95 pages