Expectation, Variance and Covariance

The expected value defined for every random variable by one formula, shown to agree with the sum for discrete variables and the integral for densities, with linearity, monotonicity and the product rule for independent variables proved without assuming either form. Then the variance, the covariance and its bilinearity, the variance of a linear combination, the correlation coefficient with its bounds and the cases of equality, the range a correlation can reach for given margins, and the best linear predictor. Worked problems on a measurement with a detection limit, a variable with no mean, the covariance matrix of three blood pressure readings, the correlation of two indicators, and a four-point population.

Among the 5,247 adults whose blood pressure was measured in the National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018, the mean age is 49.87 years and the mean systolic pressure is 126.36 mm Hg, with variances 345.45 and 405.01 (National Center for Health Statistics, 2020a). Older adults tend to have higher pressures: the covariance of age and pressure is 182.01 and their correlation is 0.4866. The course page Random Variables, Expectation and Variance defines these summaries for a discrete variable and proves their rules on a finite sample space. What are they for a random variable of any kind, and which of the rules survive? The expected value can be defined by one formula for every random variable, and linearity, the product rule for independent variables and the bound $-1 \le \rho \le 1$ on a correlation all hold in that generality.

The expected value of a random variable is the average of its values weighted by their probabilities, the variance is the expected squared distance from that average, and the covariance of two variables is the expected product of their distances from their averages. This article is the home of these three notions for the whole site. The inequalities, generating functions and limit theorems of Probability Theory are built on the rules proved here.

First we define the expected value, show when it fails to exist, and prove that one formula covers discrete variables, densities and mixtures. Then we prove the rules: linearity, the formula for the expected value of a function, and the product rule for independent variables. Next come the variance and the covariance, with the variance of a linear combination. Finally we treat the correlation coefficient, its bounds, and the straight line that predicts one variable best from another. The article assumes Random Variables and Distribution Functions.

The expected value

For a discrete variable $X$ with values $x_1, x_2, \ldots$ and mass function $p$, and for a variable with density $f$, the expected value is

$$E[X] = \sum_k x_k\,p(x_k) \qquad\text{or}\qquad E[X] = \int_{-\infty}^{\infty} x\,f(x)\,dx,$$

provided the sum or the integral converges absolutely: $\sum_k \lvert x_k\rvert\,p(x_k) < \infty$ or $\int \lvert x\rvert f(x)\,dx < \infty$. We then say that $X$ has a finite mean. For the pressures the sum has 237 terms and equals the population mean, 126.36.

Why absolute convergence is required

A sum over finitely many values always exists. With infinitely many values, or a density on an unbounded range, the expected value can fail to exist, and the requirement of absolute convergence is what detects it.

Example (the Cauchy density has no mean). The Cauchy density $f(x) = 1/(\pi(1 + x^2))$ is symmetric about 0, and it is tempting to say that its mean is 0. Integrate $x f(x)$ over a range $[-a, b]$:

$$\int_{-a}^{b} \frac{x}{\pi(1 + x^2)}\,dx = \frac{1}{2\pi}\ln\frac{1 + b^2}{1 + a^2}.$$

With $b = a$ this is 0 for every $a$. With $b = 2a$ it is $\frac{1}{2\pi}\ln\frac{1 + 4a^2}{1 + a^2}$, which is 0.2194 at $a = 10$ and tends to $\frac{\ln 4}{2\pi} = \frac{\ln 2}{\pi} = 0.2206$. The value depends on how the two ends go to infinity, so the integral over the whole line has no value. The absolute integral shows why: $\int_{-b}^{b} \lvert x\rvert f(x)\,dx = \ln(1 + b^2)/\pi$, which is 4.40 at $b = 1{,}000$ and grows without bound. A distribution without a mean is not a curiosity of calculus. The averages of Cauchy samples do not settle down as the sample grows, as The Central Limit Theorem shows, and The Laws of Large Numbers proves that a finite mean is enough for the averages to settle.

One definition for every random variable

The two formulas leave a gap. Random Variables and Distribution Functions shows that a measurement with a detection limit is neither discrete nor continuous, and the same is true of $\max(X, 0)$ for a continuous $X$ that is negative with a probability strictly between 0 and 1. A definition that needs a mass function or a density does not apply to them. One that needs only probabilities of events does.

Here is the idea for a variable that is never negative. Write a value $x \ge 0$ as the length of the interval from 0 to $x$, that is, as $\int_0^\infty \mathbf{1}\{x > t\}\,dt$, where $\mathbf{1}\{\cdot\}$ is 1 when the statement holds and 0 otherwise. Averaging over the distribution of $X$ turns the indicator into the probability $P(X > t)$.

For any random variable $X$ define

$$E^+ = \int_0^\infty P(X > t)\,dt, \qquad E^- = \int_0^\infty P(X < -t)\,dt.$$

Both integrands are nonincreasing functions of $t$ with values between 0 and 1, so both integrals exist as improper Riemann integrals, finite or $+\infty$. We say that $X$ has a finite mean when both are finite, and define

$$E[X] = E^+ - E^- = \int_0^\infty \big(1 - F(t)\big)\,dt - \int_{-\infty}^{0} F(t)\,dt,$$

where $F$ is the distribution function. (The second form uses $P(X \le -t)$ in place of $P(X < -t)$. The two differ only at the jumps of $F$, which are countably many. Every interval contains points where they agree, so their Riemann sums can be taken equal and the integrals coincide.) For a variable that is never negative, $E^- = 0$ and $E[X] = E^+$ is always defined, possibly as $+\infty$. The expected value is the area between the distribution function and the level 1 to the right of zero, minus the area under it to the left.

Theorem 1 (the general definition agrees with the sum and the integral).

  1. If $X$ is discrete, then $E^+ = \sum_{x_k > 0} x_k\,p(x_k)$ and $E^- = \sum_{x_k < 0} \lvert x_k\rvert\,p(x_k)$.
  2. If $X$ has a density $f$, then $E^+ = \int_0^\infty x f(x)\,dx$ and $E^- = \int_{-\infty}^0 \lvert x\rvert f(x)\,dx$.

In both cases $X$ has a finite mean in the new sense exactly when the sum or integral converges absolutely, and $E[X]$ is the same number.

Proof. (1) List the positive values as $y_1, y_2, \ldots$ with probabilities $q_1, q_2, \ldots$. For $t \ge 0$, $P(X > t) = \sum_j q_j\,\mathbf{1}\{y_j > t\}$. Let $g_n(t) = \sum_{j \le n} q_j\,\mathbf{1}\{y_j > t\}$ be a partial sum, a step function with $\int_0^\infty g_n(t)\,dt = \sum_{j \le n} q_j y_j$. Since $g_n(t) \le P(X > t)$, integrating gives $\sum_{j \le n} q_j y_j \le E^+$ for every $n$, so $\sum_j q_j y_j \le E^+$. For the other direction, $P(X > t) - g_n(t) = \sum_{j > n} q_j\,\mathbf{1}\{y_j > t\} \le \varepsilon_n$, where $\varepsilon_n = \sum_{j > n} q_j \to 0$. So for any $T > 0$,

$$\int_0^T P(X > t)\,dt \le \sum_{j \le n} q_j y_j + T\varepsilon_n \le \sum_j q_j y_j + T\varepsilon_n.$$

Letting $n \to \infty$ and then $T \to \infty$ gives $E^+ \le \sum_j q_j y_j$. So $E^+ = \sum_j q_j y_j$, both sides finite or both infinite. For $E^-$, apply what was just proved to the discrete variable $-X$. Its positive values are the numbers $\lvert x_k\rvert$ with $x_k < 0$, and $P(-X > t) = P(X < -t)$.

(2) $P(X > t) = \int_t^\infty f(x)\,dx$, so $E^+$ is the integral of $f(x)$ over the region $0 < t < x$ of the $(t, x)$ plane. Integrating over $t$ first,

$$E^+ = \int_0^\infty\!\!\int_t^\infty f(x)\,dx\,dt = \int_0^\infty f(x)\Big[\int_0^x dt\Big]dx = \int_0^\infty x\,f(x)\,dx.$$

The exchange of the order of integration for a nonnegative integrand is the fact about double integrals that Random Variables and Distribution Functions takes as given. For $E^-$, apply this to $-X$, which has density $f(-x)$: $E^- = \int_0^\infty x f(-x)\,dx = \int_{-\infty}^0 \lvert x\rvert f(x)\,dx$.

In both cases the absolute sum or integral is $E^+ + E^-$, so it is finite exactly when both parts are, and then the sum or integral for $E[X]$ splits into its positive and negative parts, $E^+ - E^-$. $\square$

The definition handles mixtures with no further work. If $F = \alpha F_{\mathrm{d}} + (1 - \alpha)F_{\mathrm{c}}$, the integrands are the same weighted averages, and so $E[X] = \alpha E_{\mathrm{d}} + (1 - \alpha)E_{\mathrm{c}}$: sum over the atoms, integrate over the continuous part, and add. Problem 1 does this for a detection limit. Grinstead and Snell (1997) define the expected value separately for discrete variables (their Definition 6.1) and for densities (Definition 6.4), as the course pages do. Billingsley (1995, Section 21, "Expected Values") treats it once for all random variables, as an integral with respect to the probability.

Rules that hold for every random variable

Theorems 2, 3 and 5 below are stated for arbitrary random variables on one sample space, and their proofs never ask whether a variable is discrete or continuous. Theorem 4 is a computing formula for the discrete case and for densities.

Theorem 2 (basic properties). Let $X$ and $Y$ have finite means.

  1. Monotonicity. If $X \le Y$ for every outcome, then $E[X] \le E[Y]$. If $U \le V \le W$ and $U$, $W$ have finite means, then so does $V$. An inequality that holds only with probability 1 serves as well. If $X = X'$ outside an event of probability zero, then $\{X > t\}$ and $\{X' > t\}$ differ only inside that event, so $P(X > t) = P(X' > t)$ for every $t$, and likewise $P(X < -t) = P(X' < -t)$. So $X$ and $X'$ have the same expected value, and a variable may be changed on an event of probability zero, to make an inequality hold for every outcome, without changing it.
  2. Scale and shift. $aX + b$ has a finite mean, and $E[aX + b] = aE[X] + b$.
  3. Indicators and constants. $E[\mathbf{1}_A] = P(A)$ for an event $A$, where $\mathbf{1}_A$ is 1 when $A$ occurs and 0 otherwise, and $E[c] = c$.
  4. Absolute values. $E\lvert X\rvert = E^+ + E^-$, so $X$ has a finite mean exactly when $E\lvert X\rvert < \infty$, and $\lvert E[X]\rvert \le E\lvert X\rvert$.

Proof. (1) If $X \le Y$, then $\{X > t\} \subseteq \{Y > t\}$ and $\{Y < -t\} \subseteq \{X < -t\}$. So $E^+$ of $X$ is at most that of $Y$, and $E^-$ of $X$ is at least that of $Y$, which gives $E[X] \le E[Y]$. For the second statement the same inclusions give $E^+_V \le E^+_W < \infty$ and $E^-_V \le E^-_U < \infty$.

(2) Scale. For $a > 0$, substituting $t = as$,

$$\int_0^\infty P(aX > t)\,dt = \int_0^\infty P(X > t/a)\,dt = a\int_0^\infty P(X > s)\,ds,$$

and in the same way $\int_0^\infty P(aX < -t)\,dt = a\int_0^\infty P(X < -s)\,ds$. So $E[aX] = aE[X]$. For $-X$, the events $\{-X > t\}$ and $\{-X < -t\}$ are $\{X < -t\}$ and $\{X > t\}$, so $E^+$ and $E^-$ change places and $E[-X] = -E[X]$. A negative $a$ is $-\lvert a\rvert$, and $a = 0$ gives the constant 0, with both integrals 0.

Shift. Write $G(s) = P(X > s)$, so that $E[X] = \int_0^\infty G(s)\,ds - \int_{-\infty}^0 (1 - G(s))\,ds$. The variable $X + b$ has $P(X + b > t) = G(t - b)$. Changing variables in both integrals ($s = t - b$ in the first, $s = -t - b$ in the second, where $P(X + b \le -t) = 1 - G(-t - b)$), and using the convention $\int_p^q = -\int_q^p$ so that either sign of $b$ is covered,

$$E[X + b] = \int_{-b}^\infty G(s)\,ds - \int_{-\infty}^{-b} \big(1 - G(s)\big)\,ds = \Big[\int_0^\infty G + \int_{-b}^0 G\Big] - \Big[\int_{-\infty}^0 (1 - G) - \int_{-b}^0 (1 - G)\Big] = E[X] + \int_{-b}^0 1\,ds = E[X] + b.$$

All four integrals in brackets are finite, so $X + b$ has a finite mean. Combining scale and shift gives part 2.

(3) $P(\mathbf{1}_A > t)$ is $P(A)$ for $0 \le t < 1$ and 0 for $t \ge 1$, and $\mathbf{1}_A$ is never negative, so $E[\mathbf{1}_A] = \int_0^1 P(A)\,dt = P(A)$. A constant $c$ is $c\,\mathbf{1}_\Omega$, so $E[c] = c\,P(\Omega) = c$ by part 2.

(4) For $t \ge 0$ the event $\{\lvert X\rvert > t\}$ is the union of the mutually exclusive events $\{X > t\}$ and $\{X < -t\}$. Integrating, $E\lvert X\rvert = E^+ + E^-$. And $\lvert E^+ - E^-\rvert \le E^+ + E^-$ because both are nonnegative. $\square$

Theorem 3 (linearity). If $X$ and $Y$ have finite means, then so does $X + Y$, and

$$E[X + Y] = E[X] + E[Y].$$

No independence is assumed. Hence $E[a_1X_1 + \cdots + a_nX_n] = a_1E[X_1] + \cdots + a_nE[X_n]$.

Here is the plan. For discrete variables linearity is a rearrangement of a double sum. Any variable lies within $1/n$ of a discrete one, obtained by rounding down to a multiple of $1/n$, and monotonicity carries the result across the rounding.

Proof. Step 1: discrete variables. Let $X$ and $Y$ be discrete with $p_{ij} = P(X = x_i,\ Y = y_j)$. The variable $X + Y$ is discrete, and each of its values $z$ has probability $\sum_{(i,j) :\, x_i + y_j = z} p_{ij}$. The double series $\sum_{i,j} (x_i + y_j)\,p_{ij}$ converges absolutely, because $\sum_{i,j} \lvert x_i + y_j\rvert\,p_{ij} \le \sum_{i,j} \lvert x_i\rvert\,p_{ij} + \sum_{i,j} \lvert y_j\rvert\,p_{ij} = E\lvert X\rvert + E\lvert Y\rvert$, using $\sum_j p_{ij} = P(X = x_i)$ and $\sum_i p_{ij} = P(Y = y_j)$. An absolutely convergent series may be grouped and reordered freely. Grouping the terms by the value of $x_i + y_j$ gives $E[X + Y]$, by Theorem 1. Splitting them gives

$$\sum_{i,j} (x_i + y_j)\,p_{ij} = \sum_i x_i \sum_j p_{ij} + \sum_j y_j \sum_i p_{ij} = E[X] + E[Y].$$

Step 2: rounding. For a whole number $n \ge 1$ let $X_n = \lfloor nX \rfloor / n$, the largest multiple of $1/n$ that is at most $X$. It is discrete, and $X - 1/n < X_n \le X$. By Theorem 2, $X_n$ has a finite mean, because it lies between $X - 1/n$ and $X$, and

$$E[X] - \tfrac1n \le E[X_n] \le E[X].$$

Step 3: the sandwich. Define $Y_n$ from $Y$ in the same way. Adding the two rounding inequalities, $X_n + Y_n \le X + Y \le X_n + Y_n + 2/n$. The outer variables are discrete with finite means by Step 1, so $X + Y$ has a finite mean by Theorem 2, and by monotonicity, the shift rule and Step 1,

$$E[X_n] + E[Y_n] \le E[X + Y] \le E[X_n] + E[Y_n] + \tfrac2n.$$

With Step 2 this gives $E[X] + E[Y] - \tfrac2n \le E[X + Y] \le E[X] + E[Y] + \tfrac2n$ for every $n$, and letting $n \to \infty$ proves the theorem. The statement for $n$ terms follows by applying the two-term result and the scale rule repeatedly. $\square$

Step 1 is the proof the course page gives for a finite sample space, now for countably many values. Steps 2 and 3 extend it to every pair of variables, including a discrete one added to a continuous one, for which no joint mass function or joint density exists.

Theorem 4 (the expected value of a function).

Part 1. Let $X$ be discrete, or a vector $(X_1, \ldots, X_m)$ of discrete variables, with values $x_1, x_2, \ldots$ and probabilities $p(x_k)$. For any function $g$ with $\sum_k \lvert g(x_k)\rvert\,p(x_k) < \infty$,

$$E[g(X)] = \sum_k g(x_k)\,p(x_k).$$

Part 2. Let $X$ have density $f$, and let $g$ be a function for which every set $\{x : g(x) > t\}$ and $\{x : g(x) < -t\}$ is a finite union of intervals, with $\int \lvert g(x)\rvert f(x)\,dx < \infty$. Then

$$E[g(X)] = \int_{-\infty}^{\infty} g(x)\,f(x)\,dx.$$

Part 3. Let $(X, Y)$ have joint density $f_{X,Y}$, and let $g(x, y)$ be a function for which every set $\{g > t\}$ and $\{g < -t\}$ is a region bounded by finitely many smooth curves, with $\iint \lvert g\rvert f_{X,Y} < \infty$. Then

$$E[g(X, Y)] = \int_{-\infty}^{\infty}\!\!\int_{-\infty}^{\infty} g(x, y)\,f_{X,Y}(x, y)\,dx\,dy.$$

The point of the theorem is that the distribution of $g(X)$ never has to be found. The condition on $g$ in parts 2 and 3 holds for polynomials, exponentials, absolute values, and indicators of intervals and of half-planes, which are the functions this unit uses.

Proof. (1) The variable $g(X)$ is discrete, and each of its values $w$ has probability $\sum_{k :\, g(x_k) = w} p(x_k)$. By Theorem 1, $E[g(X)] = \sum_w w \sum_{k :\, g(x_k) = w} p(x_k)$, and this is the absolutely convergent series $\sum_k g(x_k)\,p(x_k)$ with its terms grouped by the value of $g(x_k)$.

(2) Let $g^+ = \max(g, 0)$ and $g^- = \max(-g, 0)$, so that $g = g^+ - g^-$ and $\lvert g\rvert = g^+ + g^-$. The probability that $X$ falls in a finite union of intervals is the integral of $f$ over it, so $P(g(X) > t) = \int \mathbf{1}\{g(x) > t\}\,f(x)\,dx$. Exchanging the order of integration,

$$E^+ = \int_0^\infty\!\!\int_{-\infty}^{\infty} \mathbf{1}\{g(x) > t\}\,f(x)\,dx\,dt = \int_{-\infty}^{\infty} f(x)\Big[\int_0^\infty \mathbf{1}\{g(x) > t\}\,dt\Big]dx = \int_{-\infty}^{\infty} g^+(x)\,f(x)\,dx,$$

because the inner integral is the length of the set of $t \ge 0$ below $g(x)$, which is $g^+(x)$. In the same way,

$$E^- = \int_0^\infty\!\!\int_{-\infty}^{\infty} \mathbf{1}\{g(x) < -t\}\,f(x)\,dx\,dt = \int_{-\infty}^{\infty} f(x)\Big[\int_0^\infty \mathbf{1}\{-g(x) > t\}\,dt\Big]dx = \int_{-\infty}^{\infty} g^-(x)\,f(x)\,dx.$$

Both are finite because $\int \lvert g\rvert f < \infty$, and subtracting gives $E[g(X)] = \int (g^+ - g^-)\,f = \int g\,f$.

(3) The probability that $(X, Y)$ falls in a region bounded by finitely many smooth curves is the double integral of $f_{X,Y}$ over it, so $P(g(X, Y) > t) = \iint \mathbf{1}\{g(x, y) > t\}\,f_{X,Y}(x, y)\,dx\,dy$. Integrating over $t$ first, as in part 2,

$$E^+ = \iint f_{X,Y}(x, y)\Big[\int_0^\infty \mathbf{1}\{g(x, y) > t\}\,dt\Big]dx\,dy = \iint g^+(x, y)\,f_{X,Y}(x, y)\,dx\,dy,$$

$$E^- = \iint f_{X,Y}(x, y)\Big[\int_0^\infty \mathbf{1}\{-g(x, y) > t\}\,dt\Big]dx\,dy = \iint g^-(x, y)\,f_{X,Y}(x, y)\,dx\,dy,$$

and subtracting gives the formula. $\square$

Parts 2 and 3 rest on the exchange of the order of integration for nonnegative integrands, the cited fact about multiple integrals. Grinstead and Snell (1997) state part 2 as their Theorem 6.11 and refer elsewhere for its proof, and Casella and Berger (2002) treat expected values of functions in their Section 2.2.

Theorem 5 (the product rule for independent variables). If $X$ and $Y$ are independent with finite means, then $XY$ has a finite mean and

$$E[XY] = E[X]\,E[Y].$$

Consequently $E[g(X)\,h(Y)] = E[g(X)]\,E[h(Y)]$ whenever the two factors have finite means, and $E[\prod_i g_i(X_i)] = \prod_i E[g_i(X_i)]$ for independent $X_1, \ldots, X_n$ when each factor has a finite mean.

The plan is that of Theorem 3: a factored double sum for discrete variables, then rounding.

Proof. Step 1: discrete variables. Let $X$ and $Y$ be discrete and independent, so that $p_{ij} = P(X = x_i)\,P(Y = y_j)$. By part 1 of Theorem 4 for the pair, with $g(x, y) = \lvert xy\rvert$ and then $g(x, y) = xy$,

$$E\lvert XY\rvert = \sum_{i,j} \lvert x_i\rvert\,\lvert y_j\rvert\,P(X = x_i)\,P(Y = y_j) = E\lvert X\rvert\,E\lvert Y\rvert < \infty, \qquad E[XY] = \sum_i x_i\,P(X = x_i)\sum_j y_j\,P(Y = y_j) = E[X]\,E[Y].$$

Step 2: variables that are never negative. Let $X \ge 0$ and $Y \ge 0$, and round as before: $X_n = \lfloor nX\rfloor/n$ and $Y_n = \lfloor nY\rfloor/n$. These are functions of $X$ and of $Y$, so they are independent, by part 3 of the theorem on independence in Random Variables and Distribution Functions. So are $X_n + 1/n$ and $Y_n + 1/n$. Since $0 \le X_n \le X \le X_n + 1/n$ and likewise for $Y$,

$$X_nY_n \le XY \le \big(X_n + \tfrac1n\big)\big(Y_n + \tfrac1n\big).$$

The outer variables are products of independent discrete variables with finite means, so by Step 1 they have finite means $E[X_n]E[Y_n]$ and $(E[X_n] + \tfrac1n)(E[Y_n] + \tfrac1n)$. By Theorem 2, $XY$ has a finite mean lying between them. As $n \to \infty$, $E[X_n] \to E[X]$ and $E[Y_n] \to E[Y]$ by Step 2 of the proof of Theorem 3, so both bounds tend to $E[X]E[Y]$.

Step 3: the general case. Write $X = X^+ - X^-$ with $X^+ = \max(X, 0)$ and $X^- = \max(-X, 0)$, and the same for $Y$. Both parts are never negative, are at most $\lvert X\rvert$ and so have finite means, and are functions of $X$. Expanding,

$$XY = X^+Y^+ - X^+Y^- - X^-Y^+ + X^-Y^-.$$

Each product is of a function of $X$ and a function of $Y$, which are independent and never negative, so Step 2 applies to each. By linearity, $XY$ has a finite mean and

$$E[XY] = E[X^+]E[Y^+] - E[X^+]E[Y^-] - E[X^-]E[Y^+] + E[X^-]E[Y^-] = \big(E[X^+] - E[X^-]\big)\big(E[Y^+] - E[Y^-]\big) = E[X]\,E[Y].$$

The first consequence is the theorem applied to the independent variables $g(X)$ and $h(Y)$. For $n$ variables, $\prod_{i < n} g_i(X_i)$ is a function of $(X_1, \ldots, X_{n-1})$ and so independent of $g_n(X_n)$. (That a function of one group of independent variables is independent of a function of the rest is the grouping fact cited in Random Variables and Distribution Functions.) The theorem peels off the last factor, and induction on $n$ finishes. $\square$

Variance

The mean says where a distribution is centered. The variance says how far from the center the variable tends to fall.

$X$ has a finite variance if $E[X^2] < \infty$. Then $X$ has a finite mean, because $\lvert X\rvert \le 1 + X^2$ and Theorem 2 gives $E\lvert X\rvert \le 1 + E[X^2]$. With $\mu = E[X]$, the variance and the standard deviation are

$$\operatorname{Var}(X) = \sigma^2 = E\big[(X - \mu)^2\big], \qquad \sigma = \sqrt{\operatorname{Var}(X)}.$$

Expanding the square and using linearity gives the computing formula and the effect of a change of scale:

$$\operatorname{Var}(X) = E[X^2] - 2\mu E[X] + \mu^2 = E[X^2] - \mu^2, \qquad \operatorname{Var}(aX + b) = E\big[(aX - a\mu)^2\big] = a^2\operatorname{Var}(X).$$

These are the formulas of the course page, and by Theorems 2 and 3 they now hold for every variable with a finite variance. By Theorem 4, $E[X^2]$ is $\sum_k x_k^2\,p(x_k)$ or $\int x^2 f(x)\,dx$.

The mean is the center in a precise sense. For any constant $c$, write $X - c = (X - \mu) + (\mu - c)$ and expand:

$$E\big[(X - c)^2\big] = \operatorname{Var}(X) + 2(\mu - c)\,E[X - \mu] + (\mu - c)^2 = \operatorname{Var}(X) + (\mu - c)^2.$$

The mean squared distance from $c$ is the variance plus the squared distance from $c$ to the mean. It is smallest at $c = \mu$, and the minimum is the variance.

The median is the center in another sense. A median of $X$ is a number $m$ with $P(X \le m) \ge \tfrac12$ and $P(X \ge m) \ge \tfrac12$. The quantile $Q(\tfrac12)$ is one, by the inequality $P(X < Q(u)) \le u \le P(X \le Q(u))$ of Random Variables and Distribution Functions.

Proposition (the median minimizes the mean absolute distance). If $X$ has a finite mean and $m$ is a median, then $E\lvert X - m\rvert \le E\lvert X - c\rvert$ for every $c$.

Proof. Let $c > m$. If $X \le m$, then $\lvert X - c\rvert - \lvert X - m\rvert = (c - X) - (m - X) = c - m$. If $X > m$, the triangle inequality gives $\lvert X - c\rvert - \lvert X - m\rvert \ge -(c - m)$. So for every outcome

$$\lvert X - c\rvert - \lvert X - m\rvert \ge (c - m)\big(\mathbf{1}\{X \le m\} - \mathbf{1}\{X > m\}\big),$$

and taking expected values, $E\lvert X - c\rvert - E\lvert X - m\rvert \ge (c - m)\big(2P(X \le m) - 1\big) \ge 0$.

Let $c < m$. If $X \ge m$, then $\lvert X - c\rvert - \lvert X - m\rvert = (X - c) - (X - m) = m - c$. If $X < m$, the triangle inequality gives $\lvert X - c\rvert - \lvert X - m\rvert \ge -(m - c)$. So $\lvert X - c\rvert - \lvert X - m\rvert \ge (m - c)\big(\mathbf{1}\{X \ge m\} - \mathbf{1}\{X < m\}\big)$, and $E\lvert X - c\rvert - E\lvert X - m\rvert \ge (m - c)\big(2P(X \ge m) - 1\big) \ge 0$. $\square$

The two centers on the data. The median pressure is 123.33 mm Hg. The mean absolute distance from it is 15.24, against 15.47 from the mean. The mean squared distance from the median is 414.16, which is the variance 405.01 plus the squared distance between the two centers, 9.16 (all three from unrounded values). For C-reactive protein in the 5,146 adults with a result, the two centers are far apart: the mean is 4.17 mg/L and the median 1.94 (National Center for Health Statistics, 2020b). Only 27.15% of the adults are above the mean, which a few very large values pull upward. The mean absolute distance is 3.23 from the median and 3.81 from the mean. Which center to report depends on which distance matters.

One more property closes the list. $\operatorname{Var}(X) = 0$ exactly when $X$ equals its mean with probability 1. If $P(X = \mu) = 1$ then $(X - \mu)^2$ is 0 with probability 1, and its expected value $\int_0^\infty P((X - \mu)^2 > t)\,dt$ is 0. Conversely, $(X - \mu)^2$ is never negative, and a variable that is never negative and has mean zero is zero with probability 1, a lemma proved in Probability Inequalities.

Covariance

Variance describes one variable. For two variables on the same experiment the question is how they move together.

Let $X$ and $Y$ have finite variances, with means $\mu_X$ and $\mu_Y$. The product $XY$ has a finite mean, because $\lvert XY\rvert \le (X^2 + Y^2)/2$. The covariance of $X$ and $Y$ is

$$\operatorname{Cov}(X, Y) = E\big[(X - \mu_X)(Y - \mu_Y)\big] = E[XY] - \mu_X\mu_Y,$$

where the second form comes from expanding the product and using linearity. It is positive when the variables tend to be above their means together, and negative when one tends to be above while the other is below. For age and pressure, $E[XY]$ is 6,483.33 and $\mu_X\mu_Y$ is 6,301.33, so the covariance is 182.01, in years times mm Hg.

Theorem 6 (properties of covariance). For variables with finite variances and constants $a$, $b$, $a_i$, $b_j$:

  1. $\operatorname{Cov}(X, Y) = \operatorname{Cov}(Y, X)$ and $\operatorname{Cov}(X, X) = \operatorname{Var}(X)$.
  2. $\operatorname{Cov}(aX + b,\ Y) = a\operatorname{Cov}(X, Y)$ and $\operatorname{Cov}(X + Z,\ Y) = \operatorname{Cov}(X, Y) + \operatorname{Cov}(Z, Y)$.
  3. $\operatorname{Cov}\big(\sum_i a_iX_i,\ \sum_j b_jY_j\big) = \sum_i\sum_j a_ib_j\operatorname{Cov}(X_i, Y_j)$.
  4. $\operatorname{Var}\big(\sum_i a_iX_i\big) = \sum_i\sum_j a_ia_j\operatorname{Cov}(X_i, X_j) = \sum_i a_i^2\operatorname{Var}(X_i) + 2\sum_{i < j} a_ia_j\operatorname{Cov}(X_i, X_j)$.
  5. If $X$ and $Y$ are independent, then $\operatorname{Cov}(X, Y) = 0$.

Proof. (1) Both are immediate from the definition. (2) The mean of $aX + b$ is $a\mu_X + b$, so its deviation from its mean is $a(X - \mu_X)$, and $E[a(X - \mu_X)(Y - \mu_Y)] = a\operatorname{Cov}(X, Y)$. The deviation of $X + Z$ from its mean is $(X - \mu_X) + (Z - \mu_Z)$, and linearity splits the expected product. (3) Apply part 2 repeatedly in the first argument, and then, by the symmetry of part 1, in the second. (4) Take $Y_j = X_j$ and $b_j = a_j$ in part 3 and use $\operatorname{Cov}(X, X) = \operatorname{Var}(X)$. The terms with $i = j$ are the variances, and each pair $i \ne j$ appears twice. (5) By Theorem 5, $E[XY] = \mu_X\mu_Y$. $\square$

Part 4 is the formula the course page proves on a finite sample space, with coefficients added. When every pair is uncorrelated, in particular when the variables are independent, the variance of a sum is the sum of the variances. Part 5 has no converse: Random Variables, Expectation and Variance gives a discrete pair with covariance zero in which one variable is a function of the other. A continuous one is $X$ uniform on $(-1, 1)$ with $Y = X^2$, where $\operatorname{Cov}(X, Y) = E[X^3] - E[X]\,E[X^2] = 0 - 0 = 0$ by symmetry.

The covariance matrix. For variables $X_1, \ldots, X_n$, collect the covariances in the $n \times n$ matrix $\Sigma$ with entries $\Sigma_{ij} = \operatorname{Cov}(X_i, X_j)$. It is symmetric with the variances on its diagonal. Part 4 says that the variance of the linear combination with coefficient vector $a$ is the quadratic form

$$\operatorname{Var}\Big(\sum_i a_iX_i\Big) = a^{\mathsf T}\Sigma\,a \ge 0,$$

so a covariance matrix is positive semidefinite. More generally part 3 gives the covariance matrix of the transformed vector $AX$ as $A\Sigma A^{\mathsf T}$. Multivariate Statistics builds on these two facts.

The three readings. NHANES takes three consecutive readings, and 4,730 of the adults have all three. Their means are 125.74, 125.40 and 124.96 mm Hg, and their covariance matrix is

$$\Sigma = \begin{pmatrix} 387.28 & 373.17 & 365.34 \\ 373.17 & 385.21 & 368.64 \\ 365.34 & 368.64 & 374.94 \end{pmatrix}.$$

The covariances are nearly as large as the variances: a person's readings rise and fall together. Two consequences follow from part 4. The mean of the three readings has coefficients $(\tfrac13, \tfrac13, \tfrac13)$, so its variance is the sum of the nine entries divided by 9, $3{,}361.73/9 = 373.53$. Averaging three independent readings with these variances would give $382.48/3 = 127.49$. Averaging a person's three readings removes little of the variance, because most of it is variation between people. The difference between the first and third readings has coefficients $(1, 0, -1)$ and variance $387.28 + 374.94 - 2(365.34) = 31.54$, against 762.21 for two unrelated adults (both from unrounded entries). That reduction is what The Paired-Samples t-Test exploits. Problem 3 completes the calculation.

Correlation and its bounds

A covariance has units, and its size means nothing until it is compared with the spreads of the two variables. Age in months and pressure in kilopascals have covariance 291.19 in place of 182.01, for the same people.

For variables with finite and positive variances, the correlation coefficient is

$$\rho = \operatorname{Corr}(X, Y) = \frac{\operatorname{Cov}(X, Y)}{\sigma_X\,\sigma_Y}.$$

It is the covariance of the standardized variables $X^* = (X - \mu_X)/\sigma_X$ and $Y^* = (Y - \mu_Y)/\sigma_Y$, each with mean 0 and variance 1, by part 2 of Theorem 6. For age and pressure, $\rho = 182.01/(18.59 \times 20.12) = 0.4866$.

Theorem 7 (bounds on the correlation).

  1. $-1 \le \rho \le 1$.
  2. $\rho = 1$ exactly when $Y = \mu_Y + \frac{\sigma_Y}{\sigma_X}(X - \mu_X)$ with probability 1, and $\rho = -1$ exactly when $Y = \mu_Y - \frac{\sigma_Y}{\sigma_X}(X - \mu_X)$ with probability 1.
  3. $\operatorname{Corr}(aX + b,\ cY + d) = \rho$ if $ac > 0$ and $-\rho$ if $ac < 0$.

The idea: the sum and the difference of the two standardized variables have variances $2(1 + \rho)$ and $2(1 - \rho)$, and a variance cannot be negative.

Proof. (1) By part 4 of Theorem 6, with $\operatorname{Var}(X^*) = \operatorname{Var}(Y^*) = 1$ and $\operatorname{Cov}(X^*, Y^*) = \rho$,

$$\operatorname{Var}(X^* + Y^*) = 2 + 2\rho \ge 0, \qquad \operatorname{Var}(X^* - Y^*) = 2 - 2\rho \ge 0.$$

The first gives $\rho \ge -1$ and the second $\rho \le 1$.

(2) $\rho = 1$ exactly when $\operatorname{Var}(X^* - Y^*) = 0$. A variance is zero exactly when the variable equals its mean with probability 1, and the mean of $X^* - Y^*$ is 0. So $\rho = 1$ exactly when $Y^* = X^*$ with probability 1, which is the stated line. Likewise $\rho = -1$ exactly when $\operatorname{Var}(X^* + Y^*) = 0$, that is, when $Y^* = -X^*$ with probability 1.

(3) The standardized form of $aX + b$ is $X^*$ if $a > 0$ and $-X^*$ if $a < 0$, since $aX + b$ has mean $a\mu_X + b$ and standard deviation $\lvert a\rvert\sigma_X$. The same holds for $cY + d$. So the correlation is $\operatorname{Cov}(\pm X^*, \pm Y^*)$, which is $\rho$ when the signs agree and $-\rho$ when they differ. $\square$

For age and pressure the two variances are $2(1 + \rho) = 2.9732$ and $2(1 - \rho) = 1.0268$, computed directly from the standardized values. Part 1 is also a consequence of the Cauchy–Schwarz inequality, $(E[UV])^2 \le E[U^2]\,E[V^2]$, applied to the deviations; Probability Inequalities proves that inequality with its case of equality. Casella and Berger (2002) treat covariance and correlation in their Section 4.5.

What correlation measures: the best straight line

Correlation is a measure of linear association, and the next theorem says exactly what that means.

Theorem 8 (the best linear predictor). Let $X$ and $Y$ have finite variances with $\sigma_X > 0$ and $\sigma_Y > 0$. Among all constants $a$ and $b$, the mean squared error $E[(Y - a - bX)^2]$ is smallest for

$$b^* = \frac{\operatorname{Cov}(X, Y)}{\operatorname{Var}(X)} = \rho\,\frac{\sigma_Y}{\sigma_X}, \qquad a^* = \mu_Y - b^*\mu_X,$$

and only for them. The minimum is $\sigma_Y^2(1 - \rho^2)$, and the error $Y - a^* - b^*X$ has mean 0 and covariance 0 with $X$.

The idea is to choose the intercept first, which is the problem of the best constant, and then complete the square in the slope.

Proof. Fix $b$ and let $W = Y - bX$. By the identity for the mean squared distance from a constant, $E[(W - a)^2] = \operatorname{Var}(W) + (E[W] - a)^2$, which is smallest when $a = E[W] = \mu_Y - b\mu_X$, and then equals $\operatorname{Var}(Y - bX)$. By part 4 of Theorem 6,

$$\operatorname{Var}(Y - bX) = \sigma_Y^2 - 2b\operatorname{Cov}(X, Y) + b^2\sigma_X^2 = \sigma_X^2\Big(b - \frac{\operatorname{Cov}(X, Y)}{\sigma_X^2}\Big)^2 + \sigma_Y^2 - \frac{\operatorname{Cov}(X, Y)^2}{\sigma_X^2}.$$

The square vanishes only at $b = b^*$, and what remains is $\sigma_Y^2 - \rho^2\sigma_Y^2$. The error has mean $\mu_Y - a^* - b^*\mu_X = 0$, and $\operatorname{Cov}(Y - a^* - b^*X,\ X) = \operatorname{Cov}(Y, X) - b^*\operatorname{Var}(X) = 0$. $\square$

So $\rho^2$ is the share of the variance of $Y$ that the best straight line in $X$ removes. For pressure on age, $b^* = 182.01/345.45 = 0.5269$ mm Hg per year and $a^* = 126.36 - 0.5269 \times 49.87 = 100.09$. The line predicts 115.89 at age 30 and 136.97 at age 70. Its mean squared error is $405.01 \times (1 - 0.2368) = 309.11$ from unrounded values, a root mean squared error of 17.58 mm Hg against the standard deviation 20.12: age removes 23.68% of the variance. The same formulas with sample moments in place of population moments are the least squares line of Simple Linear Regression. Whether a curve could do better than the line is the question Conditional Expectation and the Laws of Total Expectation and Variance answers.

Example (zero correlation is not independence). "The correlation is zero, so the variables are unrelated." Correlation zero means only that the best straight line is flat: $b^* = 0$. For $X$ uniform on $(-1, 1)$ and $Y = X^2$ the correlation is 0, and $Y$ is completely determined by $X$.

How large a correlation can be for given margins

The bounds $\pm 1$ are reached only when one variable is a linear function of the other. If the two distributions have different shapes, no such relation is possible, and the attainable range is narrower. For indicators it can be computed.

Theorem 9 (the correlation of two indicators). Let $A$ and $B$ be events with $P(A) = p$ and $P(B) = q$, where $0 < p \le q < 1$. Then

$$\max(0,\ p + q - 1) \le P(A \cap B) \le p,$$

$\operatorname{Cov}(\mathbf{1}_A, \mathbf{1}_B) = P(A \cap B) - pq$, and the correlation of the two indicators satisfies

$$\rho \le \sqrt{\frac{p(1 - q)}{q(1 - p)}}, \qquad \rho \ge -\sqrt{\frac{pq}{(1 - p)(1 - q)}} \ \text{ if } p + q \le 1, \qquad \rho \ge -\sqrt{\frac{(1 - p)(1 - q)}{pq}} \ \text{ if } p + q \ge 1.$$

The upper bound is 1 only when $p = q$.

Proof. $A \cap B \subseteq A$ gives $P(A \cap B) \le p$. Inclusion–exclusion, from The Rules of Probability, gives $P(A \cap B) = p + q - P(A \cup B) \ge p + q - 1$, and a probability is at least 0. The product $\mathbf{1}_A\mathbf{1}_B$ is the indicator of $A \cap B$, so $E[\mathbf{1}_A\mathbf{1}_B] = P(A \cap B)$ and the covariance is $P(A \cap B) - pq$. An indicator has variance $E[\mathbf{1}_A^2] - p^2 = p(1 - p)$. Dividing the largest covariance, $p - pq = p(1 - q)$, by $\sqrt{p(1 - p)\,q(1 - q)}$ gives the upper bound. The smallest covariance is $-pq$ when $p + q \le 1$ and $p + q - 1 - pq = -(1 - p)(1 - q)$ when $p + q \ge 1$, and dividing by the same square root gives the two lower bounds. The upper bound equals 1 exactly when $p(1 - q) = q(1 - p)$, that is, $p = q$. $\square$

The bounds are attained. With $U$ uniform on $(0, 1)$, the events $A = \{U \le p\}$ and $B = \{U \le q\}$ are nested and have $P(A \cap B) = p$, and $A = \{U \le p\}$ with $B = \{U > 1 - q\}$ overlap as little as possible.

Example (a correlation of 0.34 between an event and one that implies it). Among the 4,730 adults, 289 have a first reading at or above 160 mm Hg, a share $p = 0.0611$, and 1,697 have a first reading at or above 130, a share $q = 0.3588$. The first event implies the second, so their dependence could not be stronger, and the correlation of the two indicators is at the upper bound $\sqrt{p(1 - q)/(q(1 - p))}$ for these margins. It is 0.3410. A correlation far from 1 does not show that a relation is weak, only that it is far from a straight line with these margins. Problem 4 applies the theorem to the first and second readings.

Equal correlations and the variance of a mean

Part 4 of Theorem 6 has a special case that the course pages use repeatedly. Let $X_1, \ldots, X_n$ have a common variance $\sigma^2$ and a common correlation $\rho$ between every pair. There are $n$ variances and $n(n - 1)$ ordered pairs with covariance $\rho\sigma^2$, so

$$\operatorname{Var}\Big(\sum_{i=1}^n X_i\Big) = n\sigma^2\big[1 + (n - 1)\rho\big], \qquad \operatorname{Var}(\bar{X}_n) = \frac{\sigma^2}{n}\big[1 + (n - 1)\rho\big].$$

Three things follow.

  • A lower bound on a common correlation. The variance cannot be negative, so $\rho \ge -1/(n - 1)$. Three variables cannot all have pairwise correlation $-0.6$. The bound is attained by sampling without replacement. By symmetry every draw has the population variance $\sigma^2$ and every pair of draws the same correlation: The Sampling Distribution of the Mean shows that each draw has the population distribution, and every ordered pair of draws is equally likely to be any ordered pair of different members. When all $N$ members of a finite population are drawn, the sum is constant, its variance is 0, and so $\rho = -1/(N - 1)$. That is the covariance $-\sigma^2/(N - 1)$ between two draws that the same page derives on the way to the finite population correction.
  • Positive correlation and the mean. With $\rho > 0$ the variance of the mean tends to $\rho\sigma^2$, not to 0, as $n$ grows. Averaging cannot remove a component the observations share.
  • The design effect. The factor $1 + (n - 1)\rho$ is the design effect of The Binomial Distribution, derived there for indicators within a cluster. For the three readings the variances are not quite equal, but the ratio of the variance of their mean to the value under independence, $373.53/127.49 = 2.930$, agrees to three decimals with $1 + 2\bar\rho$ for the average correlation $\bar\rho = 0.9650$.

What the experiment shows

The moments above are population quantities. A sample gives estimates of them, and the rules of this article say which estimates are right on average.

Proposition (the sample covariance is unbiased). Let $(X_1, Y_1), \ldots, (X_n, Y_n)$ be independent pairs with the same distribution (mutually independent vectors, as defined in Random Variables and Distribution Functions), with finite variances and covariance $c = \operatorname{Cov}(X_1, Y_1)$. Then

$$E\Big[\frac{1}{n - 1}\sum_{i=1}^n (X_i - \bar{X}_n)(Y_i - \bar{Y}_n)\Big] = c.$$

Proof. Expanding, $\sum_i (X_i - \bar{X}_n)(Y_i - \bar{Y}_n) = \sum_i X_iY_i - n\bar{X}_n\bar{Y}_n$. Each $E[X_iY_i] = c + \mu_X\mu_Y$. By part 3 of Theorem 6, $\operatorname{Cov}(\bar{X}_n, \bar{Y}_n) = \frac{1}{n^2}\sum_i\sum_j \operatorname{Cov}(X_i, Y_j) = \frac{c}{n}$, since $X_i$ and $Y_j$ are independent for $i \ne j$, being functions of different independent pairs (part 3 of the theorem on independence in that article), and the $n$ terms with $i = j$ each equal $c$. So $E[\bar{X}_n\bar{Y}_n] = c/n + \mu_X\mu_Y$, and the expected sum is $n(c + \mu_X\mu_Y) - n(c/n + \mu_X\mu_Y) = (n - 1)c$. $\square$

With $Y_i = X_i$ this is the unbiasedness of the sample variance with divisor $n - 1$. The script drew 10,000 samples of 25 adults, with replacement, for each of 30 seeds (NumPy's default_rng with seeds 0 to 29) and averaged three statistics of age and pressure over the 10,000 samples.

Statistic, averaged over 10,000 samples of 25 Population value Median over 30 seeds Interquartile range
Sample covariance, divisor $n - 1$ 182.01 182.05 181.75 to 182.49
Sample covariance, divisor $n$ 182.01 174.76 174.48 to 175.19
Sample correlation 0.4866 0.4902 0.4895 to 0.4911

The first row agrees with the proposition. The second is low by the factor $24/25$, which gives 174.73. The third is not covered by any result here: the sample correlation is a ratio, and the expected value of a ratio is not the ratio of expected values. In these runs its average was slightly above $\rho$, and its standard deviation across samples was 0.1470 (interquartile range 0.1463 to 0.1476). These are observations on this population at $n = 25$, not general facts about the sample correlation.

What is proved here, what is cited, and what measure theory would add

Proved on this page: Theorems 1 to 9, the two propositions and the identities for the variance. Theorems 2, 3 and 5 to 8 and the propositions hold for every random variable with the required finite moments; Theorems 1 and 4 concern discrete variables and densities. Theorem 1 part 2 and Theorem 4 parts 2 and 3 use the exchange of the order of integration for nonnegative integrands.

Taken as given or cited:

  • That exchange (Fubini's theorem; Billingsley, 1995, Section 18), in Theorem 1 part 2 and Theorem 4 parts 2 and 3.
  • That the probability that a pair with a joint density falls in a region bounded by finitely many smooth curves is the double integral of the density over the region, and the one-variable form for finite unions of intervals. Theorem 4 parts 2 and 3 use it.
  • The free grouping and reordering of absolutely convergent series and double series, in Theorem 1 part 1, Theorem 3 Step 1, Theorem 4 part 1 and Theorem 5 Step 1.
  • The grouping fact for independent variables, in the product rule for $n$ factors, and the definition of independent vectors in the proposition on the sample covariance. Both are in Random Variables and Distribution Functions.
  • As throughout the unit, that the sets in question are events.

The lemma that a nonnegative variable with mean zero vanishes with probability 1 is proved in Probability Inequalities.

What measure theory would add. The general theory defines $E[X]$ as the integral of the function $X$ over the sample space with respect to $P$ (Billingsley, 1995, Section 21, under "Expected Value as Integral"). The definition used here needs no such integral, and the rules proved from it are the ones every later article of the unit uses. What the general integral adds is a set of theorems about limits: when $E[X_n] \to E[X]$ follows from $X_n \to X$, when an infinite sum may be taken outside an expected value, and when a derivative may. Those are the monotone and dominated convergence theorems (Billingsley, 1995, Section 16, under "Integration to the Limit"). This article proves no such theorem. Where a later article of the unit needs one, for a moment generating function differentiated under the expectation or for a sum of infinitely many terms, it says so and cites it.

Worked problems

These problems are posed here, on this unit's NHANES data and on small exact cases; none is from a textbook. The script tools/experiments/statistics/expectation-variance-and-covariance.py in the site's repository checks every number. Decimals are rounded from unrounded values.

Problem 1 (the mean and variance of a measurement with a detection limit). A concentration $Y$ has the density $e^{-y}$ on $y > 0$, and an assay reports $R = \max(Y, \tfrac12)$. Find $E[R]$ and $\operatorname{Var}(R)$, the mean both from the general definition and as a sum over the atom plus an integral over the continuous part.

Solution. $R$ is never negative, so $E[R] = \int_0^\infty P(R > t)\,dt$. For $t < \tfrac12$ the event $\{R > t\}$ is certain, and for $t \ge \tfrac12$ it is $\{Y > t\}$, with probability $e^{-t}$. So

$$E[R] = \int_0^{1/2} 1\,dt + \int_{1/2}^\infty e^{-t}\,dt = \tfrac12 + e^{-1/2} = 1.1065.$$

Piece by piece: $R$ equals $\tfrac12$ with probability $1 - e^{-1/2} = 0.3935$, and otherwise has the density $e^{-x}$ on $x > \tfrac12$. The atom contributes $\tfrac12 \times 0.3935 = 0.1967$. The continuous part contributes $\int_{1/2}^\infty x e^{-x}\,dx = \big[-(x + 1)e^{-x}\big]_{1/2}^\infty = \tfrac32 e^{-1/2} = 0.9098$. The total is $0.1967 + 0.9098 = 1.1065$, as the remark after Theorem 1 requires. The limit raises the mean from $E[Y] = 1$, since values below $\tfrac12$ are reported as $\tfrac12$.

For the variance we need $E[R^2]$. The variable $R^2$ is never negative, and $\{R^2 > t\}$ is certain for $t < \tfrac14$ and is $\{Y > \sqrt t\}$ for $t \ge \tfrac14$. Substituting $t = s^2$,

$$E[R^2] = \tfrac14 + \int_{1/4}^\infty e^{-\sqrt t}\,dt = \tfrac14 + \int_{1/2}^\infty 2s\,e^{-s}\,ds = \tfrac14 + 2\cdot\tfrac32 e^{-1/2} = \tfrac14 + 3e^{-1/2} = 2.0696.$$

So $\operatorname{Var}(R) = 2.0696 - 1.1065^2 = 2.0696 - 1.2244 = 0.8452$, below $\operatorname{Var}(Y) = 1$: the limit removes the variation among the small values.

Problem 2 (a variable with no finite mean). Let $X$ take the value $2^k$ with probability $2^{-k}$, for $k = 1, 2, \ldots$ (a) Find $E[X]$. (b) Find $E[\min(X, 2^m)]$. (c) Let $Y$ take the value $(-2)^k$ with probability $2^{-k}$. Does $Y$ have a mean?

Solution. (a) The probabilities add to $\tfrac12 + \tfrac14 + \cdots = 1$. Each term of the sum is $2^k \cdot 2^{-k} = 1$, so $E[X] = 1 + 1 + \cdots = +\infty$. The variable is never negative, so its expected value is defined, and it is infinite.

(b) $\min(X, 2^m)$ equals $2^k$ with probability $2^{-k}$ for $k < m$, and equals $2^m$ when $k \ge m$, which has probability $2^{-m} + 2^{-m-1} + \cdots = 2^{-(m-1)}$. So

$$E[\min(X, 2^m)] = \sum_{k=1}^{m-1} 1 + 2^m \cdot 2^{-(m-1)} = (m - 1) + 2 = m + 1.$$

Capping the variable at about a million ($m = 20$) gives a mean of only 21. The mean of the uncapped variable is infinite, but only barely: it grows like the logarithm of the cap.

(c) The positive values of $Y$ are $2^k$ for even $k$ and the negative ones $-2^k$ for odd $k$. By Theorem 1, $E^+ = \sum_{k \text{ even}} 1 = \infty$ and $E^- = \sum_{k \text{ odd}} 1 = \infty$. Neither part is finite, so $Y$ has no mean, finite or infinite. The series $\sum_k (-2)^k 2^{-k} = -1 + 1 - 1 + \cdots$ has partial sums that alternate between $-1$ and 0 and settles nowhere.

Problem 3 (the covariance matrix of three readings). From the covariance matrix $\Sigma$ of the three readings given above, find (a) the correlation of each pair; (b) the variance of the mean of the three readings and its ratio to the value for uncorrelated readings; (c) the variance of the second difference $R_1 - 2R_2 + R_3$, and say why it could not have been negative.

Solution. (a) Each correlation is a covariance divided by the two standard deviations: $373.17/\sqrt{387.28 \times 385.21} = 0.9662$ for readings 1 and 2, $365.34/\sqrt{387.28 \times 374.94} = 0.9587$ for readings 1 and 3, and $368.64/\sqrt{385.21 \times 374.94} = 0.9700$ for readings 2 and 3. Their average is 0.9650. Adjacent readings are slightly more alike than the first and third.

(b) With coefficients $(\tfrac13, \tfrac13, \tfrac13)$, part 4 of Theorem 6 gives $\frac19$ times the sum of all nine entries. The diagonal adds to $387.28 + 385.21 + 374.94 = 1{,}147.43$ and the six off-diagonal entries to $2(373.17 + 365.34 + 368.64) = 2{,}214.30$, a total of 3,361.73, so the variance of the mean is 373.53. For uncorrelated readings only the diagonal would count: $1{,}147.43/9 = 127.49$. The ratio is 2.930. The script confirms 373.53 by computing the variance of each adult's mean reading directly.

(c) With coefficients $a = (1, -2, 1)$,

$$a^{\mathsf T}\Sigma\,a = 387.28 + 4(385.21) + 374.94 + 2\big[(1)(-2)(373.17) + (1)(1)(365.34) + (-2)(1)(368.64)\big] = 2{,}303.074 - 2{,}236.576 = 66.498,$$

where the two totals are computed from the unrounded entries and shown to three places, so that the subtraction can be checked. To two places the variance is 66.50. (The rounded entries shown give 2,303.06 and 2,236.56.)

It could not have been negative because it is the variance of a random variable, $R_1 - 2R_2 + R_3$. The same holds for every coefficient vector, which is the statement that $\Sigma$ is positive semidefinite. The large covariances nearly cancel the variances here, because the second difference removes both the person's level and any steady drift across the readings.

Problem 4 (how large could the correlation of two indicators be?). Of the 4,730 adults, 1,697 have a first reading at or above 130 mm Hg, 1,709 have a second reading at or above 130, and 1,525 have both. (a) Find the correlation of the two indicators. (b) Find the largest and smallest values it could take for these two margins.

Solution. (a) $p = 1{,}697/4{,}730 = 0.3588$, $q = 1{,}709/4{,}730 = 0.3613$ and $P(A \cap B) = 1{,}525/4{,}730 = 0.3224$. The covariance is $0.3224 - 0.3588 \times 0.3613 = 0.1928$, the value found on the course page. Dividing by $\sqrt{p(1 - p)\,q(1 - q)}$ gives $\rho = 0.8367$.

(b) By Theorem 9, $P(A \cap B)$ can be at most $p = 0.3588$, a covariance of $p(1 - q) = 0.2291$ and a correlation of $\sqrt{p(1 - q)/(q(1 - p))} = 0.9945$. Since $p + q = 0.7201 \le 1$, the events could be disjoint, a covariance of $-pq$, which is −0.1296, and a correlation of $-\sqrt{pq/((1 - p)(1 - q))}$, which is −0.5626. So for these margins the correlation must lie between −0.5626 and 0.9945. The upper limit is below 1 because the two readings are high for slightly different shares of adults, and the lower limit is far from $-1$ because two events that each have probability near 0.36 cannot be complementary. The observed 0.8367 is 84% of the way from 0 to the largest value possible.

Problem 5 (the best line through a four-point population). A population has four members with $(x, y)$ equal to $(1, 2)$, $(2, 1)$, $(4, 5)$ and $(5, 8)$, and one is drawn at random. Find the covariance, the correlation, the best linear predictor of $Y$ from $X$ and its mean squared error, and check that the error is uncorrelated with $X$.

Solution. Each point has probability $\tfrac14$. The means are $\mu_X = 12/4 = 3$ and $\mu_Y = 16/4 = 4$. The deviations are $(-2, -2)$, $(-1, -3)$, $(1, 1)$ and $(2, 4)$, so

$$\operatorname{Var}(X) = \frac{4 + 1 + 1 + 4}{4} = \frac52, \qquad \operatorname{Var}(Y) = \frac{4 + 9 + 1 + 16}{4} = \frac{15}{2}, \qquad \operatorname{Cov}(X, Y) = \frac{4 + 3 + 1 + 8}{4} = 4.$$

The correlation is $4/\sqrt{\tfrac52 \cdot \tfrac{15}{2}} = 4/\sqrt{18.75} = 0.9238$. By Theorem 8 the slope is $b^* = 4/\tfrac52 = \tfrac85$ and the intercept $a^* = 4 - \tfrac85 \cdot 3 = -\tfrac45$. The minimum mean squared error is $\operatorname{Var}(Y) - \operatorname{Cov}^2/\operatorname{Var}(X) = \tfrac{15}{2} - \tfrac{16}{5/2} = \tfrac{15}{2} - \tfrac{32}{5} = \tfrac{11}{10}$.

The check: the predictions $-\tfrac45 + \tfrac85 x$ at $x = 1, 2, 4, 5$ are $0.8$, $2.4$, $5.6$ and $7.2$, and the errors are $1.2$, $-1.4$, $-0.6$ and $0.8$. They add to 0. Their squares, $1.44$, $1.96$, $0.36$ and $0.64$, add to $4.4$, a mean of $1.1 = \tfrac{11}{10}$. And the covariance of the error with $X$ is $\tfrac14\big[(-2)(1.2) + (-1)(-1.4) + (1)(-0.6) + (2)(0.8)\big] = \tfrac14(-2.4 + 1.4 - 0.6 + 1.6) = 0$. The script also draws every one of the $4^3 = 64$ ordered samples of three with replacement and finds that the sample covariance with divisor $n - 1$ averages exactly 4, as the proposition on the sample covariance requires.

Summary

Quantity Definition or rule
Expected value $E[X] = \int_0^\infty P(X > t)\,dt - \int_0^\infty P(X < -t)\,dt$; equals $\sum x\,p(x)$ or $\int x f(x)\,dx$
Finite mean $E\lvert X\rvert < \infty$
Linearity $E[aX + bY + c] = aE[X] + bE[Y] + c$, with no independence
Function $E[g(X)] = \sum g(x)\,p(x)$ or $\int g(x) f(x)\,dx$
Independence $E[g(X)h(Y)] = E[g(X)]\,E[h(Y)]$
Variance $E[(X - \mu)^2] = E[X^2] - \mu^2$; $E[(X - c)^2] = \operatorname{Var}(X) + (\mu - c)^2$
Covariance $E[(X - \mu_X)(Y - \mu_Y)] = E[XY] - \mu_X\mu_Y$; bilinear
Linear combination $\operatorname{Var}(\sum a_iX_i) = \sum_i\sum_j a_ia_j\operatorname{Cov}(X_i, X_j) = a^{\mathsf T}\Sigma a$
Correlation $\rho = \operatorname{Cov}(X, Y)/(\sigma_X\sigma_Y)$, $-1 \le \rho \le 1$, with $\pm 1$ only on a line
Best line slope $\rho\sigma_Y/\sigma_X$, through the means, error variance $\sigma_Y^2(1 - \rho^2)$
Equal correlations $\operatorname{Var}(\bar{X}_n) = \frac{\sigma^2}{n}[1 + (n - 1)\rho]$, so $\rho \ge -\frac{1}{n - 1}$

What we have learned

The expected value is the area between a distribution function and the level 1 to the right of zero, minus the area under it to the left. That one formula gives the sum for a discrete variable and the integral for a density, covers a measurement with a detection limit, and refuses a Cauchy variable, whose two areas are both infinite. Linearity and the product rule for independent variables hold for all random variables, because every variable is within $1/n$ of a discrete one. Variances and covariances are expected values of squares and products, so the variance of any linear combination is a quadratic form in the covariance matrix. The correlation is confined to $[-1, 1]$ because the standardized sum and difference have variances $2(1 \pm \rho)$. Its square is the share of variance a straight line removes: 23.68% for systolic pressure on age.

A straight line is the best predictor only among straight lines. The next article, Conditional Expectation and the Laws of Total Expectation and Variance, finds the best predictor of any form, the conditional expectation, and proves the two laws that split a mean and a variance according to it.

The numbers in this article are computed and checked by tools/experiments/statistics/expectation-variance-and-covariance.py in the site's repository, from the NHANES files loaded by tools/experiments/applied-statistics/statdata.py.

References

  • Billingsley, P. (1995). Probability and Measure (3rd ed.). Wiley. Sections 16 ("Properties of the Integral"), 18 ("Product Measure and Fubini's Theorem") and 21 ("Expected Values"). Section numbers and titles are those of the contents of the 2012 Anniversary Edition, which reissues the third edition. record · 8 pages
  • Casella, G., and Berger, R. L. (2002). Statistical Inference (2nd ed.). Duxbury. Sections 2.2 ("Expected Values") and 4.5 ("Covariance and Correlation"). record · 11 pages
  • Grinstead, C. M., and Snell, J. L. (1997). Introduction to Probability (2nd rev. ed.). American Mathematical Society. Section 6.1 (Definition 6.1), Section 6.3 (Definition 6.4, Theorem 6.11). record · 20 pages
  • National Center for Health Statistics (2020a). National Health and Nutrition Examination Survey 2017–2018: Demographic Variables and Sample Weights (DEMO_J) and Blood Pressure (BPX_J) data files. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 67 pages
  • National Center for Health Statistics (2020b). National Health and Nutrition Examination Survey 2017–2018 Data Documentation: High-Sensitivity C-Reactive Protein (HSCRP_J). Centers for Disease Control and Prevention. https://wwwn.cdc.gov/Nchs/Data/Nhanes/Public/2017/DataFiles/HSCRP_J.htm record · 3 pages