Random Variables, Expectation and Variance

The language the theoretical distributions are written in. Random variables, probability mass functions and distribution functions, on the number of an adult's three blood pressure readings that are high; the expected value and the variance, with linearity of expectation, the variance of a sum with its covariances and the additivity of variances for independent variables proved; and conditional expectation, with the laws of total expectation and total variance proved.

After a participant in the National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018 has rested for five minutes, the examiner takes three consecutive blood pressure readings. Pick one of the examined adults at random and count how many of the three readings are 130 mm Hg or higher. The count is 0, 1, 2 or 3, and which of them it is depends on who was picked. How can a number that depends on chance be described before it is observed? By its distribution, the list of its possible values with their probabilities, and by two summaries of that list: the expected value, which says where the distribution is centered, and the variance, which says how spread out it is. For this count they are 1.0677 and 1.8360.

A number determined by the outcome of a random experiment is called a random variable. This article is where the course defines random variables, expected values and variances. Later articles, from The Binomial Distribution and The Poisson Distribution to The Sampling Distribution of the Mean, use the rules proved here.

First we define random variables, probability mass functions and distribution functions. Then we define the expected value and the variance, and prove the rules for computing them, including the variance of a sum with its covariances. Finally we define conditional expectation and prove the laws of total expectation and total variance. The article assumes The Rules of Probability and, for its last section, Conditional Probability and the Law of Total Probability.

Random variables

A random variable is a function that assigns a number to each outcome of a random experiment. Drawing one adult from NHANES is the experiment. The adult's body mass index, the number of household members, and an indicator that is 1 for obesity and 0 otherwise are all random variables defined on it. Random variables are written with capital letters, $X$, $Y$, $K$, and the values they take with lower-case letters.

A random variable is discrete when its possible values can be listed, usually as whole numbers: counts of events, of people, of positive tests. It is continuous when it can take any value in an interval, as a measurement such as blood pressure or height can, at least in principle. The two kinds need different mathematical tools: sums for discrete variables and integrals for continuous ones. This article treats discrete variables, and The Normal Distribution treats continuous ones.

The probability mass function

A first example comes from the way NHANES measures blood pressure. After the participant has rested quietly for five minutes, the examiner takes three consecutive readings (National Center for Health Statistics, 2020, BPX_J documentation). Of the 5,247 examined adults with at least one systolic reading, 4,730 have all three. Pick one of them at random and let $K$ be the number of the three readings that are 130 mm Hg or higher. $K$ takes the values 0, 1, 2 and 3. Its probabilities are the shares of adults with each count:

$k$ Adults $P(K = k)$ $F(k) = P(K \le k)$
0 2,778 0.5873 0.5873
1 269 0.0569 0.6442
2 268 0.0567 0.7008
3 1,415 0.2992 1.0000

The function $p(k) = P(K = k)$ is the probability mass function (pmf) of $K$. Every pmf has two properties, which follow directly from the axioms of The Rules of Probability. Each value is between 0 and 1. And the values add to 1, because the events $\{K = 0\}, \{K = 1\}, \ldots$ are mutually exclusive and together make up the whole sample space.

The distribution function

The cumulative distribution function (CDF) of a random variable $X$ is

$$F(x) = P(X \le x),$$

defined for every real number $x$, not only the values $X$ takes. For a discrete variable it is the running total of the pmf, as in the last column above. Its graph is a staircase that jumps by $p(k)$ at each value $k$ and is flat between them.

Every CDF rises from 0 at the far left to 1 at the far right and never decreases. The reason is the monotonicity rule: the event $\{X \le a\}$ is contained in $\{X \le b\}$ when $a < b$. Probabilities of ranges come from differences: $P(a < X \le b) = F(b) - F(a)$. For $K$, $P(1 \le K \le 2) = F(2) - F(0) = 0.7008 - 0.5873 = 0.1135$, which agrees with $0.0569 + 0.0567$ up to rounding.

The same table could have come from a model rather than from counting. That is the point of a theoretical distribution: a formula for $p(k)$, with one or two parameters, that describes many different situations at once. Before building one, we need the two numbers that summarize any distribution.

Expectation and variance

The expected value

The expected value, or mean, of a discrete random variable is the average of its values weighted by their probabilities:

$$E[X] = \mu = \sum_x x\, p(x).$$

For $K$,

$$E[K] = 0(0.5873) + 1(0.0569) + 2(0.0567) + 3(0.2992) = 1.0677,$$

computed with the unrounded shares (the rounded ones give 1.0679).

When the probabilities are shares of a finite population, as here, $E[K]$ is simply the population mean of $K$: the average adult has 1.0677 of three readings at or above 130. In the relative-frequency view of The Rules of Probability, it is the value the average of many independent draws of $K$ settles toward.

Example (an expected value that cannot occur). The expected value 1.0677 is a long-run average, not a typical value: no adult can have 1.0677 high readings. The most common value of $K$ is 0, and 1 is one of the rarest. The expected value is the balance point of the distribution, and it need not be a value the variable can take.

Three rules for expected values are used constantly. For constants $a$ and $b$, a function $g$, and random variables $X$ and $Y$ on the same experiment,

$$E[aX + b] = aE[X] + b, \qquad E[X + Y] = E[X] + E[Y], \qquad E[g(X)] = \sum_x g(x)\,p(x).$$

The idea of the proof is that an expected value can be computed outcome by outcome, and sums can then be split term by term.

Proof. Take a finite sample space with outcomes $\omega$ of probability $P(\omega)$. First, the expected value of any random variable $Z$ can be computed outcome by outcome. Group the outcomes by the value $Z$ gives them. The outcomes with $Z(\omega) = z$ make up the event $\{Z = z\}$, whose probability is $p(z)$, so

$$\sum_\omega Z(\omega) P(\omega) = \sum_z \; \sum_{\omega :\, Z(\omega) = z} z\,P(\omega) = \sum_z z\,p(z) = E[Z].$$

Apply this to $Z = X + Y$:

$$E[X + Y] = \sum_\omega \big(X(\omega) + Y(\omega)\big) P(\omega) = \sum_\omega X(\omega) P(\omega) + \sum_\omega Y(\omega) P(\omega) = E[X] + E[Y].$$

Apply it to $Z = aX + b$:

$$E[aX + b] = \sum_\omega \big(aX(\omega) + b\big)P(\omega) = a\sum_\omega X(\omega)P(\omega) + b\sum_\omega P(\omega) = aE[X] + b,$$

since the probabilities add to 1. Apply it to $Z = g(X)$, and this time group the outcomes by the value of $X$:

$$E[g(X)] = \sum_\omega g\big(X(\omega)\big)P(\omega) = \sum_x \; \sum_{\omega :\, X(\omega) = x} g(x)\,P(\omega) = \sum_x g(x)\,p(x). \qquad \square$$

The second rule, linearity, needs no independence. The expected value of a sum is the sum of the expected values, whatever the relation between the terms. Applied repeatedly, it gives $E[X_1 + \cdots + X_n] = E[X_1] + \cdots + E[X_n]$ for any number of terms. The proof above is for a finite sample space. The same results hold for infinite sample spaces whenever the sums converge; that extension is not proved here and is cited from Grinstead and Snell (1997, §6.1).

The variance

The mean says where a distribution is centered. How spread out is it? The variance is the expected squared distance from the mean,

$$\operatorname{Var}(X) = \sigma^2 = E\big[(X - \mu)^2\big] = \sum_x (x - \mu)^2 p(x),$$

and the standard deviation $\sigma$ is its square root, in the units of $X$. Expanding the square and using linearity gives the computing formula

$$\operatorname{Var}(X) = E[X^2] - 2\mu E[X] + \mu^2 = E[X^2] - \mu^2.$$

For $K$, $E[K^2] = 1(0.0569) + 4(0.0567) + 9(0.2992) = 2.9759$ from the unrounded shares. So $\operatorname{Var}(K) = 2.9759 - 1.0677^2 = 1.8360$, and the standard deviation is 1.3550. These are the population variance and standard deviation of Measures of Center and Spread, with divisor $N$, written for a distribution rather than a list of values.

Adding a constant shifts a distribution without changing its spread. Multiplying by $a$ multiplies distances from the mean by $|a|$:

$$\operatorname{Var}(aX + b) = E\big[(aX + b - a\mu - b)^2\big] = a^2 \operatorname{Var}(X).$$

For sums, independence matters. Two discrete random variables are independent when $P(X = x, Y = y) = P(X = x)\,P(Y = y)$ for all values $x$ and $y$. In words, every event about $X$ is independent, in the sense of The Rules of Probability, of every event about $Y$. For independent variables the expected value of a product factors. Grouping the outcomes by the pair of values $(X(\omega), Y(\omega))$, exactly as in the proof of the third rule, gives $E[XY] = \sum_{x,y} xy\,P(X = x, Y = y)$, and independence factors each term:

$$E[XY] = \sum_{x, y} xy\, P(X = x)P(Y = y) = \Big(\sum_x x\, P(X = x)\Big)\Big(\sum_y y\, P(Y = y)\Big) = E[X]\,E[Y].$$

Now expand the variance of a sum, writing $\mu_X$ and $\mu_Y$ for the means:

$$\operatorname{Var}(X + Y) = E\big[\big((X - \mu_X) + (Y - \mu_Y)\big)^2\big] = \operatorname{Var}(X) + \operatorname{Var}(Y) + 2E\big[(X - \mu_X)(Y - \mu_Y)\big].$$

The last expectation is the covariance of $X$ and $Y$, written $\operatorname{Cov}(X, Y)$. By linearity,

$$\operatorname{Cov}(X, Y) = E[XY] - \mu_X E[Y] - \mu_Y E[X] + \mu_X\mu_Y = E[XY] - \mu_X \mu_Y,$$

which is zero for independent variables, by the product rule just shown. So

$$\operatorname{Var}(X + Y) = \operatorname{Var}(X) + \operatorname{Var}(Y) \quad \text{for independent } X, Y.$$

The same expansion works for any number of terms. Write $\mu_i$ for the mean of $X_i$. The mean of the sum is $\sum \mu_i$, and squaring the sum of the $n$ deviations $X_i - \mu_i$ gives $n$ squares and a product for every ordered pair of different indices:

$$\operatorname{Var}\Big(\sum_{i=1}^n X_i\Big) = E\Big[\Big(\sum_{i=1}^n (X_i - \mu_i)\Big)^2\Big] = \sum_{i=1}^n \operatorname{Var}(X_i) + \sum_{i \ne j} \operatorname{Cov}(X_i, X_j).$$

This holds with or without independence. If every pair $X_i, X_j$ is independent, every covariance is zero, and the variance of the sum is the sum of the variances (Grinstead and Snell, 1997, §6.2). Unlike linearity of expectation, this rule fails for dependent variables, whose covariances need not vanish. That failure is the subject of the last section of The Binomial Distribution.

Example (the variance of a difference). For independent $X$ and $Y$, the variance of the difference $X - Y$ is $\operatorname{Var}(X) + \operatorname{Var}(Y)$, not the difference of the variances. Write $X - Y = X + (-1)Y$. The variables $X$ and $-Y$ are independent, because $P(X = x, -Y = y) = P(X = x, Y = -y) = P(X = x)P(-Y = y)$, and $\operatorname{Var}(-Y) = (-1)^2\operatorname{Var}(Y)$. Subtracting a variable adds its uncertainty, just as adding one does.

Conditional expectation and the laws of total expectation and variance

The rules so far concern one distribution at a time. Some random experiments run in two stages: the first fixes a probability, and the second draws with it. The Binomial Distribution has one, in which a survey location is chosen at random and adults are then drawn from it. The two laws proved in this section are the tools for such experiments.

Let $P$ be a random variable with finitely many values $q$. In the two-stage experiment it is the probability fixed by the first stage. The conditional expectation of a variable $W$ given $P = q$ is its expected value under the conditional probabilities of Conditional Probability and the Law of Total Probability:

$$E[W \mid P = q] = \sum_w w\,P(W = w \mid P = q).$$

Written $E[W \mid P]$, it is a function of $P$ and so a random variable itself. The conditional variance is the variance of the conditional distribution, $\operatorname{Var}(W \mid P = q) = \sum_w (w - E[W \mid P = q])^2\,P(W = w \mid P = q)$. The computing formula applies to it as to any distribution, so $\operatorname{Var}(W \mid P) = E[W^2 \mid P] - (E[W \mid P])^2$. Two general facts are needed.

The law of total expectation: $E[W] = E\big[E[W \mid P]\big]$. Averaging the conditional expectations over the values of $P$,

$$E\big[E[W \mid P]\big] = \sum_q P(P = q) \sum_w w\,P(W = w \mid P = q) = \sum_w w \sum_q P(W = w,\ P = q) = \sum_w w\,P(W = w) = E[W],$$

by the multiplication rule and then the law of total probability.

The law of total variance: $\operatorname{Var}(X) = E[\operatorname{Var}(X \mid P)] + \operatorname{Var}(E[X \mid P])$. Apply the law of total expectation to $X^2$ and to $X$:

$$E[X^2] = E\big[E[X^2 \mid P]\big] = E\big[\operatorname{Var}(X \mid P)\big] + E\big[(E[X \mid P])^2\big], \qquad (E[X])^2 = \big(E\big[E[X \mid P]\big]\big)^2.$$

Subtracting the second from the first,

$$\operatorname{Var}(X) = E\big[\operatorname{Var}(X \mid P)\big] + \Big\{E\big[(E[X \mid P])^2\big] - \big(E\big[E[X \mid P]\big]\big)^2\Big\} = E\big[\operatorname{Var}(X \mid P)\big] + \operatorname{Var}\big(E[X \mid P]\big),$$

since the expression in braces is the variance of the random variable $E[X \mid P]$. $\square$

The Binomial Distribution applies both laws to a count drawn from one survey location, where they give the design effect of sampling in clusters. Overdispersion in Count Data applies the law of total variance to a Poisson count whose rate varies.

Summary

Quantity Formula
pmf and CDF $p(x) = P(X = x)$; $F(x) = P(X \le x) = \sum_{k \le x} p(k)$
Expected value $E[X] = \sum_x x\,p(x)$; $E[aX + b] = aE[X] + b$; $E[X + Y] = E[X] + E[Y]$
Variance $\operatorname{Var}(X) = E[(X - \mu)^2] = E[X^2] - \mu^2$; $\operatorname{Var}(aX + b) = a^2\operatorname{Var}(X)$
Independent sums $\operatorname{Var}(X + Y) = \operatorname{Var}(X) + \operatorname{Var}(Y)$
Sum of any variables $\operatorname{Var}\big(\sum X_i\big) = \sum \operatorname{Var}(X_i) + \sum_{i \ne j} \operatorname{Cov}(X_i, X_j)$
Total expectation and variance $E[W] = E\big[E[W \mid P]\big]$; $\operatorname{Var}(X) = E[\operatorname{Var}(X \mid P)] + \operatorname{Var}(E[X \mid P])$

What we have learned

A random variable is described by its distribution, and summarized by its expected value and its variance: for the number of an adult's three readings at or above 130 mm Hg, 1.0677 and 1.8360. Expectations add whatever the dependence; variances add only for independent variables. The variance of a sum is the sum of the variances plus the covariances of every ordered pair, and the laws of total expectation and total variance split a two-stage experiment into its stages.

The next article, The Binomial Distribution, builds the first theoretical distribution on these rules: the number of successes in a fixed number of independent trials. It then uses the covariances and the law of total variance to measure what happens when the trials are not independent.

The numbers in this article are computed by tools/experiments/applied-statistics/binomial-distribution.py in the site's repository (its Part 1 and the checks under "Reference pass"), from the NHANES files loaded by descdata.py.

References