Conditional Expectation and the Laws of Total Expectation and Variance

The conditional expectation of one random variable given another, defined for a discrete conditioning variable, for a joint density, for a count whose parameter is itself random and for a conditional distribution that mixes atoms with a density, with the law of total expectation proved in each case and everything else derived from it. The conditional expectation as the best predictor, the laws of total variance and total covariance, and the tower property. The design effect of cluster sampling, the overdispersion of a Poisson count with a varying rate and the mean and variance of a random sum follow as corollaries, and an exact example shows why conditioning on an event of probability zero depends on how the event is described. Worked problems on a density on a triangle, a binomial count with a uniform probability, a Poisson count with an exponential rate, a random sum, and two conditional means given the same event.

Among the 5,247 adults whose blood pressure was measured in the National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018, the 73 who are 30 years old have a mean systolic pressure of 115.42 mm Hg, and the 74 who are 70 have a mean of 132.64 (National Center for Health Statistics, 2020). The mean pressure depends on the age, and the function that gives it, age by age, predicts pressure from age better than any straight line can. What is that function for a pair of random variables in general, including a conditioning variable that takes each value with probability zero, and what does it say about the overall mean and variance? It is the conditional expectation, and two laws connect it to the whole: the mean of the conditional means is the overall mean, and the overall variance is the mean of the conditional variances plus the variance of the conditional means.

The conditional expectation of $Y$ given $X = x$ is the expected value of $Y$ computed from the conditional distribution of $Y$ given $X = x$. The course page Random Variables, Expectation and Variance defines it when both variables take finitely many values and proves the two laws in that case. This article gives the general treatment within Probability Theory, and the course's results on clusters and on overdispersed counts become corollaries.

First we condition on an event and then on a variable, in four settings that cover the uses on this site. Then we prove the law of total expectation in each setting and derive the rest from it: the conditional expectation as the best predictor, the laws of total variance and total covariance, and the tower property. Next we derive the corollaries. Finally we show by an exact example why conditioning on an event of probability zero is delicate, and say what measure theory adds. The article assumes Expectation, Variance and Covariance and Conditional Probability and the Law of Total Probability.

Conditioning on an event

Start with the simplest condition, an event $A$ with $P(A) > 0$, such as "the adult is 30 years old". The conditional probabilities $P(B \mid A) = P(A \cap B)/P(A)$ satisfy the three axioms, as Conditional Probability and the Law of Total Probability shows, so they are a probability in their own right, and every random variable has an expected value under it. For a variable $Y$ with a finite mean, the conditional expectation of $Y$ given $A$ is that expected value, and it has a simple formula:

$$E[Y \mid A] = \frac{E[Y\,\mathbf{1}_A]}{P(A)},$$

where $\mathbf{1}_A$ is the indicator of $A$. To see that the formula is right, use the general definition of the expected value from Expectation, Variance and Covariance. For $t \ge 0$ the event $\{Y\,\mathbf{1}_A > t\}$ is $\{Y > t\} \cap A$, and $\{Y\,\mathbf{1}_A < -t\}$ is $\{Y < -t\} \cap A$ for $t > 0$. So

$$\int_0^\infty P(Y > t \mid A)\,dt - \int_0^\infty P(Y < -t \mid A)\,dt = \frac{1}{P(A)}\Big[\int_0^\infty P(Y\,\mathbf{1}_A > t)\,dt - \int_0^\infty P(Y\,\mathbf{1}_A < -t)\,dt\Big] = \frac{E[Y\,\mathbf{1}_A]}{P(A)}.$$

For a discrete $Y$ this is $\sum_y y\,P(Y = y \mid A)$, the definition of the course page and of Grinstead and Snell (1997, Definition 6.2). In the population of adults, $E[Y \mid A]$ is the mean pressure of the adults in $A$: the sum of their pressures divided by their number.

Two facts about small events are needed before infinitely many conditions can be combined. Here is the idea: a variable with a finite mean cannot carry much of its expected value on an event of small probability.

Lemma (small events and countable partitions). Let $Y$ be a random variable.

  1. If $E\lvert Y\rvert < \infty$ and $B_1, B_2, \ldots$ are events with $P(B_n) \to 0$, then $E[\lvert Y\rvert\,\mathbf{1}_{B_n}] \to 0$. In particular $E[Y\,\mathbf{1}_B] = 0$ when $P(B) = 0$.
  2. Let $A_1, A_2, \ldots$ be mutually exclusive events whose union is the whole sample space. If $E\lvert Y\rvert < \infty$, then $E[Y] = \sum_k E[Y\,\mathbf{1}_{A_k}]$. If $Y \ge 0$ and $\sum_k E[Y\,\mathbf{1}_{A_k}] < \infty$, then $Y$ has a finite mean, equal to that sum.

Proof. (1) Fix $M > 0$. For any event $B$, $\lvert Y\rvert\,\mathbf{1}_B \le M\,\mathbf{1}_B + \lvert Y\rvert\,\mathbf{1}\{\lvert Y\rvert > M\}$: where $\lvert Y\rvert \le M$ the first term suffices, and elsewhere the second does. So $E[\lvert Y\rvert\,\mathbf{1}_B] \le M\,P(B) + E[W_M]$ with $W_M = \lvert Y\rvert\,\mathbf{1}\{\lvert Y\rvert > M\}$. The variable $W_M$ is never negative, and $P(W_M > t)$ equals $P(\lvert Y\rvert > M)$ for $t < M$ and $P(\lvert Y\rvert > t)$ for $t \ge M$. So

$$E[W_M] = M\,P(\lvert Y\rvert > M) + \int_M^\infty P(\lvert Y\rvert > t)\,dt.$$

The integral $\int_0^\infty P(\lvert Y\rvert > t)\,dt = E\lvert Y\rvert$ is finite, so its tail beyond $M$ tends to 0 as $M \to \infty$. The first term does too: the integrand is at least $P(\lvert Y\rvert > M)$ on the interval from $M/2$ to $M$, so $M\,P(\lvert Y\rvert > M) \le 2\int_{M/2}^{M} P(\lvert Y\rvert > t)\,dt$, a piece of the tail beyond $M/2$. Given $\varepsilon > 0$, choose $M$ with $E[W_M] < \varepsilon/2$, and then $n$ so large that $M\,P(B_n) < \varepsilon/2$. For an event with $P(B) = 0$, take $B_n = B$ for all $n$: $E[\lvert Y\rvert\,\mathbf{1}_B]$ is below every $\varepsilon$, so it is 0, and $\lvert E[Y\,\mathbf{1}_B]\rvert \le E[\lvert Y\rvert\,\mathbf{1}_B] = 0$.

(2) Let $C_n = A_1 \cup \cdots \cup A_n$ and $B_n = C_n^c$. By countable additivity $P(C_n) \to 1$, so $P(B_n) \to 0$. If $E\lvert Y\rvert < \infty$, linearity for finitely many terms gives $E[Y] = \sum_{k \le n} E[Y\,\mathbf{1}_{A_k}] + E[Y\,\mathbf{1}_{B_n}]$, and the last term tends to 0 by part 1. If instead $Y \ge 0$ and the sum $s = \sum_k E[Y\,\mathbf{1}_{A_k}]$ is finite, then for $t \ge 0$ the events $\{Y > t\}$ and $\{Y\,\mathbf{1}_{C_n} > t\}$ differ only inside $B_n$, so $P(Y > t) \le P(Y\,\mathbf{1}_{C_n} > t) + P(B_n)$. Integrating from 0 to $K$,

$$\int_0^K P(Y > t)\,dt \le E[Y\,\mathbf{1}_{C_n}] + K\,P(B_n) \le s + K\,P(B_n).$$

Letting $n \to \infty$ and then $K \to \infty$ gives $E[Y] \le s < \infty$, and the first case then applies. $\square$

Part 2 is the law of total expectation for a partition: since $E[Y\,\mathbf{1}_{A_k}] = E[Y \mid A_k]\,P(A_k)$ when $P(A_k) > 0$ and is 0 otherwise,

$$E[Y] = \sum_{k :\, P(A_k) > 0} E[Y \mid A_k]\,P(A_k).$$

Grinstead and Snell (1997, Theorem 6.5) prove it for a discrete variable and finitely many events.

Conditioning on a random variable

An adult's age is a random variable $X$, and to each of its values belongs an event and a conditional mean. Collecting them gives a function of $x$. The same idea needs a different formula when $X$ has a density, because then every event $\{X = x\}$ has probability zero. Four settings cover the uses on this site.

Setting D: $X$ is discrete. For each value $x$ with $P(X = x) > 0$,

$$m(x) = E[Y \mid X = x] = \frac{E[Y\,\mathbf{1}\{X = x\}]}{P(X = x)}.$$

Here $Y$ may be any random variable with a finite mean, or a function of several variables. When $Y$ is discrete too, $m(x) = \sum_y y\,p_{Y \mid X}(y \mid x)$ with the conditional mass function $p_{Y \mid X}(y \mid x) = P(X = x,\ Y = y)/P(X = x)$.

Setting C: $X$ and $Y$ have a joint density. With the conditional density $f_{Y \mid X}(y \mid x) = f_{X,Y}(x, y)/f_X(x)$ of Random Variables and Distribution Functions, defined where $f_X(x) > 0$,

$$m(x) = E[Y \mid X = x] = \int_{-\infty}^{\infty} y\,f_{Y \mid X}(y \mid x)\,dy.$$

Setting H: a hierarchical model. $X$ has a density $f_X$, and for each $x$ there is a mass function $p_{Y \mid X}(y \mid x)$ on a fixed finite or countable set of values $y$. The pair is described by

$$P(X \in B,\ Y = y) = \int_B p_{Y \mid X}(y \mid x)\,f_X(x)\,dx \qquad \text{for intervals } B,$$

and $m(x) = \sum_y y\,p_{Y \mid X}(y \mid x)$. This is the setting of a count whose probability or rate is itself drawn from a continuous distribution. Casella and Berger (2002) treat such models in their Section 4.4, "Hierarchical Models and Mixture Distributions".

Setting M: a mixed conditional distribution. $X$ has a density $f_X$. Given $X = x$, the variable $Y$ equals one of finitely many points $a_1(x), \ldots, a_k(x)$ with probabilities $\pi_1(x), \ldots, \pi_k(x)$, and otherwise has a density part $q(y \mid x) \ge 0$, where $\sum_j \pi_j(x) + \int q(y \mid x)\,dy = 1$. The pair is described by

$$P\big((X, Y) \in G\big) = \int_{-\infty}^{\infty}\Big[\sum_j \pi_j(x)\,\mathbf{1}\{(x, a_j(x)) \in G\} + \int_{-\infty}^{\infty} \mathbf{1}\{(x, y) \in G\}\,q(y \mid x)\,dy\Big]f_X(x)\,dx$$

for rectangles $G$ and, as for a joint density, for regions $G$ of the plane bounded by finitely many smooth curves. Here $m(x) = \sum_j a_j(x)\,\pi_j(x) + \int y\,q(y \mid x)\,dy$. Setting C is the case with no atoms, where $q$ is the conditional density, and setting H with finitely many values of $Y$ is the case with no density part and atoms that do not move with $x$. Setting M is needed when $Y$ can coincide with a function of $X$ with positive probability, as an observation coincides with the largest of its sample. Such a pair has no joint density.

In every setting, the conditional expectation of $Y$ given $X$, written $E[Y \mid X]$, is the random variable $m(X)$: the function $m$ evaluated at the random value of $X$. More generally, for a function $\psi(x, y)$ we write

$$m_\psi(x) = E[\psi(x, Y) \mid X = x]$$

for the conditional mean of the variable $\psi(x, Y)$, computed by the rule of the setting: $E[\psi(x, Y)\,\mathbf{1}\{X = x\}]/P(X = x)$ in D, $\int \psi(x, y)\,f_{Y \mid X}(y \mid x)\,dy$ in C, $\sum_y \psi(x, y)\,p_{Y \mid X}(y \mid x)$ in H, and $\sum_j \psi(x, a_j(x))\,\pi_j(x) + \int \psi(x, y)\,q(y \mid x)\,dy$ in M. The conditional variance is the case $\psi(x, y) = (y - m(x))^2$:

$$\operatorname{Var}(Y \mid X = x) = E\big[(Y - m(x))^2 \mid X = x\big].$$

The settings extend to vectors. In settings C and H, $Y$ may be a vector with a joint conditional density or mass function, and the sums and integrals over $y$ are then multiple. In settings D and C the conditioning variable may be a vector $(X_1, \ldots, X_k)$ as well, with a joint mass function or joint density in place of that of $X$; the letter $x$ then stands for a vector of values and the integrals over $x$ are multiple. Nothing else in the definitions or in the proofs below changes. For more than two variables with a joint density we take as given the same two facts about multiple integrals that Random Variables and Distribution Functions takes as given for double integrals, and that integrating a joint density over some of its variables gives the joint density of the others.

Two standing assumptions keep the formulas of Expectation, Variance and Covariance applicable. In settings C, H and M the functions $\psi$ and $m_\psi$ are ones to which the formula for the expected value of a function applies (the condition on $g$ in that theorem), and the integrals and sums that define $m_\psi(x)$ converge absolutely at every $x$ with $f_X(x) > 0$. Both hold in every example of this article.

Why the density formula is the right one. The ratio that defines $m(x)$ in setting D is $0/0$ when $X$ has a density. The natural substitute is to condition on $X$ falling in a short interval at $x$, which has positive probability, and let the interval shrink.

Proposition (the conditional density as a limit). In setting C, suppose $E\lvert Y\rvert < \infty$, $f_X$ is continuous at $x$ with $f_X(x) > 0$, and $N(s) = \int y\,f_{X,Y}(s, y)\,dy$ is continuous at $s = x$. Then

$$E[Y \mid x \le X \le x + \delta] \to m(x) \qquad \text{as } \delta \downarrow 0,$$

and the same holds for any intervals that contain $x$ and whose lengths tend to 0, such as $x - \delta < X < x + \delta$.

Proof. By the formula for the expected value of a function of a pair with a joint density, applied to $g(s, y) = y\,\mathbf{1}\{x \le s \le x + \delta\}$, and by the definition of the marginal density,

$$E[Y \mid x \le X \le x + \delta] = \frac{E[Y\,\mathbf{1}\{x \le X \le x + \delta\}]}{P(x \le X \le x + \delta)} = \frac{\frac1\delta\int_x^{x+\delta} N(s)\,ds}{\frac1\delta\int_x^{x+\delta} f_X(s)\,ds}.$$

The average of a function over intervals that contain a point of continuity and shrink to it tends to its value there, which is why the form of the interval does not matter. So the numerator tends to $N(x)$ and the denominator to $f_X(x)$, and $N(x)/f_X(x) = \int y\,f_{X,Y}(x, y)\,dy / f_X(x) = m(x)$. $\square$

The proposition is the justification for the definition, and it carries a warning: the limit is taken along intervals of $X$. The section on events of probability zero shows that shrinking a different family of sets toward the same event can give a different answer.

The law of total expectation

One theorem carries the whole subject. It says that an expected value can be computed in two stages: first average over $Y$ with $X$ held at $x$, then average over $X$.

Theorem 1 (the law of total expectation). In each of the settings D, C, H and M, let $\psi(X, Y)$ have a finite mean. Then

$$E[\psi(X, Y)] = E[m_\psi(X)], \qquad\text{that is,}\qquad E\big[\psi(X, Y)\big] = E\Big[E\big[\psi(X, Y) \mid X\big]\Big].$$

With $\psi(x, y) = y$ this is $E[Y] = E\big[E[Y \mid X]\big]$.

Proof. Setting D. The events $\{X = x\}$, over the values of $X$, are a countable partition (the remaining outcomes have probability zero and contribute nothing, by part 1 of the lemma). On the event $\{X = x\}$ the variable $\psi(X, Y)$ equals $\psi(x, Y)$. By part 2 of the lemma,

$$E[\psi(X, Y)] = \sum_x E\big[\psi(x, Y)\,\mathbf{1}\{X = x\}\big] = \sum_x m_\psi(x)\,P(X = x) = E[m_\psi(X)].$$

The last step is the formula for the expected value of a function of a discrete variable. Its series converges absolutely, because $\lvert m_\psi(x)\rvert\,P(X = x) \le E[\lvert\psi(X, Y)\rvert\,\mathbf{1}\{X = x\}]$ and these add to $E\lvert\psi(X, Y)\rvert$.

Setting C. By the formula for the expected value of a function of a pair, and computing the double integral with $y$ inside (for a $\psi$ that changes sign, separately for its positive and negative parts, whose inner integrals converge by the standing assumption),

$$E[\psi(X, Y)] = \int_{-\infty}^{\infty}\Big[\int_{-\infty}^{\infty} \psi(x, y)\,f_{X,Y}(x, y)\,dy\Big]dx = \int_{-\infty}^{\infty} m_\psi(x)\,f_X(x)\,dx = E[m_\psi(X)].$$

Where $f_X(x) > 0$ the inner integral is $m_\psi(x)\,f_X(x)$ by the definition of the conditional density. Where $f_X(x) = 0$, the nonnegative function $f_{X,Y}(x, \cdot)$ has integral zero, so the inner integral is zero as well. The last step is the formula for the expected value of the function $m_\psi$ of $X$. Its integral converges absolutely because $\lvert m_\psi(x)\rvert \le m_{\lvert\psi\rvert}(x)$, and the computation just made, applied to $\lvert\psi\rvert$, gives $\int m_{\lvert\psi\rvert}(x)\,f_X(x)\,dx = E\lvert\psi(X, Y)\rvert < \infty$.

Setting H. The events $\{Y = y\}$ are a countable partition, and on $\{Y = y\}$ the variable $\psi(X, Y)$ equals $\psi(X, y)$, so $E[\psi(X, Y)] = \sum_y E[\psi(X, y)\,\mathbf{1}\{Y = y\}]$ by part 2 of the lemma. Fix $y$ with $P(Y = y) > 0$. Under the conditional probability given $\{Y = y\}$, the variable $X$ has the density $p_{Y \mid X}(y \mid x)\,f_X(x)/P(Y = y)$, because that function integrates over an interval $B$ to $P(X \in B \mid Y = y)$. So by the formula for the expected value of a function of a variable with a density,

$$E\big[\psi(X, y)\,\mathbf{1}\{Y = y\}\big] = P(Y = y)\,E[\psi(X, y) \mid Y = y] = \int_{-\infty}^{\infty} \psi(x, y)\,p_{Y \mid X}(y \mid x)\,f_X(x)\,dx.$$

If $P(Y = y) = 0$, both sides of this equation are 0: the left by part 1 of the lemma, and the right because $p_{Y \mid X}(y \mid \cdot)\,f_X$ is nonnegative with integral $P(Y = y) = 0$. Summing over $y$ and exchanging the sum with the integral,

$$E[\psi(X, Y)] = \int_{-\infty}^{\infty}\Big[\sum_y \psi(x, y)\,p_{Y \mid X}(y \mid x)\Big]f_X(x)\,dx = \int_{-\infty}^{\infty} m_\psi(x)\,f_X(x)\,dx = E[m_\psi(X)].$$

The last step is again the formula for the expected value of a function of $X$, and its integral converges absolutely for the reason given in setting C, with the sum over $y$ in place of the inner integral.

Setting M. Let $\psi^+ = \max(\psi, 0)$ and $\psi^- = \max(-\psi, 0)$. Apply the description of the pair to the region $G = \{(x, y) : \psi(x, y) > t\}$:

$$P(\psi(X, Y) > t) = \int_{-\infty}^{\infty}\Big[\sum_j \pi_j(x)\,\mathbf{1}\{\psi(x, a_j(x)) > t\} + \int_{-\infty}^{\infty} \mathbf{1}\{\psi(x, y) > t\}\,q(y \mid x)\,dy\Big]f_X(x)\,dx.$$

Integrate over $t$ from 0 to $\infty$ and exchange the order of integration. Each indicator integrates to the length of the set of $t \ge 0$ below the value of $\psi$, which is $\psi^+$, so

$$\int_0^\infty P(\psi(X, Y) > t)\,dt = \int_{-\infty}^{\infty}\Big[\sum_j \pi_j(x)\,\psi^+(x, a_j(x)) + \int_{-\infty}^{\infty} \psi^+(x, y)\,q(y \mid x)\,dy\Big]f_X(x)\,dx = \int_{-\infty}^{\infty} m_{\psi^+}(x)\,f_X(x)\,dx.$$

For the other half apply the description to the region $\{(x, y) : \psi(x, y) < -t\}$, on which $\psi^- > t$:

$$\int_0^\infty P(\psi(X, Y) < -t)\,dt = \int_{-\infty}^{\infty}\Big[\sum_j \pi_j(x)\,\psi^-(x, a_j(x)) + \int_{-\infty}^{\infty} \psi^-(x, y)\,q(y \mid x)\,dy\Big]f_X(x)\,dx = \int_{-\infty}^{\infty} m_{\psi^-}(x)\,f_X(x)\,dx.$$

Both are finite because $\psi(X, Y)$ has a finite mean. By the general definition of the expected value, $E[\psi(X, Y)]$ is their difference, and $m_{\psi^+} - m_{\psi^-} = m_\psi$ because the rule that defines $m_\psi$ is linear in $\psi$. So $E[\psi(X, Y)] = \int m_\psi(x)\,f_X(x)\,dx = E[m_\psi(X)]$, the last step being the formula for the expected value of a function of $X$; its integral converges absolutely because $\lvert m_\psi\rvert \le m_{\psi^+} + m_{\psi^-}$. $\square$

Three things are used and not proved. In setting C it is the computation of a double integral as an iterated integral, the form of Fubini's theorem that this unit takes as given for nonnegative integrands; a signed integrand is handled through its positive and negative parts. In setting H it is the exchange of a sum over $y$ with an integral over $x$. For finitely many values of $y$ that is the linearity of the integral. For countably many it is the same theorem for a sum in place of one of the integrals, and it requires the series of absolute values to have a finite integral, which the hypothesis of a finite mean supplies (Billingsley, 1995, Section 18). In setting M it is the exchange of the integral over $t$ with the sum and integrals over $x$ and $y$, again for nonnegative integrands, together with the description of the pair for regions, which is part of the definition of the setting.

Example (an observation given the sample maximum). Let $X_1, \ldots, X_n$ be independent and uniform on $(0, \theta)$, with $n \ge 2$, and let $T$ be the largest of them. Sufficient Statistics finds that, given $T = x$, the observation $X_1$ equals $x$ with probability $1/n$ and is otherwise uniform on $(0, x)$. The pair $(T, X_1)$ has no joint density, because $P(X_1 = T) = 1/n$. It is in setting M, with $T$ as the conditioning variable. By the formula for a maximum, $P(T \le x) = (x/\theta)^n$, so $f_T(x) = nx^{n-1}/\theta^n$ on $(0, \theta)$. There is one atom, $a_1(x) = x$ with $\pi_1(x) = 1/n$, and the density part is $q(y \mid x) = (1 - \tfrac1n)/x$ for $0 < y < x$.

To confirm the description, take $0 < u \le s \le \theta$ and the set $\{(x', y') : x' \le s,\ y' \le u\}$. For $x \le u$ the bracket in the description is $\tfrac1n + (1 - \tfrac1n) = 1$, and for $u < x \le s$ it is $(1 - \tfrac1n)\,u/x$, since the atom at $x$ lies above $u$. So the description gives

$$\int_0^u \frac{nx^{n-1}}{\theta^n}\,dx + \int_u^s \Big(1 - \frac1n\Big)\frac{u}{x}\cdot\frac{nx^{n-1}}{\theta^n}\,dx = \frac{u^n}{\theta^n} + \frac{u\,(s^{n-1} - u^{n-1})}{\theta^n} = \frac{u}{\theta}\Big(\frac{s}{\theta}\Big)^{n-1}.$$

That is $P(T \le s,\ X_1 \le u)$ computed directly: $X_1 \le u$ and the other $n - 1$ observations at most $s$. For $u > s$ both are $(s/\theta)^n$, and the cases $u \le 0$ and $s \ge \theta$ are immediate. So the joint distribution function of the pair is the one the description gives, and a joint distribution function determines the joint distribution, the two-variable form of the uniqueness fact that this unit takes as given. The conditional mean is

$$E[X_1 \mid T = x] = \frac{x}{n} + \Big(1 - \frac1n\Big)\frac{x}{2} = \frac{(n + 1)\,x}{2n},$$

and Theorem 1 can be checked on it: $E[T] = \int_0^\theta x \cdot nx^{n-1}/\theta^n\,dx = n\theta/(n + 1)$, so $E\big[E[X_1 \mid T]\big] = \frac{n + 1}{2n}\cdot\frac{n\theta}{n + 1} = \frac{\theta}{2} = E[X_1]$. For $n = 3$ the law of total variance, proved below, can be checked as well. The conditional second moment is $\tfrac13x^2 + \tfrac23\cdot\tfrac13x^2 = \tfrac59x^2$ and the conditional mean $\tfrac23x$, so $\operatorname{Var}(X_1 \mid T = x) = \tfrac59x^2 - \tfrac49x^2 = \tfrac19x^2$. With $E[T^2] = \tfrac35\theta^2$ and $\operatorname{Var}(T) = \tfrac35\theta^2 - \tfrac{9}{16}\theta^2 = \tfrac{3}{80}\theta^2$, the two parts are $\tfrac19\cdot\tfrac35\theta^2 = \tfrac{1}{15}\theta^2$ and $\tfrac49\cdot\tfrac{3}{80}\theta^2 = \tfrac{1}{60}\theta^2$. They add to $\tfrac{1}{12}\theta^2$, the variance of a uniform variable on $(0, \theta)$.

Theorem 2 (rules for conditional means). In each setting, with all the variables below having finite means:

  1. Linearity. $E[a\,\psi_1(X, Y) + b\,\psi_2(X, Y) \mid X] = a\,E[\psi_1(X, Y) \mid X] + b\,E[\psi_2(X, Y) \mid X]$.
  2. Taking out what is known. $E[g(X)\,\psi(X, Y) \mid X] = g(X)\,E[\psi(X, Y) \mid X]$.
  3. Positivity. If $\psi \ge 0$ then $E[\psi(X, Y) \mid X] \ge 0$. Hence, when $Y$ has a finite variance, $\operatorname{Var}(Y \mid X) = E[Y^2 \mid X] - m(X)^2 \ge 0$.
  4. Independence. Let $X$ and $Y$ be independent. In setting D, $E[\psi(x, Y) \mid X = x] = E[\psi(x, Y)]$ for every $x$ with $P(X = x) > 0$. In setting C the same holds for each $x$ when the joint density is taken to be $f_X(x)\,f_Y(y)$, and in setting H when the conditional mass function is taken to be $p_Y(y)$ for every $x$. In particular $E[Y \mid X] = E[Y]$. This part is stated for settings D, C and H only.

Proof. Each statement is about the function $m_\psi(x)$ at a fixed $x$, where $g(x)$ is a constant.

(1) and (2). In setting D, $m_\psi(x)$ is $E[\psi(x, Y)\,\mathbf{1}\{X = x\}]/P(X = x)$, and the expected value is linear and lets the constant $g(x)$ out. In setting C it is an integral against $f_{Y \mid X}(\cdot \mid x)$, in setting H a sum against $p_{Y \mid X}(\cdot \mid x)$, and in setting M a finite sum against the $\pi_j(x)$ plus an integral against $q(\cdot \mid x)$; integrals and absolutely convergent sums are linear, and the constant $g(x)$ comes out of them.

(3) If $\psi \ge 0$, then $m_\psi(x)$ is the expected value of a nonnegative variable in D, the integral of a nonnegative function in C, a sum of nonnegative terms in H, and a sum of nonnegative terms plus the integral of a nonnegative function in M. For the variance, expand $(y - m(x))^2 = y^2 - 2m(x)\,y + m(x)^2$ and apply parts 1 and 2: $\operatorname{Var}(Y \mid X = x) = E[Y^2 \mid X = x] - 2m(x)\,m(x) + m(x)^2$.

(4) In setting D, $\psi(x, Y)$ is a function of $Y$ and $\mathbf{1}\{X = x\}$ a function of $X$, so the product rule for independent variables gives $E[\psi(x, Y)\,\mathbf{1}\{X = x\}] = E[\psi(x, Y)]\,P(X = x)$. In setting C, independence means that $f_X(x)\,f_Y(y)$ is a joint density, and with that joint density $f_{Y \mid X}(y \mid x) = f_Y(y)$ and $m_\psi(x) = \int \psi(x, y)\,f_Y(y)\,dy = E[\psi(x, Y)]$. In setting H, independence means $P(X \in B,\ Y = y) = P(X \in B)\,P(Y = y)$ for all intervals $B$, so the model is described by $p_{Y \mid X}(y \mid x) = p_Y(y)$, and with that choice $m_\psi(x) = \sum_y \psi(x, y)\,p_Y(y) = E[\psi(x, Y)]$. $\square$

Part 4 is a rule of substitution: under independence, to condition on $X = x$ is to replace $X$ by the number $x$ and average over $Y$ as usual. The choice of density in settings C and H matters only at single points. A joint density may be changed along a line $x = x_0$ without changing any probability, and that changes $m_\psi(x_0)$; the section on events of probability zero returns to this.

The best predictor and the two laws

Theorems 3 and 4 of this section follow from Theorems 1 and 2 with no further reference to the four settings; Theorem 5 returns to them. Throughout, $Y$ has a finite variance and $m(X) = E[Y \mid X]$.

Theorem 3 (consequences of the law of total expectation). Let $g$ be a function with $E[g(X)^2] < \infty$. In setting D it may be any such function. In settings C, H and M it must meet the standing assumption, as must the products and squares formed from it below; every $g$ used in this article does.

  1. Averaging. $E[g(X)\,Y] = E[g(X)\,m(X)]$. Equivalently, the error $Y - m(X)$ has mean zero and is uncorrelated with every $g(X)$: $E[(Y - m(X))\,g(X)] = 0$.
  2. Finite variance. $E[m(X)^2] \le E[Y^2]$.
  3. Pythagoras. $E\big[(Y - g(X))^2\big] = E\big[(Y - m(X))^2\big] + E\big[(m(X) - g(X))^2\big]$.
  4. Best predictor. $E[(Y - g(X))^2] \ge E[(Y - m(X))^2]$, with equality only if $g(X) = m(X)$ with probability 1.
  5. The law of total variance. $\operatorname{Var}(Y) = E[\operatorname{Var}(Y \mid X)] + \operatorname{Var}(E[Y \mid X])$.
  6. Uniqueness. If $h$ is a function with $E[h(X)^2] < \infty$ (in settings C, H and M, one that meets the standing assumption) and $E[g(X)\,Y] = E[g(X)\,h(X)]$ for every such $g$, then $h(X) = m(X)$ with probability 1.

Here is the idea. Among functions of $X$, $m(X)$ is the one whose error is uncorrelated with every function of $X$. That makes the error and any function of $X$ perpendicular in the sense of mean squares, and the rest is the theorem of Pythagoras.

Proof. (2) By part 3 of Theorem 2, $m(x)^2 \le E[Y^2 \mid X = x]$ for each $x$ at which $m$ is defined, a set that $X$ falls in with probability 1. Taking expected values and applying Theorem 1 to $\psi(x, y) = y^2$ gives $E[m(X)^2] \le E\big[E[Y^2 \mid X]\big] = E[Y^2]$.

(1) The product $g(X)\,Y$ has a finite mean because $\lvert g(X)\,Y\rvert \le (g(X)^2 + Y^2)/2$. Apply Theorem 1 to $\psi(x, y) = g(x)\,y$. By part 2 of Theorem 2 its conditional mean is $g(x)\,m(x)$, so $E[g(X)\,Y] = E[g(X)\,m(X)]$. Subtracting, $E[(Y - m(X))\,g(X)] = 0$, and $g = 1$ gives $E[Y - m(X)] = 0$.

(3) Write $Y - g(X) = (Y - m(X)) + (m(X) - g(X))$ and square. The cross term is $2E[(Y - m(X))\,h(X)]$ with $h = m - g$, and $E[h(X)^2] < \infty$ by part 2, so the cross term is zero by part 1.

(4) The second term of part 3 is never negative. If it is zero, then $(m(X) - g(X))^2$ is a nonnegative variable with mean zero, so it is zero with probability 1, by the lemma of Probability Inequalities.

(5) Take $g$ to be the constant $\mu = E[Y]$ in part 3: $\operatorname{Var}(Y) = E[(Y - m(X))^2] + E[(m(X) - \mu)^2]$. The second term is $\operatorname{Var}(m(X))$, since $E[m(X)] = \mu$ by Theorem 1. For the first, apply Theorem 1 to $\psi(x, y) = (y - m(x))^2$, whose conditional mean is $\operatorname{Var}(Y \mid X = x)$ by definition: $E[(Y - m(X))^2] = E[\operatorname{Var}(Y \mid X)]$.

(6) Let $d = m - h$, so that $E[d(X)^2] < \infty$. By part 1 and the hypothesis, both with $g = d$,

$$E\big[d(X)^2\big] = E[d(X)\,m(X)] - E[d(X)\,h(X)] = E[d(X)\,Y] - E[d(X)\,Y] = 0,$$

and a nonnegative variable with mean zero is zero with probability 1. $\square$

Part 5 is the course page's law, now for all four settings. Its two terms have names. $E[\operatorname{Var}(Y \mid X)]$ is the variation of $Y$ within the values of $X$, and by part 4 it is the smallest mean squared error with which any function of $X$ can predict $Y$. $\operatorname{Var}(E[Y \mid X])$ is the variation between them. Since both are nonnegative, $\operatorname{Var}(E[Y \mid X]) \le \operatorname{Var}(Y)$: replacing a variable by its conditional expectation never increases the variance. Sufficient Statistics turns that inequality into the Rao–Blackwell theorem.

Theorems 2 and 3 hold with any function $\psi(X, Y)$ of finite variance in place of $Y$, and $m_\psi$ in place of $m$ (in settings C, H and M, with the functions formed from $\psi$ in the proofs meeting the standing assumption). Their proofs apply Theorem 1 to functions of $(x, y)$ and never use the form of $Y$.

Theorem 4 (the law of total covariance). Let $Y$ and $Z$ have finite variances, in a setting where the pair $(Y, Z)$ plays the part of $Y$. Define $\operatorname{Cov}(Y, Z \mid X = x) = E[YZ \mid X = x] - m_Y(x)\,m_Z(x)$, where $m_Y(x) = E[Y \mid X = x]$ and $m_Z(x) = E[Z \mid X = x]$. Then

$$\operatorname{Cov}(Y, Z) = E\big[\operatorname{Cov}(Y, Z \mid X)\big] + \operatorname{Cov}\big(E[Y \mid X],\ E[Z \mid X]\big).$$

Proof. By Theorem 1 applied to the product $YZ$, and the definition of the conditional covariance,

$$E[YZ] = E\big[E[YZ \mid X]\big] = E\big[\operatorname{Cov}(Y, Z \mid X)\big] + E\big[m_Y(X)\,m_Z(X)\big].$$

By Theorem 1 applied to $Y$ and to $Z$, $E[Y]\,E[Z] = E[m_Y(X)]\,E[m_Z(X)]$. Subtracting the second equation from the first gives the claim, because $E[m_Y(X)\,m_Z(X)] - E[m_Y(X)]\,E[m_Z(X)]$ is the covariance of $m_Y(X)$ and $m_Z(X)$. $\square$

With $Z = Y$ this is the law of total variance again.

Theorem 5 (the tower property). Conditioning on more and then on less is the same as conditioning on less:

$$E\big[E[Y \mid X, Z] \mid X\big] = E[Y \mid X],$$

for a $Y$ with a finite mean, when $(X, Z)$ is discrete and when $(X, Z, Y)$ has a joint density. Here $E[Y \mid X, Z]$ is $m(X, Z)$, where $m(x, z)$ is the conditional mean of $Y$ given the pair of values: $E[Y\,\mathbf{1}\{X = x, Z = z\}]/P(X = x, Z = z)$ in the discrete case, and $\int y\,f(x, z, y)\,dy / f_{X,Z}(x, z)$ for a joint density $f$ of $(X, Z, Y)$ with marginal density $f_{X,Z}$ of the pair.

Proof. Discrete $(X, Z)$. Given $X = x$, the variable $Z$ takes the value $z$ with probability $P(X = x, Z = z)/P(X = x)$, so

$$E[m(X, Z) \mid X = x] = \sum_z m(x, z)\,\frac{P(X = x, Z = z)}{P(X = x)} = \frac{1}{P(X = x)}\sum_z E\big[Y\,\mathbf{1}\{X = x, Z = z\}\big] = \frac{E[Y\,\mathbf{1}\{X = x\}]}{P(X = x)},$$

where the last step is part 2 of the lemma applied to the variable $Y\,\mathbf{1}\{X = x\}$ and the partition by the values of $Z$. The series converges absolutely, since $\sum_z \lvert m(x, z)\rvert\,P(X = x, Z = z) \le E[\lvert Y\rvert\,\mathbf{1}\{X = x\}]$. The right side is $E[Y \mid X = x]$.

Joint density. The marginal densities $f_{X,Z}$ and $f_X$ are obtained from $f$ by integrating out $y$, and then $z$, and the conditional density of $Z$ given $X = x$ is $f_{X,Z}(x, z)/f_X(x)$. So

$$E[m(X, Z) \mid X = x] = \int m(x, z)\,\frac{f_{X,Z}(x, z)}{f_X(x)}\,dz = \frac{1}{f_X(x)}\int\!\!\int y\,f(x, z, y)\,dy\,dz = \frac{1}{f_X(x)}\int y\,f_{X,Y}(x, y)\,dy,$$

which is $E[Y \mid X = x]$, since integrating $f(x, z, y)$ over $z$ gives the joint density of $(X, Y)$. $\square$

Combined with the law of total variance, the tower property shows that conditioning on more information never increases the within part, for a $Y$ with a finite variance. Apply the law twice. With the pair $(X, Z)$ as the conditioning variable, $\operatorname{Var}(Y) = E[\operatorname{Var}(Y \mid X, Z)] + \operatorname{Var}(m(X, Z))$. The function $m(X, Z)$ has a finite variance, by part 2 of Theorem 3 with $(X, Z)$ as the conditioning vector. With $X$ as the conditioning variable and $Z$ in the part of $Y$, it takes the place of $Y$, as the remark after Theorem 3 allows, and $\operatorname{Var}(m(X, Z)) = E[\operatorname{Var}(m(X, Z) \mid X)] + \operatorname{Var}(E[m(X, Z) \mid X])$. The last term is $\operatorname{Var}(E[Y \mid X])$ by the tower property, so $\operatorname{Var}(E[Y \mid X, Z]) \ge \operatorname{Var}(E[Y \mid X])$. The total is fixed at $\operatorname{Var}(Y)$, so $E[\operatorname{Var}(Y \mid X, Z)] \le E[\operatorname{Var}(Y \mid X)]$.

Pressure given age

The theorems can be read off the 5,247 adults. Age $X$ is recorded in whole years from 18 to 80, with 80 standing for 80 and over, so it takes 63 values and the pair is in setting D. The conditional mean $m(x)$ is the mean pressure of the adults of age $x$.

Age $x$ Adults $m(x) = E[Y \mid X = x]$ $\operatorname{Var}(Y \mid X = x)$ Best line at $x$
20 67 111.12 100.39 110.62
30 73 115.42 151.55 115.89
50 74 123.14 171.14 126.43
70 74 132.64 328.50 136.97
80 and over 350 143.90 513.55 142.24

The last column is the best linear predictor $100.09 + 0.5269x$ of Expectation, Variance and Covariance. Both the mean and the spread of pressure rise with age: the conditional standard deviation is 12.31 mm Hg at 30 and 18.12 at 70.

The mean systolic pressure at each age among the 5,247 adults (one dot per age, larger dots for ages with more adults), with the best straight line and the overall mean. Notice that the dots do not follow the line: they run level through the twenties, and lie below the line from the mid-forties to about fifty and above it through the fifties. The conditional mean is a curve, and the line is its best straight approximation.
The mean systolic pressure at each age among the 5,247 adults (one dot per age, larger dots for ages with more adults), with the best straight line and the overall mean. Notice that the dots do not follow the line: they run level through the twenties, and lie below the line from the mid-forties to about fifty and above it through the fifties. The conditional mean is a curve, and the line is its best straight approximation.

Total expectation. Weighting each $m(x)$ by the share of adults of that age and adding the 63 terms gives 126.36, the overall mean.

Total variance and the best predictor. The overall variance, 405.01, splits into $E[\operatorname{Var}(Y \mid X)] = 304.12$ within ages and $\operatorname{Var}(E[Y \mid X]) = 100.88$ between ages, up to rounding in the last digit. The three predictors of pressure can now be compared by their mean squared errors:

Predictor of pressure Mean squared error Root
The overall mean, 126.36 405.01 20.12
The best straight line in age 309.11 17.58
The conditional mean $m(X)$ 304.12 17.44

The share of variance removed by the conditional mean is $100.88/405.01 = 0.2491$, against $\rho^2 = 0.2368$ for the line. The difference between the two errors, $309.11 - 304.12 = 4.99$, is exactly $E[(m(X) - a^* - b^*X)^2]$, the mean squared distance between the curve and the line, as part 3 of Theorem 3 says with $g$ the line. So for these data the line loses little. No function of age alone can do better than 304.12: an adult's age leaves a standard deviation of 17.44 mm Hg unexplained. A cruder function does worse than the line. Predicting by the mean of three age groups (18 to 39, 40 to 59, 60 and over; means 114.83, 126.32 and 136.85) has mean squared error 320.50.

Example (the conditional mean is not a line through the data). "The regression of pressure on age is the least squares line." The regression function is $m(x) = E[Y \mid X = x]$, whatever its shape. The least squares line is the best approximation to it among straight lines, and it coincides with $m$ only when $m$ is a straight line. Here the two differ by 3.29 mm Hg at age 50.

The tower property. Conditioning also on sex splits each age into two groups. At age 50 there are 40 men with mean 123.03 and 34 women with mean 123.27, and $(40 \times 123.03 + 34 \times 123.27)/74 = 123.14 = m(50)$. The script checks the identity at all 63 ages. As the remark after Theorem 5 requires, the within part falls: $E[\operatorname{Var}(Y \mid X, \text{sex})] = 292.62$, below 304.12.

Total covariance. For the 4,730 adults with three readings, the covariance of the first and second readings is 373.17. Conditioning on age splits it into $E[\operatorname{Cov}(R_1, R_2 \mid X)] = 276.83$ and $\operatorname{Cov}(E[R_1 \mid X], E[R_2 \mid X]) = 96.34$. About a quarter of the covariance between two readings of the same person is explained by the person's age, and the rest is what the two readings share beyond age.

The course's results as corollaries

Four results that the course pages prove case by case, or state without proof, are instances of the two laws.

Clusters and the design effect

Let $P$ be a random probability with mean $\bar{p}$ and variance $v$, and given $P = q$ let $X$ have the binomial distribution with $n$ trials and success probability $q$. If $P$ takes finitely many values this is setting D, the case The Binomial Distribution treats with the 30 survey locations of NHANES. If $P$ has a density on $(0, 1)$ it is setting H, with $P$ as the conditioning variable. In both, the binomial mean and variance give $E[X \mid P] = nP$ and $\operatorname{Var}(X \mid P) = nP(1 - P)$, and so

$$E[X] = n\bar{p}, \qquad \operatorname{Var}(X) = E[nP(1 - P)] + \operatorname{Var}(nP) = n(\bar{p} - v - \bar{p}^2) + n^2v = n\bar{p}(1 - \bar{p})\big[1 + (n - 1)\rho\big],$$

with $\rho = v/[\bar{p}(1 - \bar{p})]$, using $E[P^2] = v + \bar{p}^2$. That is the course's design effect, for a $P$ that is discrete or has a density. The meaning of $\rho$ comes from Theorem 4. Let $Y_1$ and $Y_2$ be the indicators of success on two of the trials. Given $P$ they are independent, so $\operatorname{Cov}(Y_1, Y_2 \mid P) = 0$, and each has conditional mean $P$. Hence $\operatorname{Cov}(Y_1, Y_2) = 0 + \operatorname{Cov}(P, P) = v$, and since each indicator has variance $\bar{p}(1 - \bar{p})$, their correlation is $\rho$. Problem 2 works the case of a uniform $P$, where the design effect for $n = 10$ is 4.

Any distribution of $P$. The formula does not depend on those two settings. For an arbitrary random probability $P$, the statement "given $P = q$, the $n$ trials are independent with success probability $q$" means that the indicators $Y_1, \ldots, Y_n$ of the trials satisfy

$$P(Y_1 = y_1, \ldots, Y_n = y_n,\ P \in B) = E\big[P^{s}(1 - P)^{n - s}\,\mathbf{1}\{P \in B\}\big], \qquad s = y_1 + \cdots + y_n,$$

for all $y_i$ in $\{0, 1\}$ and every interval $B$. Adding over the $\binom{n}{k}$ sequences with $s = k$ gives, for the count $X = Y_1 + \cdots + Y_n$, $P(X = k,\ P \in B) = E[b(k; P)\,\mathbf{1}\{P \in B\}]$, where $b(k; q) = \binom{n}{k}q^k(1 - q)^{n - k}$. In settings D and H these statements are Theorem 1 applied to the indicators of the events. With $B$ the whole line, $P(X = k) = E[b(k; P)]$. Since $X$ takes only the values $0, \ldots, n$, linearity for finitely many terms and the binomial mean and second moment at each fixed $q$ give

$$E[X] = \sum_{k=0}^n k\,E[b(k; P)] = E\Big[\sum_{k=0}^n k\,b(k; P)\Big] = E[nP], \qquad E[X^2] = E\Big[\sum_{k=0}^n k^2\,b(k; P)\Big] = E\big[nP(1 - P) + n^2P^2\big].$$

So $E[X] = n\bar{p}$ and $\operatorname{Var}(X) = n\bar{p} - nE[P^2] + n^2E[P^2] - n^2\bar{p}^2 = n\bar{p}(1 - \bar{p}) + n(n - 1)v$, the same formula, for every distribution of $P$. The correlation of two trials is $\rho$ in general as well. Take $B$ the whole line and add the model's probabilities over the values of $Y_3, \ldots, Y_n$ with $y_1 = y_2 = 1$. By linearity for finitely many terms the sum is $E\big[P^2\,(P + (1 - P))^{n - 2}\big] = E[P^2]$, the binomial theorem collecting the $2^{n - 2}$ terms. So $P(Y_1 = 1,\ Y_2 = 1) = E[P^2]$. Adding over $Y_2, \ldots, Y_n$ with $y_1 = 1$ gives $P(Y_1 = 1) = E[P] = \bar{p}$ in the same manner, the sum being $E\big[P\,(P + (1 - P))^{n - 1}\big]$, and likewise for $Y_2$. So the covariance of the two indicators is $E[P^2] - \bar{p}^2 = v$.

Overdispersion

Let $\Lambda \ge 0$ be a random rate with mean $\mu$ and a finite variance, and given $\Lambda = \lambda$ let $X$ have the Poisson distribution with mean $\lambda$. When $\Lambda$ is discrete (setting D) or has a density (setting H), $E[X \mid \Lambda] = \Lambda$ and $\operatorname{Var}(X \mid \Lambda) = \Lambda$, and the two laws give

$$E[X] = \mu, \qquad \operatorname{Var}(X) = E[\Lambda] + \operatorname{Var}(\Lambda) = \mu + \operatorname{Var}(\Lambda).$$

The law of total variance requires $E[X^2] < \infty$, and the next paragraph supplies it, along with the formula itself for a rate with any distribution.

Any distribution of the rate. For an arbitrary $\Lambda \ge 0$, "given $\Lambda = \lambda$, $X$ is Poisson" means that $P(X = k,\ \Lambda \in B) = E[p(k; \Lambda)\,\mathbf{1}\{\Lambda \in B\}]$ for every $k$ and every interval $B$, where $p(k; \lambda) = e^{-\lambda}\lambda^k/k!$. So $P(X = k) = E[p(k; \Lambda)]$. Take $j = 0$, 1 or 2, and write $h_j(\lambda) = \sum_k k^j\,p(k; \lambda)$, with $k^0 = 1$ for every $k$: the total probability and the first and second moments of a Poisson count, $h_0(\lambda) = 1$, $h_1(\lambda) = \lambda$ and $h_2(\lambda) = \lambda + \lambda^2$. The claim is that $\sum_k k^j\,P(X = k) = E[h_j(\Lambda)]$. For $j = 0$ it says that the probabilities $P(X = k)$ add to 1, so that $X$ takes only the values $0, 1, 2, \ldots$; for $j = 1$ and 2 it then says that $E[X^j] = E[h_j(\Lambda)]$. By linearity for finitely many terms, the partial sums satisfy

$$\sum_{k \le K} k^j\,P(X = k) = E\Big[\sum_{k \le K} k^j\,p(k; \Lambda)\Big] = E[h_j(\Lambda)] - E[r_K(\Lambda)], \qquad r_K(\lambda) = \sum_{k > K} k^j\,p(k; \lambda).$$

The remainder satisfies $0 \le r_K(\lambda) \le h_j(\lambda)$. For $\lambda \le M$ it is at most $c_K(M) = \sum_{k > K} k^jM^k/k!$, because $e^{-\lambda} \le 1$ and $\lambda^k \le M^k$, and $c_K(M) \to 0$ as $K \to \infty$, being the tail of a convergent series. Splitting according to whether $\Lambda \le M$,

$$0 \le E[r_K(\Lambda)] \le c_K(M) + E\big[h_j(\Lambda)\,\mathbf{1}\{\Lambda > M\}\big].$$

The variable $h_j(\Lambda)$ has a finite mean, because $\Lambda$ has a finite variance, and $P(\Lambda > M) \to 0$ as $M \to \infty$, so the last term tends to 0 by part 1 of the lemma. Given $\varepsilon > 0$, choose $M$ to make it less than $\varepsilon/2$ and then $K$ to make $c_K(M) < \varepsilon/2$. So the partial sums converge to $E[h_j(\Lambda)]$, that is, the probabilities add to 1, $E[X] = E[\Lambda] = \mu$ and $E[X^2] = E[\Lambda] + E[\Lambda^2] < \infty$. Subtracting $\mu^2$ gives $\operatorname{Var}(X) = \mu + \operatorname{Var}(\Lambda)$ for every distribution of the rate with a finite variance.

Overdispersion in Count Data uses this formula, proved for a rate with finitely many values, and notes that for a continuous rate the sums become integrals. Setting H of Theorem 1 is that statement with its proof, and the paragraph above removes the restriction to rates that are discrete or have a density. For a gamma rate with shape $k$, whose variance that article computes as $\mu^2/k$, it gives the negative binomial variance $\mu + \mu^2/k$. Problem 3 works the case $k = 1$, an exponential rate.

Random sums

Let $X_1, X_2, \ldots$ be independent with a common mean $\mu$ and variance $\sigma^2$, and let $N$ be a random variable with values $0, 1, 2, \ldots$ and a finite variance, independent of the sequence. That means: for every $n$, $N$ is independent of the vector $(X_1, \ldots, X_n)$, in the sense defined for vectors in Random Variables and Distribution Functions. The random sum is $S = X_1 + \cdots + X_N$, with $S = 0$ when $N = 0$: the total of a random number of terms, such as the total cost of a random number of claims.

Corollary (mean and variance of a random sum). $E[S] = \mu\,E[N]$ and $\operatorname{Var}(S) = \sigma^2\,E[N] + \mu^2\operatorname{Var}(N)$.

Proof. This is setting D with $N$ as the conditioning variable and $S$, and then $S^2$, as the variable whose conditional mean is wanted. Write $S_n = X_1 + \cdots + X_n$. On the event $\{N = n\}$, $S = S_n$ and $S^2 = S_n^2$. The variables $S_n$ and $S_n^2$ are functions of the vector $(X_1, \ldots, X_n)$, and $\mathbf{1}\{N = n\}$ is a function of $N$, so they are independent of it. By the product rule for independent variables, $E[S\,\mathbf{1}\{N = n\}] = E[S_n\,\mathbf{1}\{N = n\}] = E[S_n]\,P(N = n)$, and likewise $E[S^2\,\mathbf{1}\{N = n\}] = E[S_n^2]\,P(N = n)$. Dividing by $P(N = n)$,

$$E[S \mid N = n] = E[S_n] = n\mu, \qquad E[S^2 \mid N = n] = E[S_n^2] = \operatorname{Var}(S_n) + (n\mu)^2 = n\sigma^2 + n^2\mu^2.$$

First, $S^2$ has a finite mean: it is nonnegative and $\sum_n E[S^2\,\mathbf{1}\{N = n\}] = \sum_n (n\sigma^2 + n^2\mu^2)\,P(N = n) = \sigma^2E[N] + \mu^2E[N^2] < \infty$, so part 2 of the lemma applies, and $S$ has a finite variance. Then Theorem 1 gives $E[S] = E[N\mu] = \mu\,E[N]$, and the law of total variance, with $\operatorname{Var}(S \mid N = n) = n\sigma^2$, gives $\operatorname{Var}(S) = E[N\sigma^2] + \operatorname{Var}(N\mu) = \sigma^2E[N] + \mu^2\operatorname{Var}(N)$. $\square$

The variance has a term for the variation of the terms and a term for the variation of their number. For a Poisson number of terms with mean $\lambda$, $E[N] = \operatorname{Var}(N) = \lambda$ and the variance is $\lambda(\sigma^2 + \mu^2) = \lambda\,E[X_1^2]$. The formula for the mean holds more widely than this proof shows. Wald (1944, equation (85), p. 295) obtained the same relation between the expected number of terms and the expected sum, under conditions stated there, for the number of steps a cumulative sum takes to leave an interval. That number depends on the terms themselves and is not independent of them.

Within and between groups

Let $X$ label the group of a randomly chosen member of a finite population divided into groups, and $Y$ be a measurement. Then $E[\operatorname{Var}(Y \mid X)]$ is the average of the group variances weighted by group size, and $\operatorname{Var}(E[Y \mid X])$ is the weighted variance of the group means. Multiplied by the population size, the law of total variance is the identity that splits the total sum of squares into within-groups and between-groups sums, on which One-Way Analysis of Variance is built.

Example (one reading of one person). Choose one of the 4,730 adults at random, and then one of that adult's three readings at random. The reading has variance 382.58. The person is the conditioning variable. The variance of a person's three readings about their own mean averages 9.05 over the adults, and the variance of the adults' means is 373.53, the figure found from the covariance matrix in Expectation, Variance and Covariance. So 97.6% of the variance of a single reading is variation between people. The remaining 9.05 is the average spread of a person's three readings about their own mean, a root mean square of 3.01 mm Hg within one examination. It is not the spread of the difference between two readings. With the divisor 2 in place of 3 for each person's variance, the usual estimate of the variance of a single reading about the person's level, the average is 13.58, a standard deviation of 3.69 mm Hg.

Conditioning on an event of probability zero

Setting C conditions on $\{X = x\}$, an event of probability zero, and the proposition above justifies the formula by a limit over short intervals of $X$. The same event can often be described through a different variable. Does the description matter? It does.

Example (one event, two conditional means). Let $X$ and $Y$ be independent, each with the exponential density $e^{-x}$ on $x > 0$, and consider the event that they are equal. It is the event $\{D = 0\}$ for the difference $D = X - Y$, and it is the event $\{R = 1\}$ for the ratio $R = X/Y$. Problem 5 computes

$$E[X \mid D = 0] = \tfrac12, \qquad E[X \mid R = 1] = 1.$$

Both answers are correct, and they answer different questions. By the proposition on the conditional density as a limit, whose continuity conditions hold for the densities found in Problem 5, $E[X \mid D = 0]$ is the limit of $E[X \mid \lvert X - Y\rvert < \delta]$, an average over a strip of constant width around the diagonal. $E[X \mid R = 1]$ is the limit of $E[X \mid \lvert X/Y - 1\rvert < \delta]$, an average over a wedge that widens with distance from the origin, and so gives more weight to large values of $X$. The event $\{X = Y\}$ has probability zero, and a conditional expectation given such an event is defined only through a limit, which depends on the sets that shrink to it.

So the notation $E[Y \mid X = x]$ in settings C, H and M must be read as a statement about the variable $X$, not about the event alone. The dependence of such a conditional expectation on the description of the event is known as the Borel paradox. Kolmogorov (1933, Chapter V, §2, "Explanation of a Borel paradox") takes up Borel's example, the latitude of a point chosen uniformly on a sphere given the meridian circle it lies on, and concludes that "the concept of a conditional probability with regard to an isolated given hypothesis whose probability equals 0 is inadmissible" (p. 51 of the 1950 English translation): a distribution on the circle is obtained only by treating the circle as one member of the family of all meridian circles with the given poles. Billingsley (1995) treats the paradox in Section 33. Proschan and Presnell (1998) show that seemingly logical conditioning arguments can lead to wrong conclusions when conditional distributions and expectations are handled without rigor.

What the experiment shows

The script ran two simulations for each of 30 seeds (NumPy's default_rng with seeds 0 to 29). In the first it drew 200,000 random sums, each of a Poisson number of terms with mean 4, the terms exponential with mean 2, for which the corollary gives mean $4 \times 2 = 8$ and variance $4 \times (4 + 4) = 32$. In the second it drew 1,000,000 pairs of independent exponential variables with mean 1 and averaged $X$ over the pairs falling in each of two thin sets around the diagonal.

Quantity Exact value Median over 30 seeds Interquartile range
Mean of the random sum 8 8.001 7.993 to 8.006
Variance of the random sum 32 31.99 31.90 to 32.05
Mean of $X$ given $\lvert X - Y\rvert < 0.01$ 0.502 0.503 0.499 to 0.506
Mean of $X$ given $\lvert X/Y - 1\rvert < 0.01$ 1.000 0.997 0.991 to 1.006

The exact values in the last two rows are for the sets of width 0.01, computed by numerical integration; their limits are $\tfrac12$ and 1. At least 9,773 pairs fell in the strip and at least 4,820 in the wedge in every run. In these runs the two conditional means given nearly the same event differ by a factor of two, as the example says they must. The simulation illustrates the corollary and the example; the proofs are above.

What is proved here, what is cited, and what measure theory would add

Proved on this page: the lemma on small events and partitions; the proposition on the conditional density as a limit; Theorems 1 to 5 in the settings stated; the corollaries, with the design effect and the overdispersion formula for every distribution of the random probability or rate; and the example of the sample maximum.

Taken as given or cited:

  • Everything that Random Variables and Distribution Functions and Expectation, Variance and Covariance take as given, in particular that the sets in question are events and that a joint distribution function determines a joint distribution.
  • For a joint density, that the probability of a region bounded by smooth curves is the integral of the density over it, and that a multiple integral of a nonnegative function may be computed as an iterated integral in any order (Fubini's theorem; Billingsley, 1995, Section 18). Both are used for two variables in Theorem 1 and for three in Theorem 5, together with the fact that integrating out a variable gives the joint density of the rest. Signed integrands are handled through their positive and negative parts.
  • In setting H with countably many values, the exchange of a sum with an integral, which is the same theorem.
  • In setting M, the description of the pair for regions and not only for rectangles, which is part of the definition of the setting.
  • For random sums, that a function of the vector $(X_1, \ldots, X_n)$ is independent of $N$ when the vector is. That is immediate from the definition of independence for vectors.

What is avoided. The four settings do not exhaust the possibilities. A conditioning variable with a mixed distribution falls outside them. So does a conditioning vector with a density when the variables have no joint density as setting C requires (setting M is stated for a single conditioning variable), and for conditioning on infinitely many variables there is no density to divide by. Kolmogorov (1933) gave conditional probabilities and expectations Chapter V of his foundations of probability. His preface lists the points of the book that lay outside the ideas then familiar to specialists, ending with "especially the theory of conditional probabilities and conditional expectations (Chapter V)". The general theory, as he gives it in §4 of that chapter and as Billingsley (1995, Section 34) presents it, turns part 1 of Theorem 3 into the definition. A conditional expectation of $Y$ given $X$ is any function $m(X)$ with

$$E\big[Y\,\mathbf{1}\{X \in B\}\big] = E\big[m(X)\,\mathbf{1}\{X \in B\}\big] \qquad \text{for every set } B.$$

That such a function exists for every $Y$ with a finite mean, and is unique up to events of probability zero, is a theorem of the general theory, which rests on the Radon–Nikodym theorem (Billingsley, 1995, Sections 32 and 34). Kolmogorov (1933, Chapter V, §4) proves it that way: the existence of $E[Y]$ is sufficient for the existence of the conditional expectation, by "the theorem of Nikodym", and the conditional expectation is determined uniquely "except for equivalence" (pp. 52–53 of the translation). Theorem 1, applied to $\psi(x, y) = y\,\mathbf{1}\{x \in B\}$, shows that the function $m$ of this article satisfies that requirement for every interval $B$, and hence for every set $B$ by the extension from intervals that Random Variables and Distribution Functions cites. Part 6 of Theorem 3 shows that, for variables with finite variances, $m$ is determined by the stronger requirement that the identity hold for every $g(X)$ with a finite second moment (in settings C, H and M, every such $g$ that meets the standing assumption). Passing from indicators to such functions is an approximation argument of the general theory, not given here.

What is lost. Three things. First, existence in general: here $m$ has to be exhibited by a formula in each setting, and the proofs of Theorems 1 and 2 are done once per setting. Second, the understanding that a conditional expectation is determined only up to events of probability zero, so that its value at a single point $x$ of a continuous variable has no meaning by itself. That is the root of the two answers in the example above. Third, conditioning on information that is not a single variable or a finite list of them, which the theory of martingales needs (Billingsley, 1995, Section 35). What the four settings cover can be stated exactly: a conditioning variable or vector that is discrete, with any $Y$; a conditioning variable or vector with a density, with a $Y$ such that the whole collection has a joint density; and a single conditioning variable with a density, with a $Y$ that has a conditional mass function on a fixed finite or countable set of values, or finitely many atoms, which may move with $x$, plus a density part. A $Y$ with countably many atoms that move with $x$ is not covered. The conditional distribution of an observation given the sample maximum, used in Sufficient Statistics, is of the last kind. Conditioning on a statistic of several continuous observations in general, as the Rao–Blackwell theorem requires outside the discrete case, is covered only when the pair can be put in one of these forms; otherwise it rests on the general definition.

Worked problems

These problems are posed here, on small exact cases that exercise each theorem; none is from a textbook. The script tools/experiments/statistics/conditional-expectation.py in the site's repository checks every number, the exact ones with fractions and the integrals numerically.

Problem 1 (a density on a triangle). The pair $(X, Y)$ has joint density $f(x, y) = 2$ on the triangle $0 < y < x < 1$ and 0 elsewhere. (a) Find the marginal density of $X$, the conditional density of $Y$ given $X = x$, and $E[Y \mid X]$ and $\operatorname{Var}(Y \mid X)$. (b) Find $E[Y]$ and $\operatorname{Var}(Y)$ by the two laws, and check them from the marginal density of $Y$. (c) Compare the best predictor of $Y$ from $X$ with the best linear predictor.

Solution. (a) For $0 < x < 1$, $f_X(x) = \int_0^x 2\,dy = 2x$. The conditional density is $f_{Y \mid X}(y \mid x) = 2/(2x) = 1/x$ for $0 < y < x$: given $X = x$, $Y$ is uniform on $(0, x)$. A uniform variable on $(0, x)$ has mean $\int_0^x y/x\,dy = x/2$ and second moment $\int_0^x y^2/x\,dy = x^2/3$, so

$$E[Y \mid X] = \frac{X}{2}, \qquad \operatorname{Var}(Y \mid X) = \frac{X^2}{3} - \frac{X^2}{4} = \frac{X^2}{12}.$$

(b) First the moments of $X$: $E[X] = \int_0^1 x \cdot 2x\,dx = \tfrac23$ and $E[X^2] = \int_0^1 x^2 \cdot 2x\,dx = \tfrac12$, so $\operatorname{Var}(X) = \tfrac12 - \tfrac49 = \tfrac{1}{18}$. By the law of total expectation, $E[Y] = E[X/2] = \tfrac13$. By the law of total variance,

$$\operatorname{Var}(Y) = E\Big[\frac{X^2}{12}\Big] + \operatorname{Var}\Big(\frac{X}{2}\Big) = \frac{1}{24} + \frac{1}{72} = \frac{1}{18},$$

that is, $0.0417 + 0.0139 = 0.0556$. The check: $f_Y(y) = \int_y^1 2\,dx = 2(1 - y)$ for $0 < y < 1$, so $E[Y] = \int_0^1 2y(1 - y)\,dy = 2(\tfrac12 - \tfrac13) = \tfrac13$ and $E[Y^2] = \int_0^1 2y^2(1 - y)\,dy = 2(\tfrac13 - \tfrac14) = \tfrac16$, giving $\operatorname{Var}(Y) = \tfrac16 - \tfrac19 = \tfrac{1}{18}$. Three quarters of the variance of $Y$ is within values of $X$ and one quarter between them.

(c) The best predictor is $E[Y \mid X] = X/2$, with mean squared error $E[\operatorname{Var}(Y \mid X)] = \tfrac{1}{24}$. For the best line we need the covariance. By part 1 of Theorem 3 with $g(x) = x$, $E[XY] = E[X \cdot X/2] = \tfrac14$, so $\operatorname{Cov}(X, Y) = \tfrac14 - \tfrac23 \cdot \tfrac13 = \tfrac{1}{36}$. The slope is $\tfrac{1}{36}/\tfrac{1}{18} = \tfrac12$ and the intercept $\tfrac13 - \tfrac12 \cdot \tfrac23 = 0$: the best line is $x/2$, the conditional mean itself. Its error is $\operatorname{Var}(Y)(1 - \rho^2)$ with $\rho^2 = (\tfrac{1}{36})^2/(\tfrac{1}{18} \cdot \tfrac{1}{18}) = \tfrac14$, which is $\tfrac{1}{18} \cdot \tfrac34 = \tfrac{1}{24}$, the same. When the conditional mean is a straight line, the best linear predictor is the best predictor.

Problem 2 (a binomial count with a uniform probability). A cluster's success probability $P$ is uniform on $(0, 1)$, and given $P$ the number of successes $X$ among $n$ members is binomial. (a) Find $E[X]$ and $\operatorname{Var}(X)$, and the design effect. (b) For $n = 2$, find the distribution of $X$ and check the variance.

Solution. (a) This is setting H. The uniform distribution has $\bar{p} = \int_0^1 q\,dq = \tfrac12$ and $E[P^2] = \int_0^1 q^2\,dq = \tfrac13$, so $v = \tfrac13 - \tfrac14 = \tfrac{1}{12}$. By the laws, $E[X] = n/2$ and

$$\operatorname{Var}(X) = n\,E[P(1 - P)] + n^2\operatorname{Var}(P) = n\Big(\frac12 - \frac13\Big) + \frac{n^2}{12} = \frac{n}{6} + \frac{n^2}{12} = \frac{n(n + 2)}{12}.$$

A binomial count with the fixed probability $\tfrac12$ has variance $n/4$. The ratio is the design effect, $(n + 2)/3 = 1 + (n - 1)\rho$ with $\rho = v/[\bar{p}(1 - \bar{p})] = \tfrac{1}{12}/\tfrac14 = \tfrac13$. For $n = 10$ the variance is 10 against 2.5, a design effect of 4: ten members of such a cluster carry the information of two and a half independent ones.

(b) By the law of total expectation applied to the indicator of $\{X = k\}$, the probability of $\{X = k\}$ is the expected value of its conditional probability given the random probability $P$, which is the binomial probability with $P$ in place of $q$. For $n = 2$: $P(X = 0) = E[(1 - P)^2] = \int_0^1 (1 - q)^2\,dq = \tfrac13$, $P(X = 2) = E[P^2] = \tfrac13$, and $P(X = 1) = E[2P(1 - P)] = 2(\tfrac12 - \tfrac13) = \tfrac13$. So $X$ is uniform on $\{0, 1, 2\}$, with mean 1 and variance $E[X^2] - 1 = \tfrac{0 + 1 + 4}{3} - 1 = \tfrac23$, which is $\tfrac{2 \cdot 4}{12}$. For $n = 2$, 10 and 50 the script checks numerically that $P(X = k) = 1/(n + 1)$ at $k = 0$, 1, $n/2$ and $n$, and that a uniform count on $\{0, \ldots, n\}$ has variance $n(n + 2)/12$.

Problem 3 (a Poisson count with an exponential rate). The rate $\Lambda$ has the exponential density $\tfrac1\mu e^{-\lambda/\mu}$ on $\lambda > 0$, with mean $\mu$, and given $\Lambda = \lambda$ the count $X$ is Poisson with mean $\lambda$. (a) Find $E[X]$ and $\operatorname{Var}(X)$. (b) Find the distribution of $X$. (c) For $\mu = 3$, compare $P(X = 0)$ and $P(X \ge 10)$ with those of a Poisson count with mean 3.

Solution. (a) The exponential density has $E[\Lambda] = \mu$ and $E[\Lambda^2] = 2\mu^2$ (integrating by parts twice), so $\operatorname{Var}(\Lambda) = \mu^2$. By the overdispersion formula, $E[X] = \mu$ and $\operatorname{Var}(X) = \mu + \mu^2$.

(b) By the law of total expectation applied to the indicator of $\{X = k\}$,

$$P(X = k) = E\Big[e^{-\Lambda}\frac{\Lambda^k}{k!}\Big] = \frac{1}{\mu\,k!}\int_0^\infty \lambda^k e^{-\lambda(1 + 1/\mu)}\,d\lambda.$$

With $c = 1 + 1/\mu$, repeated integration by parts gives $\int_0^\infty \lambda^k e^{-c\lambda}\,d\lambda = k!/c^{k+1}$: each step lowers the power by one and brings out a factor $k/c$, and $\int_0^\infty e^{-c\lambda}\,d\lambda = 1/c$. So

$$P(X = k) = \frac{1}{\mu}\Big(\frac{\mu}{1 + \mu}\Big)^{k+1} = \frac{1}{1 + \mu}\Big(\frac{\mu}{1 + \mu}\Big)^{k}, \qquad k = 0, 1, 2, \ldots$$

Each probability is the previous one times $\mu/(1 + \mu)$: a geometric distribution. It is the negative binomial of the course page with shape 1, whose variance $\mu + \mu^2/1$ agrees with part (a).

(c) With $\mu = 3$: $P(X = 0) = \tfrac14 = 0.25$, against $e^{-3} = 0.0498$ for the Poisson. The tail is $P(X \ge 10) = (\tfrac34)^{10} = 0.0563$, since the probabilities from $k = 10$ on are a geometric series with first term $\tfrac14(\tfrac34)^{10}$ and ratio $\tfrac34$. For the Poisson it is 0.0011. The variance is 12 against 3. A varying rate puts more probability at zero and far more in the upper tail than a fixed rate with the same mean.

Problem 4 (a random sum). The number of claims $N$ in a month is Poisson with mean 4, and the claims are independent of $N$ and of each other, each exponential with mean 2 (so with variance 4). Let $S$ be the month's total. Find $E[S]$, the standard deviation of $S$, and the covariance and correlation of $S$ and $N$.

Solution. By the corollary, $E[S] = 2 \times 4 = 8$ and $\operatorname{Var}(S) = 4 \times 4 + 2^2 \times 4 = 32$, a standard deviation of 5.66. Half the variance comes from the variation of the claims and half from the variation of their number.

For the covariance use Theorem 4 with $N$ as the conditioning variable, $Y = S$ and $Z = N$. Given $N = n$, the variable $N$ is the constant $n$, so $\operatorname{Cov}(S, N \mid N = n) = E[S\,n \mid N = n] - E[S \mid N = n]\,n = 0$, by taking out what is known. And $E[S \mid N] = 2N$, $E[N \mid N] = N$. So

$$\operatorname{Cov}(S, N) = 0 + \operatorname{Cov}(2N, N) = 2\operatorname{Var}(N) = 8,$$

and the correlation is $8/\sqrt{32 \times 4} = 0.7071$, which is $1/\sqrt2$. The simulation reported under "What the experiment shows" gave a mean of 8.001 and a variance of 31.99 for $S$ (medians over 30 seeds).

Problem 5 (two conditional means given the same event). Let $X$ and $Y$ be independent, each with density $e^{-x}$ on $x > 0$, and let $D = X - Y$ and $R = X/Y$. Find $E[X \mid D = 0]$ and $E[X \mid R = 1]$.

Solution. We need the joint density of $(X, D)$ and of $(X, R)$. For $a < b$ and $u < v$, the event $\{u < D \le v\}$ is $\{X - v \le Y < X - u\}$. By the independence of $X$ and $Y$, and the substitution $y = x - w$ in the inner integral,

$$P(a < X \le b,\ u < D \le v) = \int_a^b f_X(x)\Big[\int_{x - v}^{x - u} f_Y(y)\,dy\Big]dx = \int_a^b\!\!\int_u^v f_X(x)\,f_Y(x - w)\,dw\,dx.$$

So $(X, D)$ has the joint density $f_{X,D}(x, w) = f_X(x)\,f_Y(x - w) = e^{-x}e^{-(x - w)}$ for $x > 0$ and $x - w > 0$, by the definition of a joint density. At $w = 0$ it is $e^{-2x}$, and the marginal density of $D$ at 0 is $\int_0^\infty e^{-2x}\,dx = \tfrac12$. The conditional density of $X$ given $D = 0$ is therefore $e^{-2x}/\tfrac12 = 2e^{-2x}$, an exponential density with mean

$$E[X \mid D = 0] = \int_0^\infty x \cdot 2e^{-2x}\,dx = \frac12.$$

For the ratio, both variables are positive, so take $0 \le a < b$ and $0 < u < v$. The event $\{u < R \le v\}$ is $\{X/v \le Y < X/u\}$, and the substitution $y = x/r$, with $dy = -(x/r^2)\,dr$, gives

$$P(a < X \le b,\ u < R \le v) = \int_a^b f_X(x)\Big[\int_{x/v}^{x/u} f_Y(y)\,dy\Big]dx = \int_a^b\!\!\int_u^v f_X(x)\,f_Y\Big(\frac{x}{r}\Big)\frac{x}{r^2}\,dr\,dx.$$

So $f_{X,R}(x, r) = e^{-x}e^{-x/r}\,x/r^2$ for $x > 0$ and $r > 0$. At $r = 1$ it is $x\,e^{-2x}$, and the marginal density of $R$ at 1 is $\int_0^\infty x\,e^{-2x}\,dx = \tfrac14$. The conditional density of $X$ given $R = 1$ is $4x\,e^{-2x}$, and

$$E[X \mid R = 1] = \int_0^\infty 4x^2e^{-2x}\,dx = 4 \cdot \frac{2}{2^3} = 1,$$

using $\int_0^\infty x^k e^{-cx}\,dx = k!/c^{k+1}$ from Problem 3. The factor $x/r^2$ is the reason for the difference: a given small change in the ratio corresponds to a change in $Y$ proportional to $x$, so the wedge around the diagonal is wider where $x$ is large. The unconditional mean of $X$ is 1. Conditioning through the difference halves it, and conditioning through the ratio happens to leave it at 1.

Summary

Result Statement
Given an event $E[Y \mid A] = E[Y\,\mathbf{1}_A]/P(A)$
Given a variable $E[Y \mid X = x]$ is $\dfrac{E[Y\,\mathbf{1}\{X = x\}]}{P(X = x)}$, or $\int y\,f_{Y \mid X}(y \mid x)\,dy$, or $\sum_y y\,p_{Y \mid X}(y \mid x)$, or atoms plus a density part
Total expectation $E[\psi(X, Y)] = E\big[E[\psi(X, Y) \mid X]\big]$
Taking out what is known $E[g(X)\,Y \mid X] = g(X)\,E[Y \mid X]$
Independence $E[\psi(x, Y) \mid X = x] = E[\psi(x, Y)]$
Best predictor $E[(Y - g(X))^2] = E[(Y - E[Y \mid X])^2] + E[(E[Y \mid X] - g(X))^2]$
Total variance $\operatorname{Var}(Y) = E[\operatorname{Var}(Y \mid X)] + \operatorname{Var}(E[Y \mid X])$
Total covariance $\operatorname{Cov}(Y, Z) = E[\operatorname{Cov}(Y, Z \mid X)] + \operatorname{Cov}(E[Y \mid X], E[Z \mid X])$
Tower $E\big[E[Y \mid X, Z] \mid X\big] = E[Y \mid X]$
Clusters $\operatorname{Var}(X) = n\bar{p}(1 - \bar{p})[1 + (n - 1)\rho]$
Varying rate $\operatorname{Var}(X) = \mu + \operatorname{Var}(\Lambda)$
Random sum $E[S] = \mu E[N]$; $\operatorname{Var}(S) = \sigma^2E[N] + \mu^2\operatorname{Var}(N)$

What we have learned

The conditional expectation $E[Y \mid X]$ is a function of $X$: the mean of $Y$ among the outcomes that share a value of $X$, defined by a ratio when that value has positive probability and by a conditional density when it does not. One theorem, the law of total expectation, was proved separately for a discrete conditioning variable, for a joint density, for a count with a random parameter and for a conditional distribution with atoms. The best-predictor property and the laws of total variance and covariance follow from it without returning to those cases. The error $Y - E[Y \mid X]$ is uncorrelated with every function of $X$, so the conditional expectation is the best predictor of $Y$ from $X$ in mean square, and the variance of $Y$ splits into a part within the values of $X$ and a part between them. For systolic pressure given age the split is 304.12 and 100.88, and the best straight line comes within 4.99 of the best possible error. The design effect of clusters, the overdispersion of a count with a varying rate and the variance of a random sum are the same law read in three models. Conditioning on an event of probability zero has a meaning only through the variable that defines it.

Expected values and variances are averages over a distribution. The next article, Probability Inequalities: Markov, Chebyshev, Jensen and Cauchy–Schwarz, shows how much an average alone says about the probabilities of a distribution, which is the step from these three articles to the laws of large numbers.

The numbers in this article are computed and checked by tools/experiments/statistics/conditional-expectation.py in the site's repository, from the NHANES files loaded by tools/experiments/applied-statistics/statdata.py. The figure is drawn by tools/figures/conditional-expectation.py.

References

  • Billingsley, P. (1995). Probability and Measure (3rd ed.). Wiley. Sections 18 ("Product Measure and Fubini's Theorem"), 32 ("The Radon–Nikodym Theorem"), 33 ("Conditional Probability"), 34 ("Conditional Expectation") and 35 ("Martingales"). Section numbers and titles are those of the contents of the 2012 Anniversary Edition, which reissues the third edition. record · 8 pages
  • Casella, G., and Berger, R. L. (2002). Statistical Inference (2nd ed.). Duxbury. Sections 4.2 ("Conditional Distributions and Independence") and 4.4 ("Hierarchical Models and Mixture Distributions"). record · 11 pages
  • Grinstead, C. M., and Snell, J. L. (1997). Introduction to Probability (2nd rev. ed.). American Mathematical Society. Section 4.2 (continuous conditional probability) and Section 6.1 (Definition 6.2 and Theorem 6.5). record · 20 pages
  • Kolmogoroff, A. (1933). Grundbegriffe der Wahrscheinlichkeitsrechnung. Springer. The chapter "Bedingte Wahrscheinlichkeiten und Erwartungen", pp. 41–50. Read in the English translation edited by N. Morrison, Foundations of the Theory of Probability (Chelsea, 1950): Chapter V, "Conditional Probabilities and Mathematical Expectations", pp. 47–56, with §2, "Explanation of a Borel Paradox" (pp. 50–51), and §4, "Conditional Mathematical Expectations" (pp. 52–56). record
  • National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018: Demographic Variables and Sample Weights (DEMO_J) and Blood Pressure (BPX_J) data files. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 67 pages
  • Proschan, M. A., and Presnell, B. (1998). Expect the unexpected from conditional expectation. The American Statistician, 52(3), 248–252. https://doi.org/10.1080/00031305.1998.10480576 record
  • Wald, A. (1944). On cumulative sums of random variables. The Annals of Mathematical Statistics, 15(3), 283–296. https://doi.org/10.1214/aoms/1177731235 record