Student's $t$ distribution with 3 degrees of freedom has variance 3, but its tails are so heavy that its third absolute moment is infinite and its moment generating function is infinite at every point except 0. The proof of the central limit theorem on the course page Moment Generating Functions and Slutsky's Theorem assumes a moment generating function, so it says nothing about means of draws from this distribution. Do they become normal all the same? They do: the largest gap between the distribution function of the standardized mean and the normal one is 0.0869 for one draw, 0.0323 for 10 draws and 0.0035 for 1,000, and the theorem proved in this article says that it tends to 0, using nothing about the population beyond its finite variance.
The central limit theorem in the form proved here is the Lindeberg–Lévy theorem. Let $X_1, X_2, \ldots$ be independent and identically distributed (i.i.d.) random variables with mean $\mu$ and finite variance $\sigma^2 > 0$, let $\bar{X}_n$ be the mean of the first $n$, and let $Z_n = \sqrt{n}\,(\bar{X}_n - \mu)/\sigma$ be the standardized mean. Then $P(Z_n \le z) \to \Phi(z)$ for every $z$, where $\Phi$ is the standard normal distribution function. The kind of convergence in that statement is convergence in distribution, one of four ways in which a sequence of random variables can converge.
First we define the four modes of convergence and prove how they are related. Then we read convergence in distribution closely and prove the tools that work with it: the criterion by bounded continuous functions, the continuous mapping theorem and Slutsky's theorem in its general form. Next we prove the central limit theorem with characteristic functions, citing two theorems about them. Finally we check each step on two populations and collect the consequences that later articles use. The article assumes Random Variables and Distribution Functions, Expectation, Variance and Covariance and Moment Generating and Characteristic Functions.
This article is the home of the modes of convergence, the continuous mapping theorem and Slutsky's theorem for the statistics collection. The Central Limit Theorem states the theorem, measures its speed, shows where it fails and gives its history; none of that is repeated here.
Four ways a sequence can converge
A sequence of numbers either converges to a limit or does not. A sequence of random variables can settle down in several different senses, and the laws of large numbers and the central limit theorem use different ones. Throughout, $X_1, X_2, \ldots$ and $X$ are random variables, with distribution functions $F_n(x) = P(X_n \le x)$ and $F(x) = P(X \le x)$. For the first three definitions they must all be defined on the same sample space $\Omega$, because the definitions compare the values $X_n(\omega)$ and $X(\omega)$ at the same outcome $\omega$.
- Almost surely. $X_n \xrightarrow{\text{a.s.}} X$ if the event that the numbers $X_n(\omega)$ converge to $X(\omega)$ has probability 1.
- In probability. $X_n \xrightarrow{p} X$ if for every $\varepsilon > 0$, $P(\lvert X_n - X\rvert > \varepsilon) \to 0$ as $n \to \infty$.
- In $r$th mean. For $r > 0$, $X_n$ converges to $X$ in $r$th mean if $E[\lvert X_n - X\rvert^r] \to 0$. The case $r = 2$ is convergence in mean square and $r = 1$ is convergence in mean.
- In distribution. $X_n \xrightarrow{d} X$ if $F_n(x) \to F(x)$ at every point $x$ where $F$ is continuous.
The course page writes convergence in probability with $\ge \varepsilon$ in place of $> \varepsilon$. The two forms define the same thing, because $P(\lvert X_n - X\rvert > \varepsilon) \le P(\lvert X_n - X\rvert \ge \varepsilon) \le P(\lvert X_n - X\rvert > \varepsilon/2)$, and each definition is required to hold for every $\varepsilon$.
The fourth definition differs in kind from the others. It mentions only the distribution functions, so it makes sense when the $X_n$ live on different sample spaces, and the limit is really a distribution, not a particular random variable. If $X_n \xrightarrow{d} X$ and $X'$ has the same distribution as $X$, then $X_n \xrightarrow{d} X'$ as well. It is common to name the limit by its distribution, as in "$Z_n$ converges in distribution to the standard normal".
The sample mean shows all four at work. Let the $X_i$ be i.i.d. with mean $\mu$ and variance $\sigma^2$.
- $\bar{X}_n$ converges to $\mu$ in mean square, because $E[(\bar{X}_n - \mu)^2] = \operatorname{Var}(\bar{X}_n) = \sigma^2/n \to 0$.
- $\bar{X}_n \xrightarrow{p} \mu$. That is the weak law of large numbers.
- $\bar{X}_n \xrightarrow{\text{a.s.}} \mu$. That is the strong law of large numbers. Both laws are the subject of The Laws of Large Numbers.
- $Z_n \xrightarrow{d} Z$ with $Z$ standard normal. That is the central limit theorem.
Example (the standardized mean converges in distribution and in no stronger sense). Suppose the population is normal. Then $Z_n$ is exactly standard normal for every $n$, so it converges in distribution trivially. It does not converge in probability to any random variable. If it did, with limit $Z$, then $Z_{2n} - Z_n$ would converge in probability to 0, because $\lvert Z_{2n} - Z_n\rvert > \varepsilon$ requires $\lvert Z_{2n} - Z\rvert > \varepsilon/2$ or $\lvert Z_n - Z\rvert > \varepsilon/2$. But split the first $2n$ observations into two halves and let $Z_n'$ be the standardized mean of the second half. Then $Z_{2n} = (Z_n + Z_n')/\sqrt{2}$, so
$$Z_{2n} - Z_n = \frac{1}{\sqrt{2}}\,Z_n' - \Big(1 - \frac{1}{\sqrt{2}}\Big)Z_n.$$
The two halves are independent, so this is a normal variable with mean 0 and variance $\tfrac12 + (1 - 1/\sqrt{2})^2 = 2 - \sqrt{2}$, the same for every $n$. The probability $P(\lvert Z_{2n} - Z_n\rvert > \varepsilon)$ is therefore a fixed positive number and does not tend to 0. Each new block of observations reshuffles the standardized mean; only its distribution is stable.
Which of the four modes imply which is the subject of the next section.
How the modes are related
Here is the whole picture in one sentence: almost sure convergence and convergence in $r$th mean each imply convergence in probability, convergence in probability implies convergence in distribution, and no other implication holds in general, except that convergence in distribution to a constant gives convergence in probability back.
Theorem (relations among the modes). Let $X_n$ and $X$ be random variables on one sample space.
- If $X_n \xrightarrow{\text{a.s.}} X$, then $X_n \xrightarrow{p} X$.
- If $X_n \to X$ in $r$th mean for some $r > 0$, then $X_n \xrightarrow{p} X$.
- If $X_n \xrightarrow{p} X$, then $X_n \xrightarrow{d} X$.
- If $X_n \xrightarrow{d} c$ for a constant $c$, then $X_n \xrightarrow{p} c$.
Proof of 1. The idea: if the sequence converges at an outcome, then at that outcome it is eventually within $\varepsilon$ of the limit and stays there. Fix $\varepsilon > 0$ and let $B_n$ be the event that $\lvert X_m - X\rvert > \varepsilon$ for at least one $m \ge n$. These events decrease, $B_1 \supseteq B_2 \supseteq \cdots$. An outcome that lies in every $B_n$ has $\lvert X_m - X\rvert > \varepsilon$ for infinitely many $m$, so at that outcome $X_m$ does not converge to $X$. The intersection of the $B_n$ is therefore contained in an event of probability 0. Probability is continuous along a decreasing sequence of events: $P(B_n)$ tends to the probability of the intersection. That is the lemma on the continuity of probability in Random Variables and Distribution Functions, a consequence of the countable additivity of probability (Durrett, 2019, Theorem 1.1.1). So $P(B_n) \to 0$, and since the event $\lvert X_n - X\rvert > \varepsilon$ is contained in $B_n$, $P(\lvert X_n - X\rvert > \varepsilon) \le P(B_n) \to 0$. $\square$
Proof of 2. By Markov's inequality (Probability Inequalities) applied to the nonnegative variable $\lvert X_n - X\rvert^r$,
$$P(\lvert X_n - X\rvert > \varepsilon) = P(\lvert X_n - X\rvert^r > \varepsilon^r) \le \frac{E[\lvert X_n - X\rvert^r]}{\varepsilon^r} \to 0. \qquad \square$$
Proof of 3. The idea: when $X_n$ is within $\varepsilon$ of $X$, the event $X_n \le x$ is caught between the events $X \le x - \varepsilon$ and $X \le x + \varepsilon$, and at a continuity point those two have nearly the same probability. Let $x$ be a point where $F$ is continuous, let $\varepsilon > 0$, and write $q_n = P(\lvert X_n - X\rvert > \varepsilon)$. If $X_n \le x$, then either $X \le x + \varepsilon$ or $\lvert X_n - X\rvert > \varepsilon$. If $X \le x - \varepsilon$, then either $X_n \le x$ or $\lvert X_n - X\rvert > \varepsilon$. Taking probabilities,
$$F(x - \varepsilon) - q_n \ \le\ F_n(x) \ \le\ F(x + \varepsilon) + q_n.$$
Now let $\eta > 0$. Since $F$ is continuous at $x$, there is an $\varepsilon$ with $F(x + \varepsilon) < F(x) + \eta$ and $F(x - \varepsilon) > F(x) - \eta$. For that $\varepsilon$, $q_n < \eta$ for all large $n$, and then $\lvert F_n(x) - F(x)\rvert < 2\eta$. Since $\eta$ was arbitrary, $F_n(x) \to F(x)$. $\square$
Proof of 4. The distribution function of the constant $c$ is 0 below $c$ and 1 from $c$ on, so it is continuous everywhere except at $c$. For $\varepsilon > 0$ the points $c - \varepsilon$ and $c + \varepsilon$ are continuity points, and
$$P(\lvert X_n - c\rvert > \varepsilon) \le P(X_n \le c - \varepsilon) + P(X_n > c + \varepsilon) = F_n(c - \varepsilon) + 1 - F_n(c + \varepsilon) \to 0 + 1 - 1 = 0. \qquad \square$$
No arrow can be added to the picture. Three examples show it. In the second and third, $U$ is a random variable uniformly distributed on the interval from 0 to 1.
Example (in distribution but not in probability). Let $X$ be standard normal and $X_n = -X$ for every $n$. By symmetry $-X$ is standard normal too, so $F_n = F$ and $X_n \xrightarrow{d} X$. But $\lvert X_n - X\rvert = 2\lvert X\rvert$, and $P(2\lvert X\rvert > \varepsilon)$ is a positive number that does not depend on $n$. The example of the standardized mean above is of the same kind.
Example (in probability and in every mean, but not almost surely). Cut the unit interval into halves, then quarters, then eighths, and let the indicator of each piece in turn be the next term of the sequence. Precisely, write each $n \ge 1$ as $n = 2^k + j$ with $0 \le j < 2^k$, and let $X_n = 1$ if $j/2^k \le U < (j + 1)/2^k$ and $X_n = 0$ otherwise. Then $P(X_n \ne 0) = 2^{-k}$, which tends to 0 as $n$ grows, so $X_n \xrightarrow{p} 0$. Also $E[\lvert X_n\rvert^r] = 2^{-k} \to 0$ for every $r > 0$. But for every value of $U$ and every $k$, exactly one of the $2^k$ pieces of stage $k$ contains $U$. So the sequence $X_n(\omega)$ takes the value 1 once in every stage and the value 0 at the other terms, infinitely often each, and converges at no outcome.
Example (almost surely but not in mean). Let $X_n = n$ if $U < 1/n$ and $X_n = 0$ otherwise. For any outcome with $U > 0$, $X_n = 0$ as soon as $n \ge 1/U$, so $X_n \to 0$. The event $U > 0$ has probability 1, so $X_n \xrightarrow{\text{a.s.}} 0$, and by part 1 also in probability. But $E[X_n] = n \cdot \tfrac{1}{n} = 1$ for every $n$, so $E[\lvert X_n - 0\rvert]$ does not tend to 0. The same holds for every $r \ge 1$, since $E[\lvert X_n\rvert^r] = n^{r-1}$. For $r < 1$ this particular sequence does converge in $r$th mean, and a steeper one is needed: replacing $n$ by $n^c$ with $c \ge 1/r$ gives $E[\lvert X_n\rvert^r] = n^{cr - 1}$, which does not tend to 0. The first worked problem works that family through for every $c$ and $r$.
| From | To | Holds? | Reason |
|---|---|---|---|
| Almost surely | In probability | Yes | Part 1 |
| In $r$th mean | In probability | Yes | Part 2 |
| In probability | In distribution | Yes | Part 3 |
| In distribution | In probability | Only for a constant limit | Part 4; first example |
| In probability | Almost surely | No | Second example |
| In $r$th mean | Almost surely | No | Second example |
| Almost surely | In $r$th mean | No | Third example for $r \ge 1$; Problem 1 for any $r$ |
| In probability | In $r$th mean | No | Third example for $r \ge 1$; Problem 1 for any $r$ |
One partial converse is cited, not proved here: a sequence that converges in probability has a subsequence that converges almost surely (Durrett, 2019, Theorem 2.3.2). In the second example the subsequence $X_1, X_2, X_4, X_8, \ldots$ of first pieces converges to 0 at every outcome with $U > 0$.
Convergence in distribution, read closely
The definition asks for $F_n(x) \to F(x)$ only at continuity points of $F$. That restriction looks like a technicality, and the first thing to settle is why it is there.
Example (why the continuity points). Let $X_n$ be the constant $1/n$. Any reasonable definition should say that $X_n$ converges to the constant 0. The distribution function of the limit is $F(x) = 0$ for $x < 0$ and $F(x) = 1$ for $x \ge 0$. For $x \ne 0$, $F_n(x) \to F(x)$. But $F_n(0) = P(1/n \le 0) = 0$ for every $n$, while $F(0) = 1$. Convergence fails at exactly one point, the jump of $F$, and the definition sets that point aside. With the constants $-1/n$ in place of $1/n$ it would hold there, which shows that what happens at a jump depends on the side from which the mass arrives.
The restriction costs little, because continuity points are plentiful. A distribution function has at most $k$ jumps of size $1/k$ or more, since the jumps add up to at most 1. So it has at most countably many jumps in all, and every interval, however short, contains continuity points. When $F$ is continuous, as $\Phi$ is, every point is a continuity point.
Three consequences of the definition are used repeatedly below.
Lemma 1 (strict inequalities and intervals). If $X_n \xrightarrow{d} X$ and $x$ is a continuity point of $F$, then $P(X_n < x) \to F(x)$ and $P(X_n > x) \to 1 - F(x)$. If $a < b$ are continuity points, then $P(a < X_n \le b) \to F(b) - F(a)$.
Proof. Let $\eta > 0$. Choose $\delta > 0$ such that $x - \delta$ is a continuity point of $F$ and $F(x - \delta) > F(x) - \eta$; that is possible because $F$ is continuous at $x$ and continuity points lie in every interval. Then $F_n(x - \delta) \le P(X_n < x) \le F_n(x)$. The left side tends to $F(x - \delta)$ and the right side to $F(x)$, so for all large $n$, $F(x) - 2\eta < P(X_n < x) < F(x) + \eta$. Hence $P(X_n < x) \to F(x)$. The second statement follows from $P(X_n > x) = 1 - F_n(x)$, and the third from $P(a < X_n \le b) = F_n(b) - F_n(a)$. $\square$
Lemma 2 (bounded in probability). If $X_n \xrightarrow{d} X$, then for every $\eta > 0$ there is a number $M$ with $P(\lvert X_n\rvert > M) < \eta$ for every $n$.
Proof. Since $F(x) \to 0$ as $x \to -\infty$ and $F(x) \to 1$ as $x \to \infty$, there is a number $M_0 > 0$ such that $\pm M_0$ are continuity points of $F$ and $F(-M_0) + 1 - F(M_0) < \eta/2$. Then $P(\lvert X_n\rvert > M_0) \le F_n(-M_0) + 1 - F_n(M_0)$, which tends to a limit below $\eta/2$. So there is an $N$ with $P(\lvert X_n\rvert > M_0) < \eta$ for all $n \ge N$. Each of the finitely many variables $X_1, \ldots, X_{N-1}$ has its own number $M_n$ with $P(\lvert X_n\rvert > M_n) < \eta$, for the same reason that $M_0$ exists. Let $M$ be the largest of $M_0, M_1, \ldots, M_{N-1}$. $\square$
A sequence with the property in Lemma 2 is called bounded in probability. The lemma says that no probability escapes to infinity along a sequence that converges in distribution.
Theorem (uniform convergence to a continuous limit). If $X_n \xrightarrow{d} X$ and $F$ is continuous, then $\sup_x \lvert F_n(x) - F(x)\rvert \to 0$.
The idea: pin $F_n$ to $F$ at finitely many points that cut the probability into $k$ equal parts; between two neighboring points both functions are trapped, because both are nondecreasing.
Proof. Fix a whole number $k \ge 2$. Since $F$ is continuous and runs from 0 to 1, the intermediate value theorem gives points $x_1 < x_2 < \cdots < x_{k-1}$ with $F(x_j) = j/k$. There is an $N$ such that $\lvert F_n(x_j) - F(x_j)\rvert < 1/k$ for all $j$ and all $n \ge N$. Take $n \ge N$ and any $x$. If $x_{j-1} \le x \le x_j$ for some $j$ from 2 to $k - 1$, then
$$F_n(x) \le F_n(x_j) < F(x_j) + \frac{1}{k} = F(x_{j-1}) + \frac{2}{k} \le F(x) + \frac{2}{k},$$
$$F_n(x) \ge F_n(x_{j-1}) > F(x_{j-1}) - \frac{1}{k} = F(x_j) - \frac{2}{k} \ge F(x) - \frac{2}{k}.$$
If $x \le x_1$, then $F_n(x) \le F_n(x_1) < F(x_1) + 1/k = 2/k \le F(x) + 2/k$, and $F_n(x) \ge 0 = F(x_1) - 1/k \ge F(x) - 1/k$. If $x \ge x_{k-1}$, then $F_n(x) \ge F_n(x_{k-1}) > F(x_{k-1}) - 1/k = 1 - 2/k \ge F(x) - 2/k$, and $F_n(x) \le 1 = F(x_{k-1}) + 1/k \le F(x) + 1/k$. So $\lvert F_n(x) - F(x)\rvert < 2/k$ for every $x$ once $n \ge N$. Since $k$ was arbitrary, the supremum tends to 0. $\square$
This is why the "largest gap" between $F_n$ and $\Phi$, reported in the opening and tabulated in The Central Limit Theorem, is a fair measure of the theorem: convergence at each $z$ to a continuous limit forces the largest gap over all $z$ to vanish. Durrett (2019) sets the statement as Exercise 3.2.9.
The criterion by bounded continuous functions
The definition by distribution functions is tied to the real line and to events of the form $X \le x$. An equivalent form uses expected values instead, and it is the one that makes the next two theorems short.
Theorem (bounded continuous functions). $X_n \xrightarrow{d} X$ if and only if $E[h(X_n)] \to E[h(X)]$ for every bounded continuous function $h$ on the real line.
The idea of the first half: a continuous function is nearly constant on short intervals, so its expected value is nearly a finite sum of values of $h$ times probabilities of intervals, and those probabilities converge. The idea of the second half: the indicator of $X \le x$ is not continuous, but it lies between two continuous ramps that differ only near $x$.
Proof. Suppose $X_n \xrightarrow{d} X$, and let $h$ be continuous with $\lvert h\rvert \le K$. Let $\eta > 0$. Choose continuity points $a < b$ of $F$ with $F(a) + 1 - F(b) < \eta$. On the closed interval from $a$ to $b$, $h$ is uniformly continuous, so there are continuity points $a = x_0 < x_1 < \cdots < x_m = b$ of $F$, close enough together that $\lvert h(x) - h(x_j)\rvert < \eta$ whenever $x_{j-1} < x \le x_j$. Define the step function $g(x) = h(x_j)$ for $x_{j-1} < x \le x_j$, and $g(x) = 0$ for $x \le a$ and for $x > b$. Then $\lvert h - g\rvert < \eta$ on the interval $a < x \le b$ and $\lvert h - g\rvert \le K$ off it. For any random variable $Y$, by the monotonicity of expectation,
$$\big\lvert E[h(Y)] - E[g(Y)]\big\rvert \le \eta + K\,P(Y \le a \text{ or } Y > b).$$
For $Y = X$ the probability on the right is $F(a) + 1 - F(b) < \eta$. For $Y = X_n$ it is $F_n(a) + 1 - F_n(b)$, which tends to the same number, so it is below $\eta$ for all large $n$. The step function takes finitely many values, so
$$E[g(X_n)] = \sum_{j=1}^{m} h(x_j)\,\big(F_n(x_j) - F_n(x_{j-1})\big) \to \sum_{j=1}^{m} h(x_j)\,\big(F(x_j) - F(x_{j-1})\big) = E[g(X)].$$
Putting the three pieces together, for all large $n$,
$$\big\lvert E[h(X_n)] - E[h(X)]\big\rvert \le (\eta + K\eta) + \eta + (\eta + K\eta) = (3 + 2K)\,\eta.$$
Since $\eta$ was arbitrary, $E[h(X_n)] \to E[h(X)]$.
Conversely, suppose $E[h(X_n)] \to E[h(X)]$ for every bounded continuous $h$. Let $x$ be a continuity point of $F$ and $\varepsilon > 0$. Let $h^+$ be the function that is 1 on $u \le x$, 0 on $u \ge x + \varepsilon$ and linear in between, and let $h^-$ be 1 on $u \le x - \varepsilon$, 0 on $u \ge x$ and linear in between. Both are bounded and continuous, and for every $u$
$$\mathbf{1}\{u \le x - \varepsilon\} \le h^-(u) \le \mathbf{1}\{u \le x\} \le h^+(u) \le \mathbf{1}\{u \le x + \varepsilon\}.$$
Taking expected values with $u = X_n$ and with $u = X$, $E[h^-(X_n)] \le F_n(x) \le E[h^+(X_n)]$, and the outer terms tend to $E[h^-(X)] \ge F(x - \varepsilon)$ and $E[h^+(X)] \le F(x + \varepsilon)$. Let $\eta > 0$ and choose $\varepsilon$ with $F(x + \varepsilon) < F(x) + \eta$ and $F(x - \varepsilon) > F(x) - \eta$. Then for all large $n$, $F(x) - 2\eta < F_n(x) < F(x) + 2\eta$. So $F_n(x) \to F(x)$. $\square$
Durrett (2019) gives this equivalence as Theorem 3.2.9, with a different proof. It is the form in which the definition extends to random vectors: a sequence of random vectors converges in distribution if $E[h(X_n)] \to E[h(X)]$ for every bounded continuous $h$ of several variables.
Theorem (continuous mapping). Let $g$ be a continuous function on the real line.
- If $X_n \xrightarrow{d} X$, then $g(X_n) \xrightarrow{d} g(X)$.
- If $X_n \xrightarrow{p} X$, then $g(X_n) \xrightarrow{p} g(X)$.
- If $X_n \xrightarrow{p} c$ for a constant $c$, and $g$ is any function that is continuous at the single point $c$, then $g(X_n) \xrightarrow{p} g(c)$.
Proof. For part 1, let $h$ be bounded and continuous. Then the composition $h(g(\cdot))$ is bounded and continuous, so $E[h(g(X_n))] \to E[h(g(X))]$ by the previous theorem. That holds for every such $h$, so by the previous theorem again $g(X_n) \xrightarrow{d} g(X)$.
For part 3, let $\varepsilon > 0$. Continuity at $c$ gives a $\delta > 0$ such that $\lvert u - c\rvert \le \delta$ implies $\lvert g(u) - g(c)\rvert \le \varepsilon$. So $P(\lvert g(X_n) - g(c)\rvert > \varepsilon) \le P(\lvert X_n - c\rvert > \delta) \to 0$.
For part 2, let $\varepsilon > 0$ and $\eta > 0$. Choose $M$ with $P(\lvert X\rvert > M) < \eta$. On the closed interval from $-M - 1$ to $M + 1$, $g$ is uniformly continuous, so there is a $\delta$ with $0 < \delta \le 1$ such that two points of that interval within $\delta$ of each other have values of $g$ within $\varepsilon$. If $\lvert X\rvert \le M$ and $\lvert X_n - X\rvert \le \delta$, both $X$ and $X_n$ lie in the interval, so $\lvert g(X_n) - g(X)\rvert \le \varepsilon$. Hence
$$P(\lvert g(X_n) - g(X)\rvert > \varepsilon) \le P(\lvert X\rvert > M) + P(\lvert X_n - X\rvert > \delta) < \eta + P(\lvert X_n - X\rvert > \delta),$$
and the last probability tends to 0. So the left side is below $2\eta$ for all large $n$. $\square$
Part 1 holds more generally, for a function $g$ whose set of discontinuities has probability 0 under the distribution of $X$. That version is cited, not proved here (Durrett, 2019, Theorem 3.2.10). Durrett's Theorem 2.3.4 is part 2.
Slutsky's theorem
The continuous mapping theorem handles a function of one sequence. Statistics constantly needs two sequences at once: a standardized estimate that converges in distribution, and a nuisance quantity, such as an estimated standard deviation, that converges in probability to a constant. Slutsky's theorem says that the constant may then be treated as if it were known.
Example (two limits in distribution do not combine). Convergence in distribution of $X_n$ and of $Y_n$ separately says nothing about $X_n + Y_n$. Let $X$ be standard normal, $X_n = X$ and $Y_n = -X$. Then $X_n \xrightarrow{d} X$ and $Y_n \xrightarrow{d} X$, but $X_n + Y_n = 0$, which is not the distribution of the sum of two independent normals, nor of $X + X$. The joint behavior of the two sequences matters. When one limit is a constant there is no joint behavior to specify, and that is the case Slutsky's theorem covers.
Lemma 3 (converging together). If $X_n \xrightarrow{d} X$ and $D_n \xrightarrow{p} 0$, then $X_n + D_n \xrightarrow{d} X$.
Proof. The argument is that of part 3 of the theorem on the modes. Let $x$ be a continuity point of $F$ and $\eta > 0$. Choose $\varepsilon > 0$ such that $x - \varepsilon$ and $x + \varepsilon$ are continuity points of $F$ with $F(x + \varepsilon) < F(x) + \eta$ and $F(x - \varepsilon) > F(x) - \eta$. Write $q_n = P(\lvert D_n\rvert > \varepsilon)$. If $X_n + D_n \le x$, then $X_n \le x + \varepsilon$ or $\lvert D_n\rvert > \varepsilon$. If $X_n \le x - \varepsilon$, then $X_n + D_n \le x$ or $\lvert D_n\rvert > \varepsilon$. So
$$F_n(x - \varepsilon) - q_n \le P(X_n + D_n \le x) \le F_n(x + \varepsilon) + q_n.$$
As $n \to \infty$ the left side tends to $F(x - \varepsilon) > F(x) - \eta$ and the right side to $F(x + \varepsilon) < F(x) + \eta$. So $\lvert P(X_n + D_n \le x) - F(x)\rvert < 2\eta$ for all large $n$. $\square$
Lemma 4 (bounded times vanishing). If the sequence $X_n$ is bounded in probability and $R_n \xrightarrow{p} 0$, then $X_nR_n \xrightarrow{p} 0$.
Proof. Let $\varepsilon > 0$ and $\eta > 0$, and choose $M > 0$ with $P(\lvert X_n\rvert > M) < \eta$ for every $n$. If $\lvert X_nR_n\rvert > \varepsilon$, then $\lvert X_n\rvert > M$ or $\lvert R_n\rvert > \varepsilon/M$. So $P(\lvert X_nR_n\rvert > \varepsilon) < \eta + P(\lvert R_n\rvert > \varepsilon/M)$, which is below $2\eta$ for all large $n$. $\square$
Theorem (Slutsky). Let $X_n \xrightarrow{d} X$, $A_n \xrightarrow{p} a$ and $B_n \xrightarrow{p} b$, where $a$ and $b$ are constants. Then
$$A_nX_n + B_n \xrightarrow{d} aX + b.$$
If $a \ne 0$, then also $X_n/A_n \xrightarrow{d} X/a$, where $X_n/A_n$ is given any fixed value on the event $A_n = 0$.
Proof. Write
$$A_nX_n + B_n = (aX_n + b) + (A_n - a)\,X_n + (B_n - b).$$
The first term converges in distribution to $aX + b$, by the continuous mapping theorem with the continuous function $g(u) = au + b$. In the second term, $X_n$ is bounded in probability by Lemma 2 and $A_n - a \xrightarrow{p} 0$, so the product converges in probability to 0 by Lemma 4. The third term converges in probability to 0 by hypothesis. A sum of two sequences that converge in probability to 0 does the same, because $\lvert U + V\rvert > \varepsilon$ requires $\lvert U\rvert > \varepsilon/2$ or $\lvert V\rvert > \varepsilon/2$. So the second and third terms together are a sequence $D_n \xrightarrow{p} 0$, and Lemma 3 gives the result.
For the quotient, let $g(u) = 1/u$ for $u \ne 0$ and $g(0) = 0$. This function is continuous at $a \ne 0$, so $g(A_n) \xrightarrow{p} 1/a$ by part 3 of the continuous mapping theorem. Apply the first statement with $g(A_n)$ in place of $A_n$ and $B_n = 0$: $g(A_n)X_n \xrightarrow{d} X/a$. The quotient $X_n/A_n$, with any fixed value assigned where $A_n = 0$, differs from $g(A_n)X_n$ only on the event $A_n = 0$, and $P(A_n = 0) \le P(\lvert A_n - a\rvert \ge \lvert a\rvert) \to 0$. Two sequences that differ only on events of vanishing probability differ by a sequence that converges in probability to 0, so Lemma 3 applies once more. $\square$
The course page proves the case $a = 1$, $b = 0$ with a normal limit, by a direct argument, and cites the general statement. The proof above is the general statement. Durrett (2019) sets its two halves as Exercises 3.2.13 and 3.2.14, and the course page cites van der Vaart (1998) for it under the name Slutsky's lemma; his Chapter 2, "Stochastic Convergence", reviews convergence in distribution and in probability for random vectors.
Characteristic functions and the two cited theorems
The central limit theorem is a statement about distribution functions of sums, and distribution functions of sums are unwieldy: the distribution of a sum of $n$ independent variables is an $n$-fold convolution. The proof therefore changes the object. The characteristic function of a random variable $X$ is
$$\varphi_X(t) = E\big[e^{itX}\big] = E[\cos tX] + i\,E[\sin tX], \qquad t \text{ real},$$
where $i^2 = -1$. Moment Generating and Characteristic Functions is its home on this site, and proves the properties used here:
- It always exists. Cosine and sine are bounded, so both expected values are finite for every random variable, and $\lvert\varphi_X(t)\rvert \le 1$. This is the advantage over the moment generating function $E[e^{tX}]$, which is infinite for heavy-tailed variables.
- Linear changes. $\varphi_{aX + b}(t) = e^{itb}\,\varphi_X(at)$.
- Independent sums. If $X$ and $Y$ are independent, $\varphi_{X + Y}(t) = \varphi_X(t)\,\varphi_Y(t)$. A convolution of distributions becomes a product of functions.
- The standard normal. If $Z$ is standard normal, $\varphi_Z(t) = e^{-t^2/2}$.
Two deeper theorems are stated in that article and cited, not proved, there or here.
- The uniqueness theorem. If $\varphi_X(t) = \varphi_Y(t)$ for every $t$, then $X$ and $Y$ have the same distribution. It follows from the inversion formula, which recovers the distribution from the characteristic function (Durrett, 2019, Theorem 3.3.11).
- The continuity theorem. If $\varphi_{X_n}(t) \to \varphi_X(t)$ for every $t$, where $X$ is a random variable, then $X_n \xrightarrow{d} X$. This is Lévy's continuity theorem (Durrett, 2019, Theorem 3.3.17, which proves more: it is enough that the limit function be continuous at 0, and then it is the characteristic function of some random variable).
The converse of the continuity theorem needs no citation. If $X_n \xrightarrow{d} X$, then $\varphi_{X_n}(t) \to \varphi_X(t)$ for every $t$, because $\cos tx$ and $\sin tx$ are bounded continuous functions of $x$ and the criterion proved above applies to each.
Both cited theorems are proved with measure theory or with Fourier analysis at a comparable level, and this unit does not assume either. The proofs are in Durrett (2019, Section 3.3). Billingsley (1995) is the reference that the course page gives for the general proof of the central limit theorem. Everything else in the proof of the central limit theorem is elementary, and is given in full below.
The estimate that needs only two moments
The proof on the course page expands the moment generating function to second order, with a remainder that it controls through the moment generating function itself. The corresponding expansion of a characteristic function can be controlled by the second moment alone, and that is what removes the need for any further assumption. The expansion is proved in Moment Generating and Characteristic Functions. It is restated here for a standardized variable, with the bound on the remainder that its proof establishes, because the tables below compute that bound.
Lemma 5 (expansion of a standardized characteristic function). Let $Y$ be a random variable with $E[Y] = 0$ and $E[Y^2] = 1$. Then
$$\varphi_Y(s) = 1 - \frac{s^2}{2} + r(s), \qquad \lvert r(s)\rvert \le s^2\,b(s), \qquad b(s) = E\Big[\min\Big(\frac{\lvert s\rvert\,\lvert Y\rvert^3}{6},\ Y^2\Big)\Big],$$
and $b(s) \to 0$ as $s \to 0$, so that $r(s)/s^2 \to 0$.
Proof. This is the case $n = 2$ of two results proved in that article, in its section on moments from a characteristic function. The lemma on the remainder for the complex exponential gives, for every real $x$,
$$\Big\lvert e^{ix} - 1 - ix + \frac{x^2}{2}\Big\rvert \le \min\Big(\frac{\lvert x\rvert^3}{6},\ x^2\Big).$$
The theorem on the expansion of a characteristic function, with $E[Y] = 0$ and $E[Y^2] = 1$, gives $\varphi_Y(s) = 1 - s^2/2 + r(s)$ with $r(s)/s^2 \to 0$. The bound displayed in its proof, for $n = 2$, is $\lvert r(s)\rvert \le s^2\,E[\min(\lvert s\rvert\,\lvert Y\rvert^3/6,\ Y^2)]$, which is $s^2\,b(s)$, and that proof shows that the expected value tends to 0 with $s$: the part of it that comes from large values of $\lvert Y\rvert$ is at most a tail of the second moment, which vanishes because $E[Y^2]$ is finite (the tail fact of that article). $\square$
The first entry of the minimum is the usual size of a second-order Taylor remainder and is the better bound for small $x$. The second grows only like $x^2$ and is the better one when $\lvert x\rvert > 6$. The finite variance is used in one place: it makes the tail term vanish. Nothing is asked of $E[\lvert Y\rvert^3]$, which may be infinite. Durrett's Lemma 3.3.19 and Theorem 3.3.20 are the two steps.
The proof of the central limit theorem
Theorem (Lindeberg–Lévy). Let $X_1, X_2, \ldots$ be independent and identically distributed with mean $\mu$ and finite variance $\sigma^2 > 0$. Then the standardized mean
$$Z_n = \frac{\bar{X}_n - \mu}{\sigma/\sqrt{n}} = \frac{1}{\sqrt{n}}\sum_{i=1}^{n}\frac{X_i - \mu}{\sigma}$$
converges in distribution to the standard normal: $P(Z_n \le z) \to \Phi(z)$ for every $z$, and $\sup_z \lvert P(Z_n \le z) - \Phi(z)\rvert \to 0$.
Here is the plan in one sentence. The characteristic function of $Z_n$ is the $n$th power of one standardized observation's, evaluated at $t/\sqrt{n}$; near 0 that function is $1 - s^2/2$ up to a remainder that is small compared with $s^2$; and the $n$th power of $1 - t^2/(2n)$ tends to $e^{-t^2/2}$, the characteristic function of the normal.
Proof. Step 1: a sum becomes a power. Let $Y_i = (X_i - \mu)/\sigma$. The $Y_i$ are i.i.d. with $E[Y_i] = 0$ and $E[Y_i^2] = 1$; write $\varphi_Y$ for their common characteristic function. Since $Z_n = n^{-1/2}(Y_1 + \cdots + Y_n)$, the rules for independent sums and for linear changes give
$$\varphi_{Z_n}(t) = \Big[\varphi_Y\Big(\frac{t}{\sqrt{n}}\Big)\Big]^n.$$
Step 2: powers of nearby numbers are nearby. If $a$ and $b$ are complex numbers with $\lvert a\rvert \le 1$ and $\lvert b\rvert \le 1$, then $\lvert a^n - b^n\rvert \le n\,\lvert a - b\rvert$. Indeed
$$a^n - b^n = (a - b)\,\big(a^{n-1} + a^{n-2}b + \cdots + ab^{n-2} + b^{n-1}\big),$$
and each of the $n$ terms in the second factor has modulus at most 1.
Step 3: compare with a real sequence. Fix $t$. Let $a_n = \varphi_Y(t/\sqrt{n})$ and $b_n = 1 - t^2/(2n)$. Every characteristic function has modulus at most 1, so $\lvert a_n\rvert \le 1$, and $0 \le b_n \le 1$ as soon as $n \ge t^2/2$. By Lemma 5 with $s = t/\sqrt{n}$, $a_n - b_n = r(t/\sqrt{n})$. So by Step 2 and the bound of Lemma 5, for $n \ge t^2/2$,
$$\big\lvert \varphi_{Z_n}(t) - b_n^n\big\rvert \le n\,\Big\lvert r\Big(\frac{t}{\sqrt{n}}\Big)\Big\rvert \le n \cdot \frac{t^2}{n}\,b\Big(\frac{t}{\sqrt{n}}\Big) = t^2\,b\Big(\frac{t}{\sqrt{n}}\Big) \to 0,$$
because $t/\sqrt{n} \to 0$ and $b(s) \to 0$ as $s \to 0$.
Step 4: the real limit. For $t = 0$, $b_n^n = 1$. For $t \ne 0$ and $n > t^2/2$, $b_n$ is positive and $\ln b_n^n = n\ln(1 - t^2/(2n))$. Since $\ln(1 - u)/u \to -1$ as $u \to 0$, this is $-\tfrac{t^2}{2} \cdot \dfrac{\ln(1 - u_n)}{-u_n}$ with $u_n = t^2/(2n) \to 0$, which tends to $-t^2/2$. So $b_n^n \to e^{-t^2/2}$.
Step 5: conclude. Steps 3 and 4 give $\varphi_{Z_n}(t) \to e^{-t^2/2} = \varphi_Z(t)$ for every real $t$, where $Z$ is standard normal. By the continuity theorem, $Z_n \xrightarrow{d} Z$. Since $\Phi$ is continuous, that means $P(Z_n \le z) \to \Phi(z)$ at every $z$, and by the theorem on uniform convergence to a continuous limit the convergence is uniform in $z$. $\square$
The argument is that of Durrett (2019, Theorem 3.4.1), who handles Steps 2 to 4 with a lemma on complex powers (his Theorem 3.4.2 and Lemma 3.4.3). The uniqueness theorem does not appear as a step, but it stands behind the last one: without it, a limit of characteristic functions would not identify a limiting distribution, and the continuity theorem could not be stated as it is.
Where each hypothesis enters. Independence is used in Step 1, to turn the characteristic function of a sum into a product. Identical distribution is used there too, to make the product a power. The finite variance is used in Lemma 5, twice: $E[Y^2] = 1$ fixes the term $-s^2/2$, the only part of $\varphi_Y$ that survives the limit, and the finiteness makes the tail term vanish. Every other feature of the population sits in the remainder $r$ and disappears.
Checking the steps on a population without a moment generating function
Back to the opening example. Let $X$ have Student's $t$ distribution with 3 degrees of freedom. By the density formula of The Chi-Squared and t Distributions with $\nu = 3$, where $\Gamma(2) = 1$ and $\Gamma(\tfrac32) = \sqrt{\pi}/2$, its density is $2/[\pi\sqrt{3}\,(1 + x^2/3)^2]$, and by that article its variance is $\nu/(\nu - 2) = 3$. The standardized variable $Y = X/\sqrt{3}$ therefore has the density
$$f_Y(y) = \frac{2}{\pi\,(1 + y^2)^2}.$$
For large $\lvert y\rvert$ this behaves like $2/(\pi y^4)$. So $E[Y^2]$ is finite, but $E[\lvert Y\rvert^3]$ is not, since its integrand behaves like $2/(\pi y)$. And $E[e^{tY}]$ is infinite for every $t \ne 0$, since an exponential outgrows any power. The proof by moment generating functions does not apply, and the Berry–Esseen bound of The Central Limit Theorem, which needs a third absolute moment, gives nothing.
The characteristic function can be found in closed form. The Cauchy density $1/[\pi(1 + y^2)]$ has characteristic function $e^{-\lvert s\rvert}$ (Grinstead and Snell, 1997, §10.3), which in scaled form says
$$\int_{-\infty}^{\infty} \frac{\cos sy}{a^2 + y^2}\,dy = \frac{\pi}{a}\,e^{-a\lvert s\rvert} \qquad (a > 0).$$
Differentiate both sides with respect to $a$. On the left the derivative of the integrand is $-2a\cos(sy)/(a^2 + y^2)^2$. For $a$ in a closed interval $a_0 \le a \le a_1$ of positive numbers it is bounded in absolute value by $2a_1/(a_0^2 + y^2)^2$, a function of $y$ with a finite integral, so the derivative may be taken under the integral sign. On the right the derivative is $-\pi e^{-a\lvert s\rvert}(1 + a\lvert s\rvert)/a^2$. Dividing by $-2a$ and setting $a = 1$,
$$\int_{-\infty}^{\infty} \frac{\cos sy}{(1 + y^2)^2}\,dy = \frac{\pi}{2}\,(1 + \lvert s\rvert)\,e^{-\lvert s\rvert}, \qquad\text{so}\qquad \varphi_Y(s) = (1 + \lvert s\rvert)\,e^{-\lvert s\rvert}.$$
(The sine part of $E[e^{isY}]$ vanishes because the density is symmetric.) The script checks this closed form against numerical integration at five values of $s$; at $s = 1$ it is $2/e = 0.7358$.
Expanding the exponential, $\varphi_Y(s) = 1 - s^2/2 + \lvert s\rvert^3/3 - \cdots$. The quadratic term is the one Lemma 5 promises. The next term is $\lvert s\rvert^3/3$, which has no third derivative at 0: that is how the infinite third moment shows itself. The table compares the remainder with its leading term and with the bound $b(s)$ of Lemma 5, which has a closed form for this density.
| $s$ | $r(s)/s^2$ | $\lvert s\rvert/3$ | Bound $b(s)$ |
|---|---|---|---|
| 1 | 0.2358 | 0.3333 | 0.4883 |
| 0.1 | 0.0321 | 0.0333 | 0.0975 |
| 0.01 | 0.0033 | 0.0033 | 0.0146 |
| 0.001 | 0.0003 | 0.0003 | 0.0020 |
All three columns go to 0, as the lemma requires. The bound goes more slowly than the remainder, since it is built to work for every distribution with a finite variance.
By Step 1 the characteristic function of the standardized mean of $n$ draws is $\varphi_{Z_n}(t) = (1 + \lvert t\rvert/\sqrt{n})^n\,e^{-\lvert t\rvert\sqrt{n}}$:
| $n$ | $\varphi_{Z_n}(1)$ | $\varphi_{Z_n}(2)$ |
|---|---|---|
| 1 | 0.7358 | 0.4060 |
| 10 | 0.6606 | 0.2408 |
| 100 | 0.6256 | 0.1707 |
| 1,000 | 0.6128 | 0.1467 |
| 10,000 | 0.6085 | 0.1389 |
| Limit $e^{-t^2/2}$ | 0.6065 | 0.1353 |
Because this characteristic function has a finite integral, the distribution function of $Z_n$ can be computed from it exactly, without simulation. The inversion formula for a characteristic function with a finite integral gives the density of $Z_n$ as $\frac{1}{2\pi}\int e^{-itz}\varphi_{Z_n}(t)\,dt$ (Durrett, 2019, Theorem 3.3.14). Integrating that density from 0 to $z$, and using the symmetry of the distribution, gives
$$P(Z_n \le z) = \frac12 + \frac{1}{\pi}\int_0^\infty \varphi_{Z_n}(t)\,\frac{\sin tz}{t}\,dt.$$
The script evaluates the integral by Simpson's rule. At $n = 1$ it reproduces the $t$ distribution function to 9 decimal places, which checks both the formula and the numerical method.
| $n$ | Largest gap $\sup_z \lvert P(Z_n \le z) - \Phi(z)\rvert$ | $P(\lvert Z_n\rvert > 1.96)$ |
|---|---|---|
| 1 | 0.0869 | 0.0426 |
| 2 | 0.0657 | 0.0441 |
| 10 | 0.0323 | 0.0462 |
| 30 | 0.0192 | 0.0472 |
| 100 | 0.0107 | 0.0482 |
| 400 | 0.0054 | 0.0490 |
| 1,000 | 0.0035 | 0.0493 |
| Normal | 0 | 0.0500 |
The gap is the largest over a grid of $z$ from $-6$ to 6 in steps of 0.01. It falls at every step of the table, as the theorem says it must in the limit. Multiplied by $\sqrt{n}$ it is 0.087 at $n = 1$ and 0.109 at $n = 1{,}000$, so in this range it shrinks roughly like $1/\sqrt{n}$. That rate is an observation about these numbers. The theorem gives no rate, and the usual source of one, the Berry–Esseen bound, is not available here. The two-sided tail probability approaches 0.05 from below. A reason consistent with that is the shape of the population: its heavy tails put some of the standardized mean's probability very far out, and since the variance is fixed at 1, less is left just beyond $\pm 1.96$. As an independent check on the inversion, the script also drew 10,000 samples of 10 from the $t$ distribution for each of 30 seeds. The share with $\lvert Z_{10}\rvert > 1.96$ had median 0.0465 (interquartile range 0.0452 to 0.0478), against the computed 0.0462.
Checking the steps on the blood pressures
The 5,247 adults with a blood pressure reading in the National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018 are the unit's running population (National Center for Health Statistics, 2020). Sampling from them with replacement gives i.i.d. draws from a population with variance 405.01 and skewness 1.038. This population is finite, so it has a moment generating function and the course page's proof covers it. The characteristic function proof covers it as well, and here every quantity in the proof can be computed. The characteristic function of the standardized population is the average $\varphi_Y(s) = \frac{1}{N}\sum_j e^{isy_j}$ over its $N = 5{,}247$ standardized values $y_j$. It is complex, because the population is skewed. The table gives the distance between $\varphi_{Z_n}(t) = [\varphi_Y(t/\sqrt{n})]^n$ and its limit, and the bound that Steps 3 and 4 put on it, $t^2\,b(t/\sqrt{n}) + \lvert b_n^n - e^{-t^2/2}\rvert$.
| $n$ | Distance at $t = 1$ | Bound at $t = 1$ | Distance at $t = 2$ | Bound at $t = 2$ |
|---|---|---|---|---|
| 10 | 0.0331 | 0.1063 | 0.0607 | 0.8157 |
| 30 | 0.0191 | 0.0594 | 0.0345 | 0.4639 |
| 400 | 0.0052 | 0.0158 | 0.0094 | 0.1252 |
| 10,000 | 0.0010 | 0.0031 | 0.0019 | 0.0249 |
Each distance is below its bound, and both columns of each pair tend to 0. The bound is loose, especially at $t = 2$. It does not need to be sharp: the proof needs only that it vanish.
Example (the Cauchy distribution: where the proof stops). The Cauchy distribution has characteristic function $e^{-\lvert s\rvert}$, and Step 1 is valid for it, since Step 1 uses only independence. The characteristic function of the mean of $n$ Cauchy draws is $[e^{-\lvert t\rvert/n}]^n = e^{-\lvert t\rvert}$, so by the uniqueness theorem the mean of $n$ draws has the same Cauchy distribution as one draw. What fails is Lemma 5. Near 0, $e^{-\lvert s\rvert} = 1 - \lvert s\rvert + \cdots$ has a corner where a finite variance would give $1 - s^2/2$, and no standardization exists. The Central Limit Theorem shows the failure in simulation.
What the theorem gives when combined with Slutsky's theorem
Three consequences are used throughout the collection.
Corollary 1 (the studentized mean, for any population with a finite variance). Let $X_1, X_2, \ldots$ be i.i.d. with mean $\mu$ and finite variance $\sigma^2 > 0$, and let $s_n$ be the sample standard deviation of the first $n$. Then
$$T_n = \frac{\bar{X}_n - \mu}{s_n/\sqrt{n}} \xrightarrow{d} Z, \qquad Z \text{ standard normal},$$
where $T_n$ is given any fixed value on the event $s_n = 0$.
Proof. As on the course page, $s_n^2 = \frac{n}{n-1}\big(V_n - (\bar{X}_n - \mu)^2\big)$ with $V_n = \frac1n\sum_{i \le n}(X_i - \mu)^2$. The variables $(X_i - \mu)^2$ are i.i.d. with finite mean $\sigma^2$, so $V_n \xrightarrow{p} \sigma^2$ by the weak law of large numbers for a finite mean, which needs a finite mean and nothing more (The Laws of Large Numbers). By the same law $\bar{X}_n - \mu \xrightarrow{p} 0$, and then $(\bar{X}_n - \mu)^2 \xrightarrow{p} 0$ by part 3 of the continuous mapping theorem. So $V_n - (\bar{X}_n - \mu)^2 \xrightarrow{p} \sigma^2$. The factor $n/(n-1)$ is a sequence of constants tending to 1, and a sequence converging in probability to $\sigma^2$, multiplied by it, still does: the difference is that sequence times the constants $1/(n-1)$, which tend to 0. The sequence converges in probability, hence in distribution, so it is bounded in probability by Lemma 2, and Lemma 4 applies. So $s_n^2 \xrightarrow{p} \sigma^2$, and $s_n/\sigma \xrightarrow{p} 1$ by part 3 of the continuous mapping theorem with the square root. Now $T_n = Z_n/(s_n/\sigma)$, the central limit theorem gives $Z_n \xrightarrow{d} Z$, and the quotient form of Slutsky's theorem gives $T_n \xrightarrow{d} Z$. $\square$
The course page proves this corollary for populations with a moment generating function finite near 0, and cites the general case. Its proof of the central limit theorem needs the moment generating function, and its law of large numbers for $V_n$, the one that follows from Chebyshev's inequality, needs a finite fourth moment. With the weak law for a finite mean and the theorem above, the general case is now proved within the unit, apart from the two cited theorems on characteristic functions. For draws from the $t$ distribution with 3 degrees of freedom, which has neither a moment generating function nor a finite fourth moment, the corollary applies and the course page's argument does not.
Corollary 2 (a confidence interval's limiting coverage). Under the hypotheses of Corollary 1, for $0 < \alpha < 1$ and $z^* = \Phi^{-1}(1 - \alpha/2)$,
$$P\Big(\bar{X}_n - z^*\frac{s_n}{\sqrt{n}} \le \mu \le \bar{X}_n + z^*\frac{s_n}{\sqrt{n}}\Big) \to 1 - \alpha.$$
Proof. When $s_n > 0$ the event is $-z^* \le T_n \le z^*$, and $P(s_n = 0) \to 0$ because $s_n \xrightarrow{p} \sigma > 0$. By Corollary 1 and Lemma 1, $P(T_n \le z^*) \to \Phi(z^*)$ and $P(T_n < -z^*) \to \Phi(-z^*)$, every point being a continuity point of $\Phi$. The difference tends to $\Phi(z^*) - \Phi(-z^*) = 1 - \alpha$. $\square$
Corollary 3 (linear combinations of a vector mean). Let $\mathbf{X}_1, \mathbf{X}_2, \ldots$ be i.i.d. random vectors with $k$ components, mean vector $\boldsymbol{\mu}$ and a covariance matrix $\Sigma$ with finite entries, and let $\bar{\mathbf{X}}_n$ be the mean of the first $n$. Then for every vector of constants $\mathbf{a}$,
$$\sqrt{n}\;\mathbf{a}\cdot(\bar{\mathbf{X}}_n - \boldsymbol{\mu}) \xrightarrow{d} \text{normal with mean 0 and variance } \mathbf{a}^{\mathsf{T}}\Sigma\,\mathbf{a}.$$
Proof. The scalars $W_i = \mathbf{a}\cdot\mathbf{X}_i$ are i.i.d. with mean $\mathbf{a}\cdot\boldsymbol{\mu}$ and variance $v = \mathbf{a}^{\mathsf{T}}\Sigma\,\mathbf{a}$, by the formula for the variance of a linear combination (Expectation, Variance and Covariance). If $v > 0$, the theorem gives $\sqrt{n}\,(\bar{W}_n - \mathbf{a}\cdot\boldsymbol{\mu})/\sqrt{v} \xrightarrow{d} Z$, and multiplying by the constant $\sqrt{v}$ (the continuous mapping theorem) gives the claim. If $v = 0$, each $W_i$ equals its mean with probability 1, the left side is 0 with probability 1, and the limit is the constant 0, which is the normal distribution with variance 0. $\square$
Corollary 3 is all that The Delta Method needs for functions of several means. The full statement for vectors is stronger. The Cramér–Wold device says that a sequence of random vectors $\mathbf{X}_n$ converges in distribution to $\mathbf{X}$ if every fixed linear combination $\mathbf{a}\cdot\mathbf{X}_n$ converges in distribution to $\mathbf{a}\cdot\mathbf{X}$ (Durrett, 2019, Theorem 3.10.6). The name refers to Cramér and Wold (1936). The first theorem listed in the review of that paper in the Jahrbuch über die Fortschritte der Mathematik (JFM 62.0596.04) is the uniqueness statement behind the device: two distributions on $\mathbb{R}^n$ that give the same probability to every half-space $\mathbf{t}\cdot\mathbf{x} \le z$ are equal. With Corollary 3 it gives the multivariate central limit theorem: $\sqrt{n}\,(\bar{\mathbf{X}}_n - \boldsymbol{\mu})$ converges in distribution to the multivariate normal distribution with mean vector $\mathbf{0}$ and covariance matrix $\Sigma$ (Durrett, 2019, Theorem 3.10.7). The device is cited, not proved here; its proof is the continuity theorem for characteristic functions of vectors.
The theorem also has versions for observations that are independent but not identically distributed. The Lindeberg–Feller theorem gives the condition under which sums of such variables are asymptotically normal (Durrett, 2019, Theorem 3.4.10), and it is cited, not proved here. Its proof follows the steps above with a product of different characteristic functions in place of a power.
What is proved, what is cited, what is computed
Proved here. The four implications among the modes of convergence and the three examples that rule out the others. Lemmas 1 to 4, the theorem on uniform convergence to a continuous limit, the criterion by bounded continuous functions, the continuous mapping theorem for a function continuous everywhere, and Slutsky's theorem. The Lindeberg–Lévy theorem for every population with a finite variance, given the continuity theorem and the expansion of a characteristic function (Lemma 5), which is proved in Moment Generating and Characteristic Functions. The three corollaries, the first given the weak law for a finite mean from The Laws of Large Numbers.
Cited. The uniqueness theorem and the continuity theorem for characteristic functions; the continuous mapping theorem for functions with discontinuities; the almost surely convergent subsequence; the Cramér–Wold device and the Lindeberg–Feller theorem.
Where measure theory is avoided. Random variables here are functions on a sample space. The expected value is the one defined for every random variable in Expectation, Variance and Covariance, which is a sum for a discrete variable and an integral for a variable with a density; the proofs above use only its linearity, its monotonicity and the inequality $\lvert E[W]\rvert \le E[\lvert W\rvert]$. Almost sure convergence concerns the probability of an event defined by a whole infinite sequence. That such events have probabilities is taken for granted, without the construction of a probability space that carries infinitely many independent variables. Random Variables and Distribution Functions says what the unit leaves out. The two cited theorems on characteristic functions are where the theory of the integral is needed in earnest. The tail fact behind Lemma 5 is proved for every random variable in the article on characteristic functions, from the unit's definition of the expected value.
Computed. The tables for the $t$ distribution with 3 degrees of freedom are exact calculations from its characteristic function, and those for the blood pressures are exact calculations over the 5,247 values. One seeded simulation checks the inversion. The observation that the gap for the $t$ distribution shrinks like $1/\sqrt{n}$ is a description of the table, not a theorem.
Worked problems
These problems were written for this article, on small exact cases and on the two populations above, to exercise each main result once. The script tools/experiments/statistics/proof-of-the-central-limit-theorem.py recomputes every number in them.
Problem 1 (almost surely, in probability, but not in every mean). Let $U$ be uniform on the interval from 0 to 1, and for a fixed number $c > 0$ let $X_n = n^c$ if $U < 1/n$ and $X_n = 0$ otherwise. Decide in which of the four senses $X_n$ converges to 0.
Solution. It converges almost surely, in probability and in distribution for every $c > 0$, and in $r$th mean exactly when $r < 1/c$.
For an outcome with $U > 0$, $X_n = 0$ for every $n \ge 1/U$, so the sequence is eventually 0 and converges to 0. The event $U > 0$ has probability 1, so $X_n \xrightarrow{\text{a.s.}} 0$. By parts 1 and 3 of the theorem on the modes, $X_n \xrightarrow{p} 0$ and $X_n \xrightarrow{d} 0$. Both can also be seen directly. For $0 < \varepsilon < 1$, $P(\lvert X_n\rvert > \varepsilon) = P(U < 1/n) = 1/n \to 0$. And $F_n(x) = 0$ for $x < 0$, $F_n(x) = 1 - 1/n$ for $0 \le x < n^c$ and $F_n(x) = 1$ from $n^c$ on, which tends to the distribution function of the constant 0 at every $x \ne 0$. At $x = 0$ it tends to 1 as well, though the definition does not ask for that.
For the means, $E[\lvert X_n\rvert^r] = n^{cr} \cdot \tfrac{1}{n} = n^{cr - 1}$. This tends to 0 exactly when $cr < 1$. With $c = 1$, the example of the text, there is convergence in $r$th mean for $r < 1$ only; $E[X_n] = 1$ for every $n$ although $X_n \to 0$ almost surely. With $c = \tfrac12$ there is convergence in mean ($r = 1$) but not in mean square: $E[X_n^2] = 1$ for every $n$. The example shows that the limit of the expected values need not be the expected value of the limit, and that the answer can depend on $r$.
Problem 2 (a limit in distribution that is not normal). Let $U_1, U_2, \ldots$ be independent and uniform on the interval from 0 to 1, and let $M_n$ be the largest of the first $n$. Show that $M_n \xrightarrow{p} 1$, and find the limit in distribution of $n(1 - M_n)$.
Solution. $n(1 - M_n)$ converges in distribution to the exponential distribution with mean 1.
The largest of $n$ values is at most $x$ exactly when all $n$ are, and the draws are independent, so $P(M_n \le x) = x^n$ for $0 \le x \le 1$. For $0 < \varepsilon < 1$, $P(\lvert M_n - 1\rvert > \varepsilon) = P(M_n < 1 - \varepsilon) = (1 - \varepsilon)^n \to 0$. So $M_n \xrightarrow{p} 1$.
That statement does not say how fast, and the question about $n(1 - M_n)$ supplies the rate. For $x \ge 0$ and $n \ge x$,
$$P\big(n(1 - M_n) > x\big) = P\Big(M_n < 1 - \frac{x}{n}\Big) = \Big(1 - \frac{x}{n}\Big)^n \to e^{-x},$$
by the limit of Step 4. For $x < 0$ the probability is 1. So the distribution function of $n(1 - M_n)$ tends to $1 - e^{-x}$ for $x \ge 0$ and to 0 for $x < 0$, at every $x$. That is the distribution function of the exponential distribution with mean 1, which is continuous, so by the theorem on uniform convergence to a continuous limit the convergence is uniform. At $x = 1$ the probabilities $(1 - 1/n)^n$ are 0.3487 for $n = 10$, 0.3660 for $n = 100$ and 0.3677 for $n = 1{,}000$, against $e^{-1} = 0.3679$. The limiting probability that the largest of $n$ draws is within $1/n$ of 1 is therefore $1 - e^{-1} = 0.6321$.
Two points are worth drawing out. The scaling is $n$, not the $\sqrt{n}$ of the central limit theorem, and the limit is not normal: convergence in distribution is a general notion, of which the central limit theorem is one instance. And by Slutsky's theorem, since $M_n \xrightarrow{p} 1$, the sequence $n(1 - M_n)/M_n$ has the same exponential limit.
Problem 3 (Student's t tends to the normal). Show that the $t$ distribution with $\nu$ degrees of freedom converges in distribution to the standard normal as $\nu \to \infty$, and compute how fast the two-sided tail beyond $\pm 1.96$ approaches 0.05.
Solution. By its definition in The Chi-Squared and t Distributions, a variable with the $t$ distribution with $\nu$ degrees of freedom can be written $T_\nu = Z/\sqrt{V_\nu/\nu}$, with $Z$ standard normal and $V_\nu$ an independent chi-squared variable with $\nu$ degrees of freedom. Convergence in distribution concerns only the distributions, so any such representation will do. Take $V_\nu = Z_1^2 + \cdots + Z_\nu^2$ for i.i.d. standard normal variables $Z_1, Z_2, \ldots$ independent of $Z$. Then $V_\nu/\nu$ is the mean of $\nu$ i.i.d. variables with mean $E[Z_1^2] = 1$ and variance 2 (both computed in that article), so $V_\nu/\nu \xrightarrow{p} 1$ by the weak law of large numbers. By part 3 of the continuous mapping theorem, $\sqrt{V_\nu/\nu} \xrightarrow{p} 1$. The numerator is the same variable $Z$ for every $\nu$, so it converges in distribution to $Z$ trivially. By the quotient form of Slutsky's theorem, $T_\nu \xrightarrow{d} Z$.
Since $\Phi$ is continuous, Lemma 1 gives $P(\lvert T_\nu\rvert > 1.96) \to 2(1 - \Phi(1.96)) = 0.0500$. From the $t$ distribution function, the tail is 0.0816 for $\nu = 9$, 0.0597 for $\nu = 29$, 0.0528 for $\nu = 99$ and 0.0507 for $\nu = 399$. Those are the error rates of an interval that uses 1.96 where the $t$ critical value belongs, for normal samples of 10, 30, 100 and 400. The argument is the same as Corollary 1, with the difference that here the distribution at each $\nu$ is known exactly, so the limit can be watched.
Problem 4 (a sample size three ways). Draws are taken independently from the $t$ distribution with 3 degrees of freedom. How many are needed for their mean to be within 0.25 of the true mean 0 with probability at least 0.95? Answer with Chebyshev's inequality, with the normal approximation, and exactly.
Solution. Chebyshev's inequality gives 960, the normal approximation gives 185, and the exact answer is 182.
Chebyshev. The mean $\bar{X}_n$ has variance $3/n$, so $P(\lvert\bar{X}_n\rvert \ge 0.25) \le 3/(0.0625\,n)$, by the inequality proved in Probability Inequalities. The bound is at most 0.05 when $n \ge 3/(0.05 \times 0.0625) = 960$. This is a guarantee.
Normal approximation. By the theorem, $Z_n = \bar{X}_n\sqrt{n}/\sqrt{3}$ is approximately standard normal, so the probability of a miss is about $2(1 - \Phi(0.25\sqrt{n}/\sqrt{3}))$. That is 0.05 when $0.25\sqrt{n}/\sqrt{3} = 1.96$, or $n = (1.96\sqrt{3}/0.25)^2 = 184.40$, so $n = 185$. At $n = 185$ the mean is allowed $0.25\sqrt{185}/\sqrt{3} = 1.9632$ standard errors, and the approximate probability of being within 0.25 is 0.9504. This is an approximation, and the theorem alone does not say how good it is at $n = 185$.
Exactly. For this population the inversion formula above gives $P(\lvert Z_n\rvert \le k)$ for any $n$ and $k$. At $n = 185$ the exact probability is 0.9518, a little above the normal value, in line with the table: the two-sided tail of $Z_n$ beyond about 1.96 is lighter than the normal's. Searching downward, the smallest $n$ with probability at least 0.95 is 182 (0.9501; at $n = 181$ it is 0.9495).
So the normal approximation is off by 3 observations in 185, on a population for which no bound on the error of the approximation is available. Chebyshev's 960 is more than five times too many, the price of a guarantee that uses only the variance. At $n = 960$ the allowed error is 4.472 standard errors, and the exact probability of a miss is 0.00024, against the guaranteed 0.05.
Summary
| Result | Statement |
|---|---|
| Modes of convergence | almost surely $\Rightarrow$ in probability $\Leftarrow$ in $r$th mean; in probability $\Rightarrow$ in distribution |
| Constant limit | $X_n \xrightarrow{d} c$ implies $X_n \xrightarrow{p} c$ |
| Bounded continuous functions | $X_n \xrightarrow{d} X$ exactly when $E[h(X_n)] \to E[h(X)]$ for all bounded continuous $h$ |
| Continuous mapping | $X_n \xrightarrow{d} X$ and $g$ continuous imply $g(X_n) \xrightarrow{d} g(X)$ |
| Slutsky | $X_n \xrightarrow{d} X$, $A_n \xrightarrow{p} a$, $B_n \xrightarrow{p} b$ imply $A_nX_n + B_n \xrightarrow{d} aX + b$ |
| Expansion | $\varphi_Y(s) = 1 - s^2/2 + r(s)$ with $\lvert r(s)\rvert \le s^2\,E[\min(\lvert s\rvert\lvert Y\rvert^3/6,\ Y^2)]$ |
| Lindeberg–Lévy | i.i.d., finite variance: $\sqrt{n}\,(\bar{X}_n - \mu)/\sigma \xrightarrow{d}$ standard normal |
| Studentized mean | the same limit with $s_n$ in place of $\sigma$ |
What we have learned
A sequence of random variables can converge almost surely, in probability, in $r$th mean or in distribution. The first and third each imply the second, the second implies the fourth, and examples show that nothing else holds in general. Convergence in distribution is the weakest and the one statistics uses most. It is equivalent to convergence of the expected values of bounded continuous functions, it survives continuous maps, and by Slutsky's theorem it is not disturbed by sequences that converge in probability to constants. The central limit theorem for i.i.d. variables needs a finite variance and nothing more. The proof turns the sum into a power of a characteristic function, expands that function to second order with a remainder controlled by the variance alone, and passes to the limit with Lévy's continuity theorem, which is cited. For draws from the $t$ distribution with 3 degrees of freedom, outside the reach of the proof by moment generating functions, the exact largest gap to the normal falls from 0.0869 at $n = 1$ to 0.0035 at $n = 1{,}000$.
The theorem gives the limiting distribution of a mean. Most statistics are functions of means: a logarithm of a proportion, a ratio of two risks, a standard deviation. The Delta Method carries the normal limit through such functions, using Slutsky's theorem as proved here.
References
- Billingsley, P. (1995). Probability and Measure (3rd ed.). Wiley. The reference given on the course page for the Lindeberg–Lévy theorem in general. record · 8 pages
- Cramér, H., and Wold, H. (1936). Some theorems on distribution functions. Journal of the London Mathematical Society, s1-11(4), 290–294. record
- Durrett, R. (2019). Probability: Theory and Examples (5th ed.). Cambridge University Press. Theorem 1.1.1 (continuity of a measure along monotone sequences of sets); Theorems 2.3.2 and 2.3.4 (subsequences; continuous functions of sequences that converge in probability); Theorems 3.2.9 and 3.2.10 and Exercises 3.2.9, 3.2.13 and 3.2.14 (weak convergence, the continuous mapping theorem, the converging together lemma); Theorems 3.3.11, 3.3.14 and 3.3.17, Lemma 3.3.19 and Theorem 3.3.20 (inversion, continuity theorem, Taylor estimate); Theorems 3.4.1, 3.4.2 and 3.4.10 and Lemma 3.4.3 (central limit theorems); Theorems 3.10.6 and 3.10.7 (the Cramér–Wold device and the central limit theorem for vectors). Theorem numbers are those of the author's version of January 11, 2019. record · 4 pages
- Grinstead, C. M., and Snell, J. L. (1997). Introduction to Probability (2nd rev. ed.). American Mathematical Society. §10.3 (the characteristic function of the Cauchy density). record · 20 pages
- National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018: Demographic Variables and Sample Weights (DEMO_J) and Blood Pressure (BPX_J) data files. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 67 pages
- van der Vaart, A. W. (1998). Asymptotic Statistics. Cambridge University Press. Chapter 2, "Stochastic Convergence", pp. 5–24 (cited on the course page for Slutsky's lemma). record · 8 pages