The Central Limit Theorem

Why sample means are approximately normal whatever the shape of the population, and how far to trust it. The Lindeberg–Lévy theorem stated and read carefully; exact distributions of exponential means; how fast the approximation works, with the skewness of a mean proved to fall as one over root n, and NHANES blood pressure against the far more skewed C-reactive protein; the Berry–Esseen bound; failures under infinite variance (Cauchy, Pareto) and dependence; and a proof sketch by moment generating functions.

C-reactive protein (CRP) is made by the liver and is one of the most sensitive, though nonspecific, indicators of inflammation. A high-sensitivity assay for it is used mainly to assess the risk of cardiovascular disease (National Center for Health Statistics, 2020b). Among the 5,146 adults of the National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018 with a measurement, its distribution is about as far from a bell curve as a medical measurement gets. The median is 1.94 mg/L, the mean 4.17 and the largest value 182.8. The skewness is 7.9, against 1.0 for systolic blood pressure, and the top 1% of adults account for 15.6% of the total. Yet the means of random samples of 400 of these adults have a nearly normal distribution, while the means of samples of 30 are clearly skewed. Why should averaging make such a distribution normal at all, and why does it take so many observations?

The central limit theorem answers the first question and, read carefully, the second too. It says that the mean of many independent observations with finite variance has a distribution close to normal, whatever the shape of the population. The Sampling Distribution of the Mean established the center and spread of a sample mean, $\mu$ and $\sigma/\sqrt{n}$, for any population. This article concerns its shape.

First we state the theorem and read it closely, because its popular paraphrase is wrong in several ways. Then we watch it work exactly on exponential means, measure how fast it works on blood pressure and CRP, and bound its error. Next we see where it fails: infinite variance and dependence. Finally we prove it with moment generating functions and trace its history.

The theorem

The version used throughout an introductory course is the one for independent, identically distributed observations, proved in this generality by Lindeberg (1922) and Lévy (1925). In plain words: standardize the mean so that it has mean 0 and standard deviation 1, and its probabilities approach those of the standard normal curve.

Theorem (Lindeberg–Lévy). Let $X_1, X_2, \ldots$ be independent random variables with a common distribution that has mean $\mu$ and finite variance $\sigma^2 > 0$, and let $\bar{X}_n = \frac{1}{n}\sum_{i=1}^n X_i$. Then for every real number $z$,

$$\lim_{n \to \infty} P\left(\frac{\bar{X}_n - \mu}{\sigma/\sqrt{n}} \le z\right) = \Phi(z),$$

where $\Phi$ is the standard normal distribution function.

Multiplying numerator and denominator by $n$ gives the same statement for the sum $S_n = \sum X_i$, standardized as $(S_n - n\mu)/(\sigma\sqrt{n})$. That is the form in which Grinstead and Snell (1997, Theorems 9.4 and 9.6) state it. In practice it is used as an approximation for a fixed, large $n$. In the notation of the course's formula sheet, where the second argument is the standard deviation,

$$\bar{X} \approx N\!\left(\mu,\ \frac{\sigma}{\sqrt{n}}\right), \qquad S_n \approx N\!\left(n\mu,\ \sigma\sqrt{n}\right).$$

The conclusion is a statement about distribution functions, called convergence in distribution: the probability that the standardized mean is at most $z$ approaches the normal probability, for every $z$. The hypotheses are three:

  • the observations are independent;
  • they have one common distribution;
  • that distribution has a finite variance.

Nothing is assumed about its shape. It may be skewed, discrete, bimodal or bounded.

What it says and what it does not

The theorem is often paraphrased as "large samples are normal". That is wrong in several ways at once.

Check your understanding. A study collects CRP from 400 adults. Will a histogram of the 400 values be close to a normal curve?

No. A sample of 400 CRP values has the same skewed shape as the population, only more of it. What becomes normal is the distribution of the mean over repeated samples. The theorem says nothing about the data themselves.

The other points are subtler.

  • It is about the standardized mean. The sample mean itself does not approach a normal distribution. It piles up at $\mu$, since its standard deviation $\sigma/\sqrt{n}$ goes to zero (that is the law of large numbers). The theorem describes the shape of the fluctuations around $\mu$ on the scale of $\sigma/\sqrt{n}$.
  • It gives no sample size. A limit says what happens eventually, not when. How large $n$ must be depends on the population, and the next sections measure it.
  • It is about distribution functions. The count of successes in $n$ trials, a sum of zeros and ones, never has a density. Its distribution function approaches the normal one, which is why approximating it needs the continuity correction of The Normal Approximation to the Binomial.
  • It uses the true $\sigma$. Replacing $\sigma$ by the sample standard deviation $s$ changes the distribution for small samples: that is Student's $t$, in Confidence Intervals for a Mean: z and t. For large samples $s$ is close to $\sigma$ and the limit is the same, but that is a separate argument.
  • It needs finite variance and independence. Both conditions can fail in practice, and the theorem then fails too, as shown below.

The theorem covers more than means of measurements.

  • A binomial count is a sum of independent Bernoulli variables, so it is approximately normal for large $n$: the de Moivre–Laplace theorem.
  • A Poisson count with a large mean $\lambda$ is a sum of many independent Poisson counts with small means, so it too is approximately normal, as The Poisson Distribution uses.
  • A sample proportion is a sample mean of zeros and ones.

Watching it work: exponential means

For one population the distribution of a sample mean can be computed exactly, so we can watch the theorem work without simulation noise. The exponential distribution with mean 1 is the waiting time for an event in a Poisson process with rate 1. It has standard deviation 1 and skewness 2, a strong right skew.

The sum of $n$ independent exponential waiting times is the time to the $n$th event of the Poisson process, which has a gamma density (Grinstead and Snell, 1997, §9.3, Example 9.8). So the mean of $n$ exponential values has a gamma distribution with shape $n$ and scale $1/n$. The probability that the standardized mean is at most $z$ is then a regularized incomplete gamma function, computable exactly for any $n$.

Exact density of the standardized mean of n exponential observations, against the standard normal curve (dashed). For n = 1 it is the exponential itself, shifted and cut off sharply at -1. Follow the curves as n grows: by n = 5 the curve is rounded but still leans to the left with a long right tail, and by n = 30 it is close to the normal curve, with its peak still slightly left of zero.
Exact density of the standardized mean of n exponential observations, against the standard normal curve (dashed). For n = 1 it is the exponential itself, shifted and cut off sharply at -1. Follow the curves as n grows: by n = 5 the curve is rounded but still leans to the left with a long right tail, and by n = 30 it is close to the normal curve, with its peak still slightly left of zero.

The table compares the exact distribution with the normal in two ways that matter for inference. One is the largest gap between the two distribution functions over all $z$. The other is the probability in each tail beyond $\pm 1.96$, which the normal puts at 0.025.

Check your understanding. The exponential is skewed to the right. At $n = 30$, will the upper tail beyond $+1.96$ hold more or less than 2.5%, and the lower tail?

The upper tail holds more and the lower tail less: 3.4% and 1.4%, as the table shows. A right-skewed population leaves its means with a long right tail and a short left one.

$n$ Largest gap $\sup_z \lvert F_n(z) - \Phi(z)\rvert$ $P(Z > 1.96)$ $P(Z < -1.96)$ Both tails
1 0.1587 0.0518 0.0000 0.0518
5 0.0596 0.0433 0.0004 0.0438
10 0.0421 0.0393 0.0058 0.0451
30 0.0243 0.0340 0.0140 0.0481
100 0.0133 0.0302 0.0192 0.0494
1,000 0.0042 0.0267 0.0232 0.0499
10,000 0.0013 0.0255 0.0244 0.0500

Three things stand out.

  • The gap shrinks like $1/\sqrt{n}$. $\sqrt{n}$ times the gap is 0.1587 at $n = 1$ and 0.1330 from $n = 30$ on. That constant has a simple source. Take a continuous population with skewness $\gamma$ and a finite third moment. The Edgeworth expansion (Feller, 1971, ch. XVI) gives the distribution function of the standardized mean as $F_n(z) = \Phi(z) + \gamma(1 - z^2)\varphi(z)/(6\sqrt{n})$ plus terms smaller than $1/\sqrt{n}$. Here $\varphi$ is the standard normal density. The correction term is largest in size at $z = 0$, where it is $\gamma\varphi(0)/(6\sqrt{n})$. With $\gamma = 2$ that is $0.1330/\sqrt{n}$, and the script finds the largest gap ever closer to $z = 0$ as $n$ grows.
  • The tails converge more slowly than their sum. At $n = 30$, where the popular rule of thumb would call the approximation good, the upper tail holds 3.4% instead of 2.5% and the lower tail 1.4%.
  • The errors in the two tails nearly cancel. So a two-sided probability (4.81% at $n = 30$) is accurate long before either one-sided probability is.

So a two-sided 95% interval for the mean of a moderately skewed population can have nearly the right coverage while its misses fall mostly on one side. One-sided bounds for such data deserve more caution (One-Sided Confidence Bounds and Sample Size). For strongly skewed data even the two-sided coverage suffers: Confidence Intervals for a Mean: z and t finds this for lognormal data, with nearly all the misses on one side.

How fast: skewness falls as one over root n

The slow convergence of the tails comes from skewness, and the skewness of a sample mean can be computed exactly. Write $\gamma = E[(X - \mu)^3]/\sigma^3$ for the skewness of the population. Here is the idea: in the cube of a sum of independent centered terms, only the cubes of single terms have nonzero expectation. There are $n$ of them, against a variance of $n\sigma^2$, so the skewness falls like $n/n^{3/2}$.

Proposition. If $X_1, \ldots, X_n$ are independent with common mean $\mu$, variance $\sigma^2$ and skewness $\gamma$, then the skewness of $\bar{X}$ is $\gamma/\sqrt{n}$.

Proof. Let $Y_i = X_i - \mu$, so that $E[Y_i] = 0$, $E[Y_i^2] = \sigma^2$ and $E[Y_i^3] = \gamma\sigma^3$. The skewness of $\bar{X}$ is that of $\sum Y_i$, since shifting by a constant and multiplying by the positive constant $1/n$ leave skewness unchanged. Expanding the cube,

$$E\Big[\Big(\sum_{i=1}^n Y_i\Big)^3\Big] = \sum_{i=1}^n \sum_{j=1}^n \sum_{k=1}^n E[Y_i Y_j Y_k].$$

In any term whose three indices are not all equal, some index occurs exactly once (if $i = j \ne k$, it is $k$; if all three differ, it is each of them). That variable is independent of the product of the others, so the expectation factors and contains $E[Y_k] = 0$ as a factor. Only the $n$ terms with $i = j = k$ survive, and the sum is $n\gamma\sigma^3$. The variance of the sum is $n\sigma^2$, so its skewness is

$$\frac{n\gamma\sigma^3}{(n\sigma^2)^{3/2}} = \frac{\gamma}{\sqrt{n}}. \qquad \square$$

The skewness falls, but slowly, and its starting value matters as much as $n$. To bring the skewness of the mean down to 0.2 takes $n = (\gamma/0.2)^2$ observations. Systolic blood pressure, with $\gamma = 1.04$, needs 27. The exponential, with $\gamma = 2$, needs 100.

Check your understanding. CRP has skewness 7.89. How many observations bring the skewness of its mean down to 0.2?

About 1,557: $(\gamma/0.2)^2$ with $\gamma = 7.892$. That is nearly sixty times the 27 that blood pressure needs.

The script checked the proposition on the two NHANES populations. It drew 10,000 samples with replacement for each of 30 seeds, and recorded the skewness of the 10,000 means and the shares of standardized means beyond $\pm 1.96$ (with the true $\sigma$). Medians over the seeds:

Population $n$ Skewness of means $\gamma/\sqrt{n}$ Above $+1.96$ Below $-1.96$
Blood pressure 5 0.460 0.464 0.0367 0.0113
Blood pressure 30 0.188 0.189 0.0300 0.0193
Blood pressure 100 0.106 0.104 0.0278 0.0215
Blood pressure 400 0.049 0.052 0.0265 0.0231
CRP 5 3.521 3.530 0.0460 0.0000
CRP 30 1.440 1.441 0.0481 0.0000
CRP 100 0.772 0.789 0.0430 0.0033
CRP 400 0.389 0.395 0.0340 0.0133
CRP 1,000 0.249 0.250 0.0316 0.0180

The measured skewness matches $\gamma/\sqrt{n}$ throughout. For blood pressure, means of 30 are close to normal, with tails of 3.0% and 1.9%. For CRP, means of 30 are more skewed than the blood pressures of individual adults. In the median seed none of the 10,000 means fell below the lower cutoff, and all of the 4.8% in the two tails was in the upper one. Means of 1,000 still put 3.2% above and 1.8% below.

Top: high-sensitivity C-reactive protein in 5,146 NHANES adults, piled up near zero with a long right tail; the mean (orange line) is 4.17 mg/L, more than twice the median. Bottom: standardized means of 10,000 samples of 30 (orange) and of 400 (green), against the standard normal curve (dashed). Notice that means of 30 are still strongly skewed, with nothing below -2 and a long right tail, while means of 400 are much closer to normal but still lean.
Top: high-sensitivity C-reactive protein in 5,146 NHANES adults, piled up near zero with a long right tail; the mean (orange line) is 4.17 mg/L, more than twice the median. Bottom: standardized means of 10,000 samples of 30 (orange) and of 400 (green), against the standard normal curve (dashed). Notice that means of 30 are still strongly skewed, with nothing below -2 and a long right tail, while means of 400 are much closer to normal but still lean.

What makes CRP slow is that a few adults with very high values supply much of its variance: 8.1% of the adults are above 10 mg/L, and the top 1% hold 15.6% of the total. A sample of 30 usually contains none of the top 1%, so its mean is a little below $\mu$. Occasionally it contains one, which pulls the mean far above. The normal approximation becomes good only when a typical sample contains many of the rare large values. A practical sign of trouble, before any theory, is a sample in which one or two observations make up a large share of the total.

Some of those high values may not describe a person's usual level. CRP is an acute-phase protein whose level may rise within six hours of an inflammatory stimulus (National Center for Health Statistics, 2020b), so a high value may reflect an inflammation at the time of the blood draw. And not every CRP result is a measured value. Results below the assay's detection limit, 0.15 mg/L, are stored by NHANES as the limit divided by $\sqrt{2}$, 0.11 mg/L (National Center for Health Statistics, 2020b). Twelve of the 5,146 adults have such a value. They affect only the far left of the population distribution and none of the conclusions here.

The Berry–Esseen bound

The skewness explains the shape of the error. How large can the error be, for a population we have not simulated? Andrew Berry (1941) and Carl-Gustav Esseen (1942) showed independently that it shrinks like $1/\sqrt{n}$, with a constant set by how far the population is from normal. Precisely, for independent identically distributed observations with a finite third absolute central moment $\rho = E\big[\lvert X - \mu\rvert^3\big]$,

$$\sup_z \left|P\left(\frac{\bar{X}_n - \mu}{\sigma/\sqrt{n}} \le z\right) - \Phi(z)\right| \le \frac{C\rho}{\sigma^3\sqrt{n}}$$

for a universal constant $C$, the same for every distribution. The ratio $\rho/\sigma^3$ is at least 1 and grows with skewness and heavy tails. So the worst error in any probability computed from the normal approximation shrinks like $1/\sqrt{n}$, faster for populations close to normal. The best value of $C$ has been narrowed over decades. Shevtsova (2011) proved $C < 0.4748$, and Esseen (1956) showed that no constant below $(\sqrt{10} + 3)/(6\sqrt{2\pi}) = 0.4097$ can work for every distribution.

For the exponential population, $\sigma = 1$ and

$$\rho = \int_0^\infty \lvert x - 1\rvert^3 e^{-x}\,dx = \int_0^1 (1 - x)^3 e^{-x}\,dx + \int_1^\infty (x - 1)^3 e^{-x}\,dx = \Big(\frac{6}{e} - 2\Big) + \frac{6}{e} = \frac{12}{e} - 2 = 2.4146.$$

The second integral is $e^{-1}\int_0^\infty u^3 e^{-u}\,du = 6/e$ after substituting $u = x - 1$, and the first follows by integrating by parts three times. The bound $0.4748 \times 2.4146/\sqrt{n}$ is 0.2093 at $n = 30$ and 0.1146 at $n = 100$, against exact gaps of 0.0243 and 0.0133.

The largest gap between the distribution function of the standardized exponential mean and the normal one, computed exactly (blue), and the Berry–Esseen bound (orange), on logarithmic axes. Notice that both fall with slope -1/2, like 1/root n, and that from n = 2 on the bound sits about 8.6 times above the actual gap.
The largest gap between the distribution function of the standardized exponential mean and the normal one, computed exactly (blue), and the Berry–Esseen bound (orange), on logarithmic axes. Notice that both fall with slope -1/2, like 1/root n, and that from n = 2 on the bound sits about 8.6 times above the actual gap.

The bound is a guarantee, not an estimate. For this population it overstates the error by a factor of about 8.6 at every $n$ from 2 on, while getting the rate exactly right. Its value is that it holds for every population with the stated moments, with no simulation. It makes precise the informal statement that skewed and heavy-tailed populations need larger samples. It also shows that the guarantee can be weak where it matters. An error of 0.01 in a distribution function is small for a probability near one half and large for a tail probability of 0.025.

Every result so far assumed the theorem's hypotheses. What happens when they fail?

Where it fails

Infinite variance

The theorem assumes a finite variance, and the assumption is not a technicality. The Cauchy distribution, with density $1/(\pi(1 + x^2))$, is symmetric about 0 but so heavy-tailed that it has no mean at all.

Check your understanding. Does the mean of 1,000 Cauchy observations locate the center, 0, better than a single observation does?

No. The average of $n$ independent Cauchy variables has exactly the same Cauchy distribution as a single one, for every $n$ (Grinstead and Snell, 1997, §10.3, prove it with characteristic functions). Averaging does not help at all.

Between the two extremes are distributions with a finite mean and infinite variance. The Pareto distribution with shape 1.5 and minimum 1 has density $1.5\,x^{-2.5}$ for $x \ge 1$ and mean 3. The law of large numbers holds, and sample means do settle at 3. But the variance is infinite, and the sample means fluctuate around 3 on a scale that shrinks only like $n^{-1/3}$, not $n^{-1/2}$. Their limiting distribution is skewed and not normal: one of the stable distributions of the generalized central limit theorem (Gnedenko and Kolmogorov, 1954).

The script measured the interquartile range of the sample mean, which exists even when the variance does not, for 2,000 samples of each size and 30 seeds:

$n$ Exponential Pareto, shape 1.5 Cauchy
1 1.100 1.308 1.998
10 0.424 1.163 2.009
100 0.135 0.688 2.005
1,000 0.043 0.358 1.991
Interquartile range of the sample mean against the sample size, on logarithmic axes. Compare the slopes. For the exponential, with finite variance, the range falls by the factor root 10 = 3.16 for every tenfold increase in n. For the Pareto with shape 1.5, whose variance is infinite, it falls more slowly. For the Cauchy, which has no mean, it does not fall at all: the mean of 1,000 observations is as uncertain as a single one.
Interquartile range of the sample mean against the sample size, on logarithmic axes. Compare the slopes. For the exponential, with finite variance, the range falls by the factor root 10 = 3.16 for every tenfold increase in n. For the Pareto with shape 1.5, whose variance is infinite, it falls more slowly. For the Cauchy, which has no mean, it does not fall at all: the mean of 1,000 observations is as uncertain as a single one.
  • For the exponential, the range falls by 3.15 and 3.16 per tenfold increase in $n$ beyond $n = 10$: the $\sqrt{10} = 3.162$ the theorem predicts.
  • For the Pareto, it falls by 1.69 and then 1.92, approaching the $10^{1/3} = 2.154$ of its stable limit only slowly.
  • For the Cauchy, it stays at 2, the distance between the quartiles $-1$ and $1$ of the standard Cauchy distribution. Half of all sample means lie beyond $\pm 1$ whatever $n$ is.

Data with infinite variance are not only a mathematical curiosity. Vilfredo Pareto proposed his distribution to describe incomes. Quantities with a few enormous values, such as costs and insurance claims, are sometimes modeled with tails this heavy, and a moderate sample cannot check the shape of such a tail. Any variable measured in a finite population has a finite variance, CRP included, so the theorem does apply to it in the end. But a variance dominated by a handful of values behaves, for samples of ordinary size, much like an infinite one.

Dependence

The theorem's other condition is independence. Many sequences of dependent observations still have approximately normal means, but the standard error is no longer $\sigma/\sqrt{n}$.

Take a sequence in which each observation is correlated 0.5 with the one before it: a first-order autoregressive process, with correlation $0.5^k$ between observations $k$ apart. For observations with standard deviation 1, the script found the standard deviation of the mean of 25 to be 0.3364 (exact value 0.3370). The independent formula gives $1/\sqrt{25} = 0.2$, so the true value is 1.685 times larger. For $n = 100$ the ratio is 1.720, approaching the large-sample value $\sqrt{(1 + 0.5)/(1 - 0.5)} = \sqrt{3} = 1.732$. An interval built with $\sigma/\sqrt{n}$ would be far too narrow. Linear Regression Assumptions measures the same failure for correlated regression errors.

Dependence can also stop the theorem altogether. Suppose every observation in a sample shares one common component: $X_i = W + E_i$. Here $W$ is a single exponential variable (minus its mean of 1) shared by the whole sample, and the $E_i$ are independent standard normal errors. This is the situation when every measurement in a study is made with one miscalibrated instrument whose offset is itself random. Then $\bar{X} = W + \bar{E}$, and however large $n$ is, the shared $W$ never averages out.

  • The variance of the mean is $1 + 1/n$, which tends to 1, not 0.
  • Its skewness, $2/(1 + 1/n)^{3/2}$, rises toward the skewness 2 of $W$ instead of falling to 0. The script found 0.691, 1.700, 1.948 and 1.982 for $n$ = 1, 10, 100 and 1,000, against theoretical values 0.707, 1.734, 1.970 and 1.997.

Clustered data are the realistic middle ground. There, correlation within clusters inflates the standard error by the design effect of The Binomial Distribution.

Unequal distributions

The identical-distribution assumption, by contrast, can be relaxed a great deal. Lindeberg (1922) gave a condition under which sums of independent variables with different distributions are still asymptotically normal. Feller showed that the condition is also necessary when no single term is a large share of the total variance (Grinstead and Snell, 1997, §9.2). Roughly, the theorem holds as long as no single observation dominates the sum. That is why it applies to sums of contributions of different sizes, such as a patient's total exposure from many sources, provided none of them is overwhelming.

We have seen what the theorem claims, how fast it works and where it fails. Why is it true?

Why it is true: a proof sketch

A full proof of the Lindeberg–Lévy theorem uses characteristic functions and a continuity theorem from analysis. The argument below, the one Grinstead and Snell (1997, §10.3) give, uses moment generating functions instead. It is complete except for one cited theorem, at the price of an extra assumption.

Here is the plan in one sentence. The moment generating function of the standardized mean is the $n$th power of one observation's, evaluated at $t/\sqrt{n}$, and expanding it near zero shows that the power tends to $e^{t^2/2}$, the normal's.

The extra assumption. The moment generating function of a random variable $Y$ is $M_Y(t) = E[e^{tY}]$. Assume it is finite for all $t$ in some interval $(-h, h)$ around 0. This is stronger than a finite variance. It implies that all moments of $Y$ are finite, and it fails for some distributions whose variance is finite, such as the lognormal. It holds for the exponential (for $t < 1$), the normal, and every bounded variable, including any finite population.

Two facts taken from analysis. First, a moment generating function that is finite near 0 determines the distribution. Second, the continuity theorem (Curtiss, 1942): if $M_{Z_n}(t) \to M_Z(t)$ for every $t$ in an interval around 0, and $M_Z$ is finite there, then $P(Z_n \le z) \to P(Z \le z)$ at every $z$ where the distribution function of $Z$ is continuous. For a normal limit that is every $z$. The rest is elementary.

Step 1: the target. For a standard normal $Z$, completing the square in the exponent,

$$M_Z(t) = \int_{-\infty}^{\infty} e^{tz}\,\frac{e^{-z^2/2}}{\sqrt{2\pi}}\,dz = e^{t^2/2}\int_{-\infty}^{\infty} \frac{e^{-(z - t)^2/2}}{\sqrt{2\pi}}\,dz = e^{t^2/2},$$

because the last integrand is the normal density with mean $t$, which integrates to 1 (The Normal Distribution proves it).

Step 2: independence turns a sum into a power. Standardize each observation, $Y_i = (X_i - \mu)/\sigma$, so that $E[Y_i] = 0$ and $E[Y_i^2] = 1$, and write the standardized mean as

$$Z_n = \frac{\bar{X} - \mu}{\sigma/\sqrt{n}} = \frac{1}{\sqrt{n}}\sum_{i=1}^n Y_i.$$

For independent variables the expected value of a product of functions of them factors, so

$$M_{Z_n}(t) = E\Big[\prod_{i=1}^n e^{tY_i/\sqrt{n}}\Big] = \prod_{i=1}^n E\big[e^{tY_i/\sqrt{n}}\big] = \Big[M_Y\Big(\frac{t}{\sqrt{n}}\Big)\Big]^n.$$

Step 3: expand near zero. A moment generating function that is finite near 0 has derivatives of all orders there, with $M_Y(0) = 1$, $M_Y'(0) = E[Y] = 0$ and $M_Y''(0) = E[Y^2] = 1$. By Taylor's theorem,

$$M_Y(s) = 1 + \frac{s^2}{2} + r(s), \qquad \frac{r(s)}{s^2} \to 0 \text{ as } s \to 0.$$

Step 4: take the limit. Fix $t$ and let $a_n = t^2/(2n) + r(t/\sqrt{n})$, so that $M_{Z_n}(t) = (1 + a_n)^n$. Then $a_n \to 0$, and

$$n\,a_n = \frac{t^2}{2} + t^2\,\frac{r(t/\sqrt{n})}{(t/\sqrt{n})^2} \to \frac{t^2}{2}$$

(for $t = 0$ everything equals 1 trivially). Since $\ln(1 + a)/a \to 1$ as $a \to 0$,

$$\ln M_{Z_n}(t) = n\ln(1 + a_n) = n a_n \cdot \frac{\ln(1 + a_n)}{a_n} \to \frac{t^2}{2},$$

so $M_{Z_n}(t) \to e^{t^2/2} = M_Z(t)$. By the continuity theorem, $P(Z_n \le z) \to \Phi(z)$ for every $z$. $\square$

Where each hypothesis enters. Independence is used once, in Step 2. Without it the generating function of the sum does not factor, and the shared-component example shows that the conclusion can fail. The finite variance fixes the $s^2/2$ term in Step 3, the only term that survives the limit. Every feature of the population beyond its mean and variance is in the remainder $r$, which vanishes. That is the precise sense in which the normal limit forgets the shape of the population.

A check on the exponential. Here $Y = X - 1$ has $M_Y(t) = e^{-t}/(1 - t)$. At $t = 0.5$, the values of $[M_Y(0.5/\sqrt{n})]^n$ for $n$ = 1, 10, 100, 1,000 and 10,000 are 1.2131, 1.1502, 1.1381, 1.1347 and 1.1336, approaching $e^{0.125} = 1.1331$.

Dropping the extra assumption. Replace $e^{tY}$ by $e^{itY}$, with $i = \sqrt{-1}$. The resulting characteristic function exists for every distribution, because $\lvert e^{itY}\rvert = 1$. The same four steps go through with Lévy's continuity theorem in place of Curtiss's, and the Taylor expansion in Step 3 then needs only a finite second moment. The Cauchy distribution shows what happens without one. Its characteristic function is $e^{-\lvert t\rvert}$, which has no quadratic term at 0, and the characteristic function of the mean of $n$ Cauchy variables is $(e^{-\lvert t\rvert/n})^n = e^{-\lvert t\rvert}$ again (Grinstead and Snell, 1997, §10.3).

A short history

The theorem grew out of the binomial distribution, and its conditions were sharpened over two centuries.

  • Abraham de Moivre approximated binomial probabilities by the normal curve in the 1730s, using what is now called Stirling's formula for factorials (Grinstead and Snell, 1997, §9.1).
  • Pierre-Simon Laplace extended the approximation to other probabilities of success and to sums of other kinds of errors in the early nineteenth century (Fischer, 2011).
  • Aleksandr Lyapunov proved the theorem for independent variables with different distributions under a condition on their third moments, using characteristic functions.
  • Jarl Waldemar Lindeberg (1922) found the weaker condition that bears his name, which covers the identically distributed case with finite variance.
  • Paul Lévy (1925) developed the characteristic function method into the form used in modern proofs, and the identically distributed theorem is named after both men.
  • William Feller showed that Lindeberg's condition cannot be weakened when no term dominates (Fischer, 2011; Grinstead and Snell, 1997, §9.2).

The name is George Pólya's. His 1920 paper called it the "central" limit theorem, for its central place in probability theory (Pólya, 1920).

Corrections to the course notes

The unit's notes state the theorem for a population with finite standard deviation, give the mean and standard error of the sampling distribution, and note that a normal population gives normal means for every $n$. They list the theorem's limits (it does not make individual observations normal, does not hold for infinite variance, and does not guarantee normality in small samples), all as here. Four points can be stated more precisely.

  • Sample size guidelines. The notes suggest $n \ge 15$ for roughly symmetric populations, $n \ge 30$ for moderately skewed ones and $n \ge 50$ for highly skewed ones. These are reasonable for blood pressure, but no fixed number covers strongly skewed data. The skewness of a mean is $\gamma/\sqrt{n}$, so CRP, with skewness 7.9, has means of 30 more skewed than individual blood pressures, and needs about 1,600 observations to bring the skewness of the mean down to 0.2. The rule of thumb of 30, which Grinstead and Snell (1997, §9.3) also give for sums, is a starting point. It should be checked against the skewness of the data and against the tail that matters, since one-sided probabilities converge more slowly than two-sided ones.
  • The Berry–Esseen bound. The notes give the bound as $C\rho/(\sigma^3\sqrt{n})$ with $\rho$ the third absolute moment. That is correct with $\rho$ the third absolute central moment, $E[\lvert X - \mu\rvert^3]$, and with the bound applying to the largest difference over all $z$. The best known constant is below 0.4748.
  • "Sample means will follow a normal distribution." They follow it approximately, with an error the previous sections measure. Only for a normal population is the normal distribution of the mean exact.
  • Other statistics. The notes say the theorem does not apply in the same way to other statistics such as medians. That is right for the theorem as stated, which concerns means and sums. Medians have sampling distributions of their own, with a different standard error: 1.11 times that of the mean for samples of 25 blood pressures, as The Sampling Distribution of the Mean shows.

Summary

Result Statement
Lindeberg–Lévy independent, identical, finite $\sigma^2$: $P\big(\frac{\bar{X} - \mu}{\sigma/\sqrt{n}} \le z\big) \to \Phi(z)$
Working form $\bar{X} \approx N(\mu, \sigma/\sqrt{n})$, $S_n \approx N(n\mu, \sigma\sqrt{n})$
Skewness of the mean $\gamma/\sqrt{n}$; $n = (\gamma/0.2)^2$ brings it to 0.2
Berry–Esseen $\sup_z \lvert F_n(z) - \Phi(z)\rvert \le C\rho/(\sigma^3\sqrt{n})$, $\rho = E\lvert X - \mu\rvert^3$, $0.4097 \le C < 0.4748$
Normal MGF $E[e^{tZ}] = e^{t^2/2}$
Fails infinite variance (Cauchy: mean of $n$ is Cauchy); a shared component; dependence changes the standard error

What we have learned

The central limit theorem makes the standardized mean of independent observations with finite variance approximately normal, whatever the population looks like. It says nothing about the data themselves and gives no sample size. Speed depends on skewness, which falls only as $\gamma/\sqrt{n}$: means of 30 blood pressures are close to normal, and means of 30 CRP values are not. One-sided tail probabilities converge more slowly than two-sided ones. With infinite variance or a shared component, the theorem fails outright, and ordinary correlation changes the standard error.

With the sampling distribution of the mean in hand, the course can turn it around. If a sample mean lands within about two standard errors of $\mu$ in 95% of samples, then $\mu$ lies within two standard errors of the sample mean in those samples. That observation is the confidence interval, and the next unit begins with it in Confidence Intervals and Coverage.

In Python

The exact distribution of an exponential mean, and a simulation for any population:

import numpy as np
from scipy import stats

n = 30
# exponential with mean 1: the sum of n values is gamma(n); P(Z > 1.96) exactly
print(stats.gamma.sf(n + 1.96 * np.sqrt(n), a=n))    # about 0.0340
print(stats.gamma.cdf(n - 1.96 * np.sqrt(n), a=n))   # about 0.0140

# simulation from a finite population pop (any array of values)
rng = np.random.default_rng(0)
pop = rng.lognormal(size=5000)
means = rng.choice(pop, (10_000, n)).mean(axis=1)
z = (means - pop.mean()) / (pop.std() / np.sqrt(n))
print((z > 1.96).mean(), (z < -1.96).mean(), stats.skew(means))

Every number in this article is computed by tools/experiments/applied-statistics/central-limit-theorem.py in the site's repository, which downloads the NHANES files from the CDC and checks their checksums; the figures are drawn by tools/figures/central-limit-theorem.py.

References

  • Berry, A. C. (1941). The accuracy of the Gaussian approximation to the sum of independent variates. Transactions of the American Mathematical Society, 49(1), 122–136. record
  • Curtiss, J. H. (1942). A note on the theory of moment generating functions. Annals of Mathematical Statistics, 13(4), 430–433. record
  • Esseen, C.-G. (1942). On the Liapounoff limit of error in the theory of probability. Arkiv för Matematik, Astronomi och Fysik, 28A(9), 1–19. record
  • Esseen, C.-G. (1956). A moment inequality with an application to the central limit theorem. Skandinavisk Aktuarietidskrift, 39, 160–170. record
  • Feller, W. (1971). An Introduction to Probability Theory and Its Applications, Vol. II (2nd ed.). Wiley. Chapter XVI (Edgeworth expansions). record
  • Fischer, H. (2011). A History of the Central Limit Theorem: From Classical to Modern Probability Theory. Springer. record
  • Gnedenko, B. V., and Kolmogorov, A. N. (1954). Limit Distributions for Sums of Independent Random Variables. Addison-Wesley. record
  • Grinstead, C. M., and Snell, J. L. (1997). Introduction to Probability (2nd rev. ed.). American Mathematical Society. §9.1–9.3 (the central limit theorem and its history) and §10.3 (the moment generating function proof and the Cauchy average). record · 10 pages
  • Lévy, P. (1925). Calcul des probabilités. Gauthier-Villars, Paris. record
  • Lindeberg, J. W. (1922). Eine neue Herleitung des Exponentialgesetzes in der Wahrscheinlichkeitsrechnung. Mathematische Zeitschrift, 15, 211–225. record
  • National Center for Health Statistics (2020a). National Health and Nutrition Examination Survey 2017–2018: Demographic Variables and Sample Weights (DEMO_J) and Blood Pressure (BPX_J) data files. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 45 pages
  • National Center for Health Statistics (2020b). National Health and Nutrition Examination Survey 2017–2018 Data Documentation: High-Sensitivity C-Reactive Protein (HSCRP_J). Centers for Disease Control and Prevention. https://wwwn.cdc.gov/Nchs/Data/Nhanes/Public/2017/DataFiles/HSCRP_J.htm record
  • Pagano, M., Gauvreau, K., and Mattie, H. (2022). Principles of Biostatistics (3rd ed.). CRC Press. The course text. record · 35 pages
  • Pólya, G. (1920). Über den zentralen Grenzwertsatz der Wahrscheinlichkeitsrechnung und das Momentenproblem. Mathematische Zeitschrift, 8, 171–181. record
  • Shevtsova, I. (2011). On the absolute constants in the Berry–Esseen type inequalities for identically distributed summands. arXiv:1111.6554. record