Draw 30 adults at random, with replacement, from the 5,247 adults with a blood pressure reading in the National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018 (National Center for Health Statistics, 2020). Their systolic pressures have skewness 1.038 and kurtosis 4.680, which is 1.680 above the normal distribution's 3. The mean of the 30 draws has skewness $1.0378/\sqrt{30} = 0.189$ and a kurtosis that exceeds 3 by only $1.680/30 = 0.056$. Where do the divisors $\sqrt{30}$ and 30 come from, and can the whole distribution of a sum be found, not only a few of its moments? Both answers come from one device: a transform of the distribution under which adding independent variables becomes multiplying functions. The divisors come from the logarithm of the transform, whose derivatives, the cumulants, add when independent variables are added. And the whole distribution of a sum can be found whenever the transform of the sum is recognized, because the transform determines the distribution.
The moment generating function of a random variable $X$ is $M_X(t) = E[e^{tX}]$, for the real numbers $t$ at which the expected value is finite. The characteristic function is $\varphi_X(t) = E[e^{itX}]$, with $i = \sqrt{-1}$, and it is finite for every real $t$ and every distribution. Each is a function of $t$ that encodes the distribution of $X$. The first is easier to handle and does not always exist. The second always exists and needs complex numbers.
First we define the moment generating function, find where it is finite and derive it for the standard families. Then we prove that its derivatives at 0 are the moments, define cumulants, and prove the rules for linear changes and for sums of independent variables, which explain the opening. Next we ask whether the function determines the distribution. Then we repeat the program for characteristic functions, prove the expansion that the central limit theorem needs, and state the uniqueness and continuity theorems. Five worked problems close the article. The article assumes Expectation, Variance and Covariance. The course page Moment Generating Functions and Slutsky's Theorem uses a moment generating function for one purpose, a proof of the central limit theorem under an extra assumption, and is not repeated here. This article is the general account, and Convergence in Distribution and the Proof of the Central Limit Theorem builds on its last sections.
The moment generating function and where it is finite
For a discrete variable with probability mass function $p_X$ and for a variable with density $f_X$,
$$M_X(t) = \sum_x e^{tx}\,p_X(x) \qquad\text{or}\qquad M_X(t) = \int_{-\infty}^{\infty} e^{tx} f_X(x)\,dx,$$
by the rule for the expected value of a function of $X$ (Expectation, Variance and Covariance). The integrand is positive, so the sum or integral either converges or is $+\infty$. We say $M_X(t)$ is finite in the first case. Always $M_X(0) = E[1] = 1$.
Proposition (the set of finiteness is an interval). If $M_X(t_1)$ and $M_X(t_2)$ are finite and $t_1 < t < t_2$, then $M_X(t)$ is finite.
Proof. For every real $x$, $e^{tx} \le e^{t_1x} + e^{t_2x}$: if $x \ge 0$ then $tx \le t_2x$, and if $x < 0$ then $tx \le t_1x$. By monotonicity of expected values, $M_X(t) \le M_X(t_1) + M_X(t_2)$. $\square$
So the $t$ at which $M_X$ is finite form an interval containing 0. Three things can happen, and all three occur among common distributions.
- Finite for every $t$. Any bounded variable: if $\lvert X\rvert \le b$ then $e^{tX} \le e^{\lvert t\rvert b}$. A draw from a finite population is bounded, so the blood pressure population has $M_X(t) = \frac{1}{5{,}247}\sum_i e^{tx_i}$ for all $t$. The normal and Poisson distributions also belong here.
- Finite on an interval around 0 that is not the whole line. The exponential distribution with mean 1 has $M_X(t) = 1/(1 - t)$ for $t < 1$ and $+\infty$ for $t \ge 1$.
- Finite only at $t = 0$, or only on one side of 0. The Cauchy distribution has $M_X(t) = +\infty$ for every $t \ne 0$. The lognormal distribution, treated below, has $M_X(t) = +\infty$ for every $t > 0$ although all its moments are finite.
Most of the theory needs the second case at least: $M_X$ finite near 0, meaning finite on some interval $(-h, h)$ with $h > 0$. That is an assumption about both tails of the distribution. It says each tail is at least as thin as an exponential's: for $0 < c < h$, Markov's inequality applied to $e^{cX}$ gives $P(X \ge a) \le e^{-ca}M_X(c)$, the Chernoff bound of Probability Inequalities, and the same for $-X$.
The standard families
Each entry of the table is derived below it. Parameters: $0 < p < 1$ and $q = 1 - p$; $\lambda > 0$; $\alpha > 0$; $a < b$; $\sigma > 0$. The gamma distribution with shape $\alpha$ and rate $\lambda$ has density $\lambda^\alpha x^{\alpha - 1}e^{-\lambda x}/\Gamma(\alpha)$ on $x > 0$, where $\Gamma$ is the gamma function defined in The Chi-Squared and t Distributions; its scale is $1/\lambda$. The geometric distribution here counts the trials up to and including the first success.
| Distribution | $M_X(t)$ | Finite for |
|---|---|---|
| Bernoulli($p$) | $q + pe^t$ | all $t$ |
| Binomial($n$, $p$) | $(q + pe^t)^n$ | all $t$ |
| Geometric($p$), values $1, 2, \ldots$ | $pe^t/(1 - qe^t)$ | $t < -\ln q$ |
| Poisson($\lambda$) | $\exp\big(\lambda(e^t - 1)\big)$ | all $t$ |
| Uniform on $(a, b)$ | $(e^{tb} - e^{ta})/\big(t(b - a)\big)$, and 1 at $t = 0$ | all $t$ |
| Exponential, rate $\lambda$ | $\lambda/(\lambda - t)$ | $t < \lambda$ |
| Gamma, shape $\alpha$, rate $\lambda$ | $\big(\lambda/(\lambda - t)\big)^{\alpha}$ | $t < \lambda$ |
| Chi-squared, $\nu$ degrees of freedom | $(1 - 2t)^{-\nu/2}$ | $t < \tfrac12$ |
| Normal, mean $\mu$, variance $\sigma^2$ | $\exp\big(\mu t + \sigma^2t^2/2\big)$ | all $t$ |
Bernoulli and binomial. The indicator of a success is 1 with probability $p$ and 0 with probability $q$, so $M(t) = q\,e^{0} + p\,e^{t}$. The binomial count is a sum of $n$ independent indicators, and the product rule proved below gives the $n$th power. Moment Generating Functions and Slutsky's Theorem works this case through in its first worked problem.
Geometric. $P(X = k) = q^{k-1}p$ for $k = 1, 2, \ldots$, so
$$M(t) = \sum_{k=1}^{\infty} e^{tk}q^{k-1}p = pe^t\sum_{k=1}^{\infty}(qe^t)^{k-1} = \frac{pe^t}{1 - qe^t},$$
a geometric series that converges exactly when $qe^t < 1$, that is, when $t < -\ln q$.
Poisson. $P(X = k) = e^{-\lambda}\lambda^k/k!$ for $k = 0, 1, \ldots$, so
$$M(t) = \sum_{k=0}^{\infty} e^{tk}\,\frac{e^{-\lambda}\lambda^k}{k!} = e^{-\lambda}\sum_{k=0}^{\infty}\frac{(\lambda e^t)^k}{k!} = e^{-\lambda}e^{\lambda e^t},$$
by the series for the exponential function, which converges for every argument.
Uniform. For $t \ne 0$, $M(t) = \frac{1}{b - a}\int_a^b e^{tx}\,dx = \frac{e^{tb} - e^{ta}}{t(b - a)}$.
Exponential and gamma. For $t < \lambda$ substitute $u = (\lambda - t)x$:
$$M(t) = \int_0^\infty e^{tx}\,\frac{\lambda^\alpha x^{\alpha-1}e^{-\lambda x}}{\Gamma(\alpha)}\,dx = \frac{\lambda^\alpha}{\Gamma(\alpha)}\int_0^\infty x^{\alpha - 1}e^{-(\lambda - t)x}\,dx = \frac{\lambda^\alpha}{\Gamma(\alpha)}\cdot\frac{\Gamma(\alpha)}{(\lambda - t)^{\alpha}} = \Big(\frac{\lambda}{\lambda - t}\Big)^{\alpha}.$$
For $t \ge \lambda$ the integrand is at least $\lambda^\alpha x^{\alpha - 1}/\Gamma(\alpha)$, whose integral to infinity diverges. The exponential distribution is the case $\alpha = 1$. The chi-squared distribution with $\nu$ degrees of freedom is the gamma distribution with shape $\nu/2$ and rate $\tfrac12$, as its density in The Chi-Squared and t Distributions shows, which gives $\big(\tfrac12/(\tfrac12 - t)\big)^{\nu/2} = (1 - 2t)^{-\nu/2}$.
Normal. For a standard normal $Z$, $M_Z(t) = e^{t^2/2}$, by completing the square (Moment Generating Functions and Slutsky's Theorem, Step 1 of its proof). A normal variable with mean $\mu$ and variance $\sigma^2$ is $\mu + \sigma Z$, and the rule for linear changes proved below gives $e^{\mu t}M_Z(\sigma t) = \exp(\mu t + \sigma^2t^2/2)$.
A distribution with every moment and no moment generating function. Let $Z$ be standard normal. The variable $X = e^Z$ has the lognormal distribution. Its moments are finite: $E[X^k] = E[e^{kZ}] = M_Z(k) = e^{k^2/2}$ for every $k$. But for $t > 0$,
$$E[e^{tX}] = \int_{-\infty}^{\infty} \exp\big(te^{z} - z^2/2\big)\,\frac{dz}{\sqrt{2\pi}} = +\infty,$$
because $te^z - z^2/2 \to \infty$ as $z \to \infty$, so the integrand does not even tend to 0. Finite moments of every order do not make $M_X$ finite near 0. The moments here grow like $e^{k^2/2}$, too fast for the series $\sum E[X^k]t^k/k!$ to converge at any $t > 0$.
Moments from derivatives
The name comes from this section. Here is the idea: $e^{tX} = 1 + tX + t^2X^2/2! + \cdots$, and taking expected values term by term gives $M_X(t) = 1 + tE[X] + t^2E[X^2]/2! + \cdots$, a power series whose coefficients are the moments. Taking expected values term by term in an infinite series is the step that needs proof.
Theorem (the moment series). Suppose $M_X$ is finite on $(-h, h)$ for some $h > 0$. Then every moment $E[X^k]$ is finite, and for $\lvert t\rvert < h$
$$M_X(t) = \sum_{k=0}^{\infty} E[X^k]\,\frac{t^k}{k!}.$$
Consequently $M_X$ has derivatives of every order on $(-h, h)$, and $M_X^{(k)}(0) = E[X^k]$ for $k = 0, 1, 2, \ldots$.
Proof. Fix $t$ with $\lvert t\rvert < h$, write $s = \lvert t\rvert$, and choose $c$ with $s < c < h$.
Moments are finite. For every real $x$, $e^{c\lvert x\rvert} \le e^{cx} + e^{-cx}$, so $E[e^{c\lvert X\rvert}] \le M_X(c) + M_X(-c)$, a finite number that we call $B$. Each term of the series for $e^{c\lvert x\rvert}$ is at most the whole sum, so $\lvert x\rvert^k \le k!\,e^{c\lvert x\rvert}/c^k$, and $E[\lvert X\rvert^k] \le k!\,B/c^k$ is finite.
The remainder. For real $x$ and $n \ge 0$, the series for the exponential gives
$$\Big\lvert e^{tx} - \sum_{k=0}^{n}\frac{(tx)^k}{k!}\Big\rvert = \Big\lvert\sum_{k=n+1}^{\infty}\frac{(tx)^k}{k!}\Big\rvert \le \sum_{k=n+1}^{\infty}\frac{(s\lvert x\rvert)^k}{k!}.$$
In each term with $k \ge n + 1$, write $s^k = (s/c)^k c^k \le (s/c)^{n+1}c^k$, which holds because $s/c < 1$. Then
$$\sum_{k=n+1}^{\infty}\frac{(s\lvert x\rvert)^k}{k!} \le \Big(\frac{s}{c}\Big)^{n+1}\sum_{k=n+1}^{\infty}\frac{(c\lvert x\rvert)^k}{k!} \le \Big(\frac{s}{c}\Big)^{n+1}e^{c\lvert x\rvert}.$$
Put $x = X$ and take expected values. By linearity the expected value of the finite sum $\sum_{k \le n}(tX)^k/k!$ is $\sum_{k \le n}E[X^k]t^k/k!$, and by monotonicity
$$\Big\lvert M_X(t) - \sum_{k=0}^{n}E[X^k]\frac{t^k}{k!}\Big\rvert \le \Big(\frac{s}{c}\Big)^{n+1}B \to 0 \quad\text{as } n \to \infty.$$
So the series converges to $M_X(t)$.
Derivatives. A power series can be differentiated term by term at every point inside its interval of convergence, any number of times, and its $k$th derivative at 0 is $k!$ times the coefficient of $t^k$. That is a theorem of calculus, used here without proof. The coefficient is $E[X^k]/k!$. $\square$
Only finitely many terms were ever inside an expected value, so no theorem about exchanging limits and expected values was needed.
Example (the moments of a gamma variable). For the gamma distribution, $M(t) = (1 - t/\lambda)^{-\alpha}$, with derivatives $M'(t) = \frac{\alpha}{\lambda}(1 - t/\lambda)^{-\alpha - 1}$ and $M''(t) = \frac{\alpha(\alpha + 1)}{\lambda^2}(1 - t/\lambda)^{-\alpha - 2}$, and in general $M^{(k)}(0) = \alpha(\alpha + 1)\cdots(\alpha + k - 1)/\lambda^k$. So $E[X] = \alpha/\lambda$, $E[X^2] = \alpha(\alpha + 1)/\lambda^2$ and $\operatorname{Var}(X) = \alpha/\lambda^2$. For the chi-squared distribution, $\alpha = \nu/2$ and $\lambda = \tfrac12$ give mean $\nu$ and variance $2\nu$.
Example (the even moments of the normal). $M_Z(t) = e^{t^2/2} = \sum_{j} t^{2j}/(2^j j!)$. Comparing with $\sum_k E[Z^k]t^k/k!$: the odd moments are 0, and $E[Z^{2j}] = (2j)!/(2^j j!)$, which is 1, 3 and 15 for $j = 1, 2, 3$. So $E[Z^4] = 3$, the reason "excess kurtosis" is measured from 3.
Cumulants
Products are awkward and sums are easy, so it pays to take the logarithm. The cumulant generating function is $K_X(t) = \ln M_X(t)$, defined where $M_X$ is finite. It has derivatives of every order near 0 because $M_X$ does and $M_X > 0$. Its derivatives at 0 are the cumulants $\kappa_k = K_X^{(k)}(0)$.
Proposition (the first four cumulants). If $M_X$ is finite near 0, and $\mu$, $\sigma^2$, $m_3 = E[(X - \mu)^3]$ and $m_4 = E[(X - \mu)^4]$ are the mean, variance and third and fourth central moments of $X$, then
$$\kappa_1 = \mu, \qquad \kappa_2 = \sigma^2, \qquad \kappa_3 = m_3, \qquad \kappa_4 = m_4 - 3\sigma^4.$$
Proof. Let $Y = X - \mu$. Then $M_X(t) = E[e^{t\mu}e^{tY}] = e^{\mu t}M_Y(t)$, so $K_X(t) = \mu t + K_Y(t)$: the first cumulant of $X$ is $\mu$ plus that of $Y$, and the higher cumulants of $X$ and $Y$ agree. Write $M = M_Y$ and $K = \ln M$. Differentiating repeatedly by the chain and quotient rules,
$$K' = \frac{M'}{M}, \qquad K'' = \frac{M''}{M} - \Big(\frac{M'}{M}\Big)^2, \qquad K''' = \frac{M'''}{M} - \frac{3M''M'}{M^2} + 2\Big(\frac{M'}{M}\Big)^3,$$
$$K'''' = \frac{M''''}{M} - \frac{4M'''M'}{M^2} - 3\Big(\frac{M''}{M}\Big)^2 + \frac{12M''(M')^2}{M^3} - 6\Big(\frac{M'}{M}\Big)^4.$$
At $t = 0$ the theorem gives $M(0) = 1$, $M'(0) = E[Y] = 0$, $M''(0) = \sigma^2$, $M'''(0) = m_3$ and $M''''(0) = m_4$. Every term containing $M'$ vanishes, leaving $K'(0) = 0$, $K''(0) = \sigma^2$, $K'''(0) = m_3$ and $K''''(0) = m_4 - 3\sigma^4$. $\square$
The skewness of a distribution is $\gamma = m_3/\sigma^3 = \kappa_3/\kappa_2^{3/2}$. Its kurtosis is $\kappa = m_4/\sigma^4$, which is 3 for every normal distribution, and its excess kurtosis is $\kappa - 3 = m_4/\sigma^4 - 3 = \kappa_4/\kappa_2^2$. The notation needs care at one point. As on the course page One-Sided Confidence Bounds and Sample Size, $\kappa$ with no subscript is the kurtosis; $\kappa_k$ with a subscript is the $k$th cumulant. The fourth cumulant divided by $\sigma^4$ is the excess kurtosis $\kappa - 3$, not $\kappa$. For the normal distribution $K(t) = \mu t + \sigma^2t^2/2$ is a quadratic, so every cumulant beyond the second is 0. Cumulants beyond the second therefore measure departure from the normal shape. For the Poisson distribution $K(t) = \lambda(e^t - 1)$, and every cumulant equals $\lambda$.
For the blood pressures the script computes the cumulants from the central moments, and independently by numerical differentiation of $K$ for the standardized pressures: $\kappa_3/\sigma^3 = 1.038$ and $\kappa_4/\sigma^4 = 1.680$ both ways. So the pressures have skewness $\gamma = 1.038$ and kurtosis $\kappa = 4.680$.
Linear changes and sums of independent variables
This section is the reason for the whole construction.
Theorem (linear change and independent sum). (a) For constants $a$ and $b$, $M_{aX + b}(t) = e^{bt}M_X(at)$ at every $t$ where $M_X(at)$ is finite. (b) If $X_1, \ldots, X_n$ are independent and each $M_{X_j}(t)$ is finite, then
$$M_{X_1 + \cdots + X_n}(t) = M_{X_1}(t)\,M_{X_2}(t)\cdots M_{X_n}(t).$$
Proof. (a) $E[e^{t(aX + b)}] = E[e^{bt}e^{(at)X}] = e^{bt}M_X(at)$ by linearity. (b) $e^{t(X_1 + \cdots + X_n)} = \prod_j e^{tX_j}$, a product of functions of independent variables. The expected value of such a product is the product of the expected values; Expectation, Variance and Covariance proves that rule for two variables, and the product of the first $n - 1$ factors is a function of $(X_1, \ldots, X_{n-1})$, which is independent of $X_n$, so induction on $n$ gives the rule for $n$ factors. $\square$
Taking logarithms, $K_{aX + b}(t) = bt + K_X(at)$ and $K_{X_1 + \cdots + X_n} = K_{X_1} + \cdots + K_{X_n}$. Differentiating $k$ times at 0:
Corollary (cumulants). If the moment generating functions are finite near 0, then $\kappa_k(aX + b) = a^k\kappa_k(X)$ for $k \ge 2$, with $\kappa_1(aX + b) = a\kappa_1(X) + b$; and for independent variables $\kappa_k(X_1 + \cdots + X_n) = \kappa_k(X_1) + \cdots + \kappa_k(X_n)$ for every $k$.
The cases $k = 1$ and $k = 2$ are the familiar rules for the mean and variance of a sum. The cases $k = 3$ and $k = 4$ are the opening of this article.
Corollary (skewness and kurtosis of a mean). Let $\bar{X}_n$ be the mean of $n$ independent variables with a common distribution whose moment generating function is finite near 0, with skewness $\gamma$ and kurtosis $\kappa$. Then $\bar{X}_n$ has skewness $\gamma/\sqrt{n}$ and excess kurtosis $(\kappa - 3)/n$, that is, kurtosis $3 + (\kappa - 3)/n$.
Proof. The sum has cumulants $n\kappa_k$ and the mean, which is the sum times $1/n$, has cumulants $n\kappa_k/n^k = \kappa_k/n^{k-1}$. So the skewness of the mean is $\frac{\kappa_3/n^2}{(\kappa_2/n)^{3/2}} = \frac{\kappa_3}{\kappa_2^{3/2}}\cdot\frac{1}{\sqrt{n}}$, and its excess kurtosis is $\frac{\kappa_4/n^3}{(\kappa_2/n)^2} = \frac{\kappa_4}{\kappa_2^2}\cdot\frac{1}{n} = \frac{\kappa - 3}{n}$. $\square$
Back to the blood pressures. A draw from the population is bounded, so the corollary applies: means of 30 draws have skewness $1.0378/\sqrt{30} = 0.189$ and excess kurtosis $(4.680 - 3)/30 = 0.056$, a kurtosis of 3.056. For $n = 2$ the script checks the corollary against the exact distribution of the sum of two draws, enumerated over all pairs of values: skewness 0.7338 and excess kurtosis 0.8399, equal to $1.038/\sqrt{2}$ and $1.680/2$ computed from unrounded values. The Central Limit Theorem proves the skewness rule by expanding a cube and uses it to say how large a sample a skewed population needs. Here it is one line, and the kurtosis comes with it.
Both rules hold whenever the moments involved are finite, with or without a moment generating function. For the skewness that is the course page's proof. For the kurtosis, let $T_n$ be the sum of the $n$ centered observations. The Laws of Large Numbers proves, in Step 1 of its proof of the strong law and assuming only a finite fourth moment, that $E[T_n^4] = n\,m_4 + 3n(n - 1)\sigma^4$. The variance of $T_n$ is $n\sigma^2$, and the mean $\bar{X}_n$ is a shift of $T_n/n$, which has the same kurtosis as $T_n$. So the kurtosis of the mean is
$$\frac{n\,m_4 + 3n(n - 1)\sigma^4}{n^2\sigma^4} = \frac{\kappa}{n} + 3 - \frac{3}{n} = 3 + \frac{\kappa - 3}{n}.$$
The route through cumulants is the short one when the moment generating function exists.
Does the function determine the distribution?
The rules above compute the moment generating function of a sum. To conclude what the distribution of the sum is, we need to know that no two distributions share a moment generating function.
Theorem (uniqueness; cited). If $M_X$ and $M_Y$ are finite and equal on an interval $(-h, h)$ with $h > 0$, then $X$ and $Y$ have the same distribution.
This is Theorem 1 of Curtiss (1942, p. 431), cited here and not proved. His proof extends $M_X$ to complex arguments, shows by analytic continuation that agreement of $M_X$ and $M_Y$ near 0 forces $\varphi_X = \varphi_Y$ on the whole real line, and then applies the uniqueness theorem for characteristic functions, the corollary of the inversion formula below. That route needs complex analysis and is not given here. One case has a short proof, and it covers every finite population.
Proposition (uniqueness for finite ranges). Let $X$ and $Y$ each take finitely many values. If $M_X(t) = M_Y(t)$ for all $t$ in some open interval, then $X$ and $Y$ have the same distribution.
Proof. Let $x_1 < \cdots < x_m$ be the values taken by either variable and $c_j = P(X = x_j) - P(Y = x_j)$. Then $g(t) = \sum_{j=1}^m c_je^{x_jt} = M_X(t) - M_Y(t)$ is 0 on the interval, so all its derivatives are 0 there. Fix a point $t_0$ inside the interval and put $d_j = c_je^{x_jt_0}$. The derivatives of orders $0, 1, \ldots, m - 1$ at $t_0$ give
$$\sum_{j=1}^m x_j^k\,d_j = 0, \qquad k = 0, 1, \ldots, m - 1.$$
This is a system of $m$ linear equations in $d_1, \ldots, d_m$ whose matrix has entries $x_j^k$, a Vandermonde matrix. Its determinant is $\prod_{j < l}(x_l - x_j)$, which is not 0 because the $x_j$ are distinct. So every $d_j = 0$, hence every $c_j = 0$. $\square$
Grinstead and Snell (1997, §10.1, Theorem 10.1) prove the case in which the function is known for every $t$, by another argument that lets $t \to \infty$ to recover the largest value first.
With uniqueness, the table of the standard families yields the families that are closed under addition of independent variables. In each line the product of the moment generating functions is recognized as a member of the same family:
- Normal. $\exp(\mu_1t + \sigma_1^2t^2/2)\exp(\mu_2t + \sigma_2^2t^2/2) = \exp\big((\mu_1 + \mu_2)t + (\sigma_1^2 + \sigma_2^2)t^2/2\big)$. The sum of independent normal variables is normal, with the means and the variances added. The Sampling Distribution of the Mean cites this fact; here it is proved, given the uniqueness theorem.
- Poisson. $\exp(\lambda_1(e^t - 1))\exp(\lambda_2(e^t - 1)) = \exp((\lambda_1 + \lambda_2)(e^t - 1))$. The Poisson Distribution proves the same by summing probabilities directly.
- Binomial with a common $p$. $(q + pe^t)^{n_1}(q + pe^t)^{n_2} = (q + pe^t)^{n_1 + n_2}$.
- Gamma with a common rate. $\big(\lambda/(\lambda - t)\big)^{\alpha_1}\big(\lambda/(\lambda - t)\big)^{\alpha_2} = \big(\lambda/(\lambda - t)\big)^{\alpha_1 + \alpha_2}$. In particular the sum of $n$ independent exponential variables with rate $\lambda$ is gamma with shape $n$ and rate $\lambda$, and chi-squared variables add their degrees of freedom.
Two of the four lines do not depend on the cited theorem. For the Poisson the course page reaches the conclusion by summing probabilities, and for the binomial it follows from the definition of the count as the number of successes in $n_1 + n_2$ independent trials. For the normal and the gamma the cited theorem is used.
Moments alone do not determine a distribution
Example (a tempting converse). "Two distributions with the same moments of every order are the same distribution" is false. When $M_X$ is finite near 0 the moments do determine the distribution, because they determine $M_X$ through the moment series and then the uniqueness theorem applies. Without that condition they need not. The lognormal distribution is the standard counterexample. Heyde (1963) established that it is not determined by its moments, and Durrett (2019, §3.3.5) gives the following family of densities under his name.
Let $f_0(x) = \frac{1}{x\sqrt{2\pi}}e^{-(\ln x)^2/2}$ for $x > 0$ be the lognormal density, and for $-1 \le a \le 1$ let
$$f_a(x) = f_0(x)\big(1 + a\sin(2\pi\ln x)\big).$$
Each $f_a$ is nonnegative. We show that $\int_0^\infty x^kf_0(x)\sin(2\pi\ln x)\,dx = 0$ for $k = 0, 1, 2, \ldots$. The case $k = 0$ says that $f_a$ integrates to 1, so it is a density, and the other cases say that $f_a$ has the same moments as $f_0$. Substitute $x = e^{u}$, so that $dx = e^u\,du$ and $f_0(e^u)\,e^u = e^{-u^2/2}/\sqrt{2\pi}$:
$$\int_0^\infty x^kf_0(x)\sin(2\pi\ln x)\,dx = \int_{-\infty}^{\infty} e^{ku}\,\frac{e^{-u^2/2}}{\sqrt{2\pi}}\,\sin(2\pi u)\,du = e^{k^2/2}\int_{-\infty}^{\infty}\frac{e^{-(u - k)^2/2}}{\sqrt{2\pi}}\,\sin(2\pi u)\,du,$$
by completing the square, $ku - u^2/2 = k^2/2 - (u - k)^2/2$. Now substitute $v = u - k$. Because $k$ is a whole number, $\sin(2\pi(v + k)) = \sin(2\pi v)$, and the integral becomes $\int e^{-v^2/2}\sin(2\pi v)\,dv/\sqrt{2\pi}$, the integral of an odd function, which is 0.
So every $f_a$ has the moments $e^{k^2/2}$ of the lognormal. The script confirms it numerically for $a = 1$ and $k = 0, \ldots, 4$: 1.0000, 1.6487, 7.3891, 90.0171 and 2,980.96 for both densities. The distributions are far from equal. Under $f_0$ the probability of a value at most 1 is 0.5000, and under $f_1$ it is 0.4348. A value between 1 and $\sqrt{e}$ has probability 0.1915 under $f_0$ and 0.3139 under $f_1$.
Characteristic functions
The moment generating function fails for heavy-tailed distributions because $e^{tx}$ is unbounded. Replacing $t$ by $it$ makes the function bounded: $e^{itx} = \cos(tx) + i\sin(tx)$ has modulus 1.
A complex-valued random variable $W = U + iV$, with $U$ and $V$ real random variables with finite means, has expected value $E[W] = E[U] + iE[V]$. Linearity holds for complex constants, by multiplying out real and imaginary parts. The characteristic function of a real random variable $X$ is
$$\varphi_X(t) = E[e^{itX}] = E[\cos(tX)] + i\,E[\sin(tX)], \qquad t \text{ real}.$$
Both expected values exist because cosine and sine are bounded by 1.
One inequality is used throughout: $\lvert E[W]\rvert \le E[\lvert W\rvert]$ for complex $W$. To see it, write $E[W] = re^{i\theta}$ with $r = \lvert E[W]\rvert$. Then $r = e^{-i\theta}E[W] = E[e^{-i\theta}W]$ is real, so it equals $E[\operatorname{Re}(e^{-i\theta}W)]$, and $\operatorname{Re}(e^{-i\theta}W) \le \lvert e^{-i\theta}W\rvert = \lvert W\rvert$.
Theorem (basic properties). For every random variable $X$ and all real $t$, $h$, $a$, $b$:
- $\varphi_X(0) = 1$ and $\lvert\varphi_X(t)\rvert \le 1$.
- $\varphi_X(-t) = \overline{\varphi_X(t)}$, the complex conjugate.
- $\lvert\varphi_X(t + h) - \varphi_X(t)\rvert \le E[\lvert e^{ihX} - 1\rvert]$, and the right side tends to 0 as $h \to 0$. So $\varphi_X$ is uniformly continuous.
- $\varphi_{aX + b}(t) = e^{ibt}\varphi_X(at)$.
- If $X_1, \ldots, X_n$ are independent, $\varphi_{X_1 + \cdots + X_n}(t) = \varphi_{X_1}(t)\cdots\varphi_{X_n}(t)$.
Proof. (1) $\varphi_X(0) = E[1] = 1$, and $\lvert\varphi_X(t)\rvert \le E[\lvert e^{itX}\rvert] = 1$. (2) $\cos$ is even and $\sin$ is odd, so $\varphi_X(-t) = E[\cos(tX)] - iE[\sin(tX)]$. (3) $\lvert\varphi_X(t + h) - \varphi_X(t)\rvert = \lvert E[e^{itX}(e^{ihX} - 1)]\rvert \le E[\lvert e^{ihX} - 1\rvert]$, and the bound does not depend on $t$. To see that it tends to 0, note $\lvert e^{iu} - 1\rvert \le \min(2, \lvert u\rvert)$: the 2 is the triangle inequality, and $\lvert e^{iu} - 1\rvert = \lvert\int_0^u ie^{is}\,ds\rvert \le \lvert u\rvert$. So for any $b > 0$, splitting according to whether $\lvert X\rvert \le b$,
$$E[\lvert e^{ihX} - 1\rvert] \le \lvert h\rvert\,b + 2\,P(\lvert X\rvert > b).$$
Given $\varepsilon > 0$, choose $b$ with $2P(\lvert X\rvert > b) < \varepsilon/2$, possible because $P(\lvert X\rvert > b) \to 0$ as $b \to \infty$ (Random Variables and Distribution Functions: a distribution function tends to 0 and 1 at the two ends). Then the right side is below $\varepsilon$ for $\lvert h\rvert < \varepsilon/(2b)$. (4) $E[e^{it(aX + b)}] = e^{itb}E[e^{i(at)X}]$. (5) For two variables, $e^{it(X_1 + X_2)} = (\cos tX_1 + i\sin tX_1)(\cos tX_2 + i\sin tX_2)$. Multiplied out, this is four products of a function of $X_1$ and a function of $X_2$. The expected value of each factors by independence, and the four products of expected values recombine into $\varphi_{X_1}(t)\varphi_{X_2}(t)$. Induction on $n$, as for moment generating functions, gives the general case. $\square$
Durrett (2019, Theorems 3.3.1 and 3.3.2) lists the same properties.
The standard families again
| Distribution | $\varphi_X(t)$ |
|---|---|
| Bernoulli($p$) | $q + pe^{it}$ |
| Binomial($n$, $p$) | $(q + pe^{it})^n$ |
| Poisson($\lambda$) | $\exp\big(\lambda(e^{it} - 1)\big)$ |
| Uniform on $(a, b)$ | $(e^{itb} - e^{ita})/\big(it(b - a)\big)$ |
| Uniform on $(-c, c)$ | $\sin(ct)/(ct)$ |
| Exponential, rate $\lambda$ | $\lambda/(\lambda - it)$ |
| Gamma, shape $\alpha$, rate $\lambda$ | $(1 - it/\lambda)^{-\alpha}$ |
| Normal, mean $\mu$, variance $\sigma^2$ | $\exp\big(i\mu t - \sigma^2t^2/2\big)$ |
| Laplace, density $\tfrac12e^{-\lvert x\rvert}$ | $1/(1 + t^2)$ |
| Cauchy, density $1/\big(\pi(1 + x^2)\big)$ | $e^{-\lvert t\rvert}$ |
In every row that also appears in the first table, $\varphi_X(t)$ is $M_X$ with $t$ replaced by $it$. That is a pattern across these rows and is not used as a rule below, because a formula valid for real $t$ does not extend to complex arguments without an argument of its own. Each row is derived directly.
Discrete families. The series for $e^z$ converges for complex $z$ and the binomial theorem holds for complex numbers, so the computations of the first table go through with $e^{it}$ in place of $e^t$: $q + pe^{it}$, its $n$th power by property 5, and $e^{-\lambda}\sum_k(\lambda e^{it})^k/k! = \exp(\lambda(e^{it} - 1))$.
Uniform, exponential and Laplace. For a complex constant $c \ne 0$ the function $e^{cx}/c$ has derivative $e^{cx}$, as differentiating the real and imaginary parts of $e^{(\alpha + i\beta)x} = e^{\alpha x}(\cos\beta x + i\sin\beta x)$ confirms. So $\frac{1}{b - a}\int_a^b e^{itx}\,dx = \frac{e^{itb} - e^{ita}}{it(b - a)}$, and with $a = -c$, $b = c$ this is $\frac{e^{itc} - e^{-itc}}{2itc} = \frac{\sin(ct)}{ct}$. For the exponential,
$$\int_0^\infty e^{itx}\lambda e^{-\lambda x}\,dx = \lambda\Big[\frac{e^{-(\lambda - it)x}}{-(\lambda - it)}\Big]_0^\infty = \frac{\lambda}{\lambda - it},$$
since $\lvert e^{-(\lambda - it)x}\rvert = e^{-\lambda x} \to 0$. A Laplace variable is $X$ or $-X$ with probability $\tfrac12$ each, for $X$ exponential with rate 1. Its density $\tfrac12e^{-\lvert x\rvert}$ is the average of the densities of $X$ and $-X$, so its characteristic function is the average of theirs, $\frac12\cdot\frac{1}{1 - it} + \frac12\cdot\frac{1}{1 + it} = \frac{1}{1 + t^2}$.
Normal and gamma. These two need a derivative of $\varphi$, so they are derived after the next theorem. The Cauchy row needs the inversion formula and is derived after that.
Moments from a characteristic function
A characteristic function need not have a power series, since moments need not exist. What survives is a Taylor expansion to the order of the moments that do exist, and its proof starts from an estimate for $e^{ix}$ that is sharper than the usual remainder.
Lemma (remainder for the complex exponential). For real $x$ and $n = 0, 1, 2, \ldots$,
$$\Big\lvert e^{ix} - \sum_{k=0}^{n}\frac{(ix)^k}{k!}\Big\rvert \le \min\Big(\frac{\lvert x\rvert^{n+1}}{(n + 1)!},\ \frac{2\lvert x\rvert^n}{n!}\Big).$$
The first bound is the usual Taylor remainder, good for small $x$. The second is better for large $x$, and it is what allows a conclusion with only $n$ finite moments.
Proof. Write $R_n(x)$ for the difference inside the bars. We prove both bounds by induction on $n$, for $x \ge 0$ first. For $n = 0$: $R_0(x) = e^{ix} - 1 = \int_0^x ie^{is}\,ds$, so $\lvert R_0(x)\rvert \le x$, and $\lvert R_0(x)\rvert \le \lvert e^{ix}\rvert + 1 = 2$. For the step, note that $R_n(0) = 0$ and
$$\frac{d}{dx}R_n(x) = ie^{ix} - \sum_{k=1}^{n}\frac{i\,(ix)^{k-1}}{(k - 1)!} = i\,R_{n-1}(x), \qquad\text{so}\qquad R_n(x) = i\int_0^x R_{n-1}(s)\,ds.$$
If $\lvert R_{n-1}(s)\rvert \le s^n/n!$ for $s \ge 0$ then $\lvert R_n(x)\rvert \le \int_0^x s^n/n!\,ds = x^{n+1}/(n + 1)!$. If $\lvert R_{n-1}(s)\rvert \le 2s^{n-1}/(n - 1)!$ then $\lvert R_n(x)\rvert \le \int_0^x 2s^{n-1}/(n - 1)!\,ds = 2x^n/n!$. For $x < 0$, $R_n(x)$ is the complex conjugate of $R_n(-x)$, because conjugation replaces $i$ by $-i$ throughout, and conjugate numbers have the same modulus. $\square$
This is Lemma 3.3.19 of Durrett (2019), proved there by integration by parts.
The next step replaces $x$ by $tX$ and takes expected values. It needs one fact about finite expected values.
Lemma (tail fact). If $E[\lvert W\rvert]$ is finite, where $W = g(X)$ is a function of $X$, then $E\big[\lvert W\rvert\,\mathbf{1}\{\lvert X\rvert > b\}\big] \to 0$ as $b \to \infty$. In fact $W$ may be any random variable with a finite mean, on the same sample space as $X$.
Proof. Let $V = \lvert W\rvert\,\mathbf{1}\{\lvert X\rvert > b\}$, a nonnegative variable. Expectation, Variance and Covariance defines the expected value of a nonnegative variable, for every kind of random variable, as $E[V] = \int_0^\infty P(V > t)\,dt$. For $t \ge 0$ the event $V > t$ requires both $\lvert W\rvert > t$ and $\lvert X\rvert > b$, so $P(V > t) \le P(\lvert W\rvert > t)$ and $P(V > t) \le P(\lvert X\rvert > b)$. Using the second bound on $[0, T]$ and the first beyond $T$, for any $T > 0$,
$$E[V] \le T\,P(\lvert X\rvert > b) + \int_T^\infty P(\lvert W\rvert > t)\,dt.$$
Given $\varepsilon > 0$, choose $T$ so large that the integral is below $\varepsilon/2$. That is possible because $\int_0^\infty P(\lvert W\rvert > t)\,dt = E[\lvert W\rvert]$ converges. Then choose $b$ so large that $T\,P(\lvert X\rvert > b) < \varepsilon/2$, possible because a distribution function tends to 0 and 1 at the two ends (Random Variables and Distribution Functions). For that $b$ and every larger one, $E[V] < \varepsilon$. $\square$
In texts built on measure theory this is an instance of the dominated convergence theorem. With the unit's definition of the expected value it needs no such theorem. For a discrete $X$ and $W = g(X)$ it says that what remains of the convergent series $\sum_x \lvert g(x)\rvert p_X(x)$ after the terms with $\lvert x\rvert \le b$ are removed tends to 0, and for $X$ with a density, that the tails $\int_{\lvert x\rvert > b}\lvert g(x)\rvert f_X(x)\,dx$ of a convergent integral tend to 0.
Theorem (expansion of a characteristic function). If $E[\lvert X\rvert^n]$ is finite for a whole number $n \ge 1$ (the moments of lower order are then finite too, by the monotonicity of moments in Probability Inequalities), then
$$\varphi_X(t) = \sum_{k=0}^{n}\frac{(it)^k}{k!}E[X^k] + r_n(t), \qquad \frac{r_n(t)}{t^n} \to 0 \text{ as } t \to 0.$$
In particular, if $E[X^2]$ is finite,
$$\varphi_X(t) = 1 + it\,E[X] - \frac{t^2}{2}E[X^2] + r_2(t), \qquad \frac{r_2(t)}{t^2} \to 0 \text{ as } t \to 0.$$
Proof. By linearity, $r_n(t) = E\big[e^{itX} - \sum_{k \le n}(itX)^k/k!\big]$, and by the lemma with $x = tX$,
$$\lvert r_n(t)\rvert \le E\Big[\min\Big(\frac{\lvert tX\rvert^{n+1}}{(n + 1)!},\ \frac{2\lvert tX\rvert^n}{n!}\Big)\Big] = \lvert t\rvert^n\,E\Big[\min\Big(\frac{\lvert t\rvert\,\lvert X\rvert^{n+1}}{(n + 1)!},\ \frac{2\lvert X\rvert^n}{n!}\Big)\Big].$$
Fix $b > 0$. Where $\lvert X\rvert \le b$ the minimum is at most its first entry, which is at most $\lvert t\rvert b^{n+1}/(n + 1)!$. Where $\lvert X\rvert > b$ it is at most its second entry. So
$$\frac{\lvert r_n(t)\rvert}{\lvert t\rvert^n} \le \frac{\lvert t\rvert\,b^{n+1}}{(n + 1)!} + \frac{2}{n!}\,E\big[\lvert X\rvert^n\,\mathbf{1}\{\lvert X\rvert > b\}\big].$$
Given $\varepsilon > 0$, the tail fact gives a $b$ for which the second term is below $\varepsilon/2$, and then the first is below $\varepsilon/2$ for all small $\lvert t\rvert$. $\square$
The case $n = 2$ is Theorem 3.3.20 of Durrett (2019). It is the step on which the proof of the central limit theorem under a finite variance alone rests, in Convergence in Distribution and the Proof of the Central Limit Theorem. A plain Taylor remainder would bound $r_2$ by $\lvert t\rvert^3E[\lvert X\rvert^3]/6$, which needs a third moment. The second entry of the minimum removes that need.
Theorem (derivatives of a characteristic function). If $E[\lvert X\rvert^n]$ is finite, then $\varphi_X$ has $n$ derivatives, $\varphi_X^{(k)}(t) = E\big[(iX)^ke^{itX}\big]$ for $k \le n$, and in particular $\varphi_X^{(k)}(0) = i^kE[X^k]$.
Proof. By induction on $k$; the case $k = 0$ is the definition. Suppose the formula holds for some $k < n$. For $h \ne 0$, by linearity,
$$\frac{\varphi_X^{(k)}(t + h) - \varphi_X^{(k)}(t)}{h} - E\big[(iX)^{k+1}e^{itX}\big] = E\Big[(iX)^ke^{itX}\,\frac{e^{ihX} - 1 - ihX}{h}\Big].$$
By the lemma with $n = 1$ and $x = hX$, $\lvert e^{ihX} - 1 - ihX\rvert \le \min(h^2X^2/2,\ 2\lvert hX\rvert)$. So the modulus of the right side is at most $E\big[\lvert X\rvert^k\min(\lvert h\rvert X^2/2,\ 2\lvert X\rvert)\big]$, and splitting at $\lvert X\rvert \le b$ as before this is at most $\lvert h\rvert b^{k+2}/2 + 2E\big[\lvert X\rvert^{k+1}\mathbf{1}\{\lvert X\rvert > b\}\big]$. Since $k + 1 \le n$, $E[\lvert X\rvert^{k+1}]$ is finite, and the bound tends to 0 as $h \to 0$ by the tail fact, as in the previous proof. So the derivative of $\varphi_X^{(k)}$ exists and equals $E[(iX)^{k+1}e^{itX}]$. $\square$
Durrett (2019, Theorem 3.3.18) states this result, with continuity of the $n$th derivative added.
The normal distribution. Let $Z$ be standard normal with density $f(z) = e^{-z^2/2}/\sqrt{2\pi}$. Since $\sin(tz)f(z)$ is an odd function of $z$, $\varphi_Z(t) = E[\cos(tZ)]$. By the theorem with $k = 1$, $\varphi_Z'(t) = E[iZe^{itZ}] = -E[Z\sin(tZ)] + iE[Z\cos(tZ)]$, and the last term is 0 because its integrand is odd. Now $zf(z) = -f'(z)$, so integrating by parts,
$$\varphi_Z'(t) = \int_{-\infty}^{\infty}\sin(tz)\,f'(z)\,dz = \big[\sin(tz)f(z)\big]_{-\infty}^{\infty} - t\int_{-\infty}^{\infty}\cos(tz)\,f(z)\,dz = -t\,\varphi_Z(t).$$
Therefore $\frac{d}{dt}\big(\varphi_Z(t)e^{t^2/2}\big) = \big(\varphi_Z'(t) + t\varphi_Z(t)\big)e^{t^2/2} = 0$, so $\varphi_Z(t)e^{t^2/2}$ is constant, equal to its value 1 at $t = 0$. Hence $\varphi_Z(t) = e^{-t^2/2}$. Property 4 gives $\exp(i\mu t - \sigma^2t^2/2)$ for $\mu + \sigma Z$. This is the derivation in Durrett (2019, Example 3.3.5).
The gamma distribution. Let $X$ be gamma with shape $\alpha$ and rate $\lambda$. Its mean is finite, so $\varphi_X'(t) = E[iXe^{itX}] = \frac{i\lambda^\alpha}{\Gamma(\alpha)}\int_0^\infty x^{\alpha}e^{-(\lambda - it)x}\,dx$. Integrate by parts with $x^\alpha$ to be differentiated and $e^{-(\lambda - it)x}$ to be integrated. The boundary terms vanish, at 0 because $\alpha > 0$ and at infinity because $\lvert e^{-(\lambda - it)x}\rvert = e^{-\lambda x}$, and
$$\int_0^\infty x^{\alpha}e^{-(\lambda - it)x}\,dx = \frac{\alpha}{\lambda - it}\int_0^\infty x^{\alpha - 1}e^{-(\lambda - it)x}\,dx, \qquad\text{so}\qquad \varphi_X'(t) = \frac{i\alpha}{\lambda - it}\,\varphi_X(t).$$
It remains to exhibit the solution of this equation with value 1 at 0. The complex number $1 - it/\lambda$ has modulus $\sqrt{1 + t^2/\lambda^2}$ and argument $-\arctan(t/\lambda)$, so it equals $e^{L(t)}$ with $L(t) = \tfrac12\ln(1 + t^2/\lambda^2) - i\arctan(t/\lambda)$. Define $(1 - it/\lambda)^{-\alpha} = e^{-\alpha L(t)}$ and call it $g(t)$. Differentiating,
$$L'(t) = \frac{t}{\lambda^2 + t^2} - \frac{i\lambda}{\lambda^2 + t^2} = \frac{-i(\lambda + it)}{(\lambda - it)(\lambda + it)} = \frac{-i}{\lambda - it},$$
so $g'(t) = -\alpha L'(t)g(t) = \frac{i\alpha}{\lambda - it}g(t)$, the same equation, and $g(0) = 1$. Since $g$ is never 0, the quotient rule gives $(\varphi_X/g)' = (\varphi_X'g - \varphi_Xg')/g^2 = 0$. So $\varphi_X/g$ is constant, equal to 1, and $\varphi_X(t) = (1 - it/\lambda)^{-\alpha}$. For a whole number $\alpha$ this is the $\alpha$th power of the exponential's $\lambda/(\lambda - it)$, as property 5 requires of a sum of $\alpha$ independent exponential variables.
Uniqueness and inversion
Theorem (inversion formula; cited). Let $X$ have characteristic function $\varphi$. For all $a < b$,
$$\lim_{T \to \infty}\frac{1}{2\pi}\int_{-T}^{T}\frac{e^{-ita} - e^{-itb}}{it}\,\varphi(t)\,dt = P(a < X < b) + \tfrac12\big(P(X = a) + P(X = b)\big).$$
This is Theorem 3.3.11 of Durrett (2019), cited and not proved here. Its proof exchanges the order of two integrations and passes a limit through an expected value, the two operations this unit does without.
Corollary (uniqueness). If $\varphi_X(t) = \varphi_Y(t)$ for every real $t$, then $X$ and $Y$ have the same distribution.
Proof, given the formula. The right side of the formula is then the same for $X$ and $Y$, for all $a < b$. A distribution function has at most countably many jumps: the jumps add up to at most 1, so at most $k$ of them exceed $1/k$, for each $k$. The set $D$ of points where neither $F_X$ nor $F_Y$ jumps therefore leaves out only countably many points, and every interval contains points of $D$. For $a < b$ in $D$ the formula gives $F_X(b) - F_X(a) = F_Y(b) - F_Y(a)$. Letting $a \to -\infty$ through $D$ gives $F_X(b) = F_Y(b)$ for $b$ in $D$, and since distribution functions are continuous from the right and every point is a limit from the right of points of $D$, $F_X = F_Y$ everywhere. $\square$
Unlike the theorem for moment generating functions, this needs no assumption: characteristic functions always exist, and they always determine the distribution. Agreement at every $t$ is needed, though. Durrett (2019, Example 3.3.24) exhibits two different distributions whose characteristic functions agree on the interval $[-1, 1]$.
Theorem (inversion for an integrable characteristic function; cited). If $\int_{-\infty}^{\infty}\lvert\varphi(t)\rvert\,dt$ is finite, then $X$ has a bounded continuous density
$$f(y) = \frac{1}{2\pi}\int_{-\infty}^{\infty}e^{-ity}\,\varphi(t)\,dt.$$
This is Theorem 3.3.14 of Durrett (2019).
The Cauchy distribution. The Laplace distribution has the integrable characteristic function $1/(1 + t^2)$ and the density $\tfrac12e^{-\lvert y\rvert}$, so the last theorem gives
$$\frac12e^{-\lvert y\rvert} = \frac{1}{2\pi}\int_{-\infty}^{\infty}\frac{e^{-ity}}{1 + t^2}\,dt.$$
Multiply by 2 and rename the variables, $t \to x$ and $y \to -t$:
$$e^{-\lvert t\rvert} = \int_{-\infty}^{\infty}e^{itx}\,\frac{1}{\pi(1 + x^2)}\,dx.$$
The right side is the characteristic function of the Cauchy density $1/(\pi(1 + x^2))$. So the Cauchy distribution has $\varphi(t) = e^{-\lvert t\rvert}$, a derivation that rests on the cited inversion theorem (Durrett, 2019, Example 3.3.16). The function $e^{-\lvert t\rvert}$ has a corner at 0. By the theorem on derivatives, a distribution with a finite mean has a differentiable characteristic function, so the corner is the mark of the Cauchy distribution's missing mean.
The continuity theorem
Uniqueness says that equal characteristic functions mean equal distributions. The continuity theorem says that nearly equal characteristic functions mean nearly equal distributions, in the sense of convergence in distribution: $X_n \xrightarrow{d} X$ means that $F_{X_n}(x) \to F_X(x)$ at every $x$ at which $F_X$ is continuous. Convergence in Distribution and the Proof of the Central Limit Theorem is the home of that definition and its properties.
Theorem (Lévy's continuity theorem; cited). Let $X_1, X_2, \ldots$ be random variables with characteristic functions $\varphi_1, \varphi_2, \ldots$.
- If $X_n \xrightarrow{d} X$, then $\varphi_n(t) \to \varphi_X(t)$ for every real $t$.
- If $\varphi_n(t)$ converges for every real $t$ to a limit $g(t)$, and $g$ is continuous at $t = 0$, then $g$ is the characteristic function of some random variable $X$, and $X_n \xrightarrow{d} X$.
This is Theorem 3.3.17 of Durrett (2019), cited and not proved here; the proof needs a compactness theorem for distribution functions. In the usual application the limit $g$ is recognized as a known characteristic function, such as $e^{-t^2/2}$, which is continuous, and part 2 then delivers convergence in distribution to that law. The theorem for moment generating functions that the course page uses (Curtiss, 1942, Theorem 3, p. 432) is the analogue for functions finite near 0. Curtiss derives it from a form of part 2, a theorem he credits to P. Lévy and reports as "sometimes called Lévy's Continuity Theorem" (p. 431).
Example (why continuity at 0 is required). Convergence of $\varphi_n(t)$ at every $t$ is not enough by itself for the distributions to converge. Let $X_n$ be normal with mean 0 and variance $n$. Then $\varphi_n(t) = e^{-nt^2/2}$, which tends to 0 for every $t \ne 0$ and to 1 at $t = 0$. The limit function is not continuous at 0, so by property 3 it is not a characteristic function. And the distributions do not converge: $P(X_n \le x) = \Phi(x/\sqrt{n}) \to \tfrac12$ for every $x$, and the constant $\tfrac12$ is not a distribution function. The probability spreads out and escapes to infinity. Durrett (2019) gives this example in the remark after the theorem.
What is proved here, what is cited, what is computed
Proved here, in full: that the set where $M_X$ is finite is an interval; the moment generating functions of the nine families of the first table (the standard normal's is proved on the course page); that the lognormal has no moment generating function on $t > 0$; the moment series and $M_X^{(k)}(0) = E[X^k]$; the first four cumulants; the rules for linear changes and independent sums, and the skewness and excess kurtosis of a mean; uniqueness for finite ranges; Heyde's family of densities with the lognormal's moments; the basic properties of characteristic functions; the characteristic functions of the table except the Cauchy's; the remainder lemma, the tail fact, the expansion to the order of the finite moments, and the derivative formula.
Cited, not proved: the uniqueness theorem for moment generating functions finite near 0 (Curtiss, 1942, Theorem 1, p. 431); the inversion formula and its form for integrable characteristic functions (Durrett, 2019, Theorems 3.3.11 and 3.3.14), on which the uniqueness corollary and the Cauchy's characteristic function rest; Lévy's continuity theorem (Durrett, 2019, Theorem 3.3.17). The statements about families closed under addition use the cited uniqueness theorem for the normal and gamma families.
Where measure theory is avoided. Expected values are those of Expectation, Variance and Covariance, defined for every random variable without the Lebesgue integral (Random Variables and Distribution Functions says what is left out). The proofs here pass a limit through an expected value at one point only, the tail fact, and that lemma is proved above from the definition, where a text built on measure theory would use the dominated convergence theorem. Term-by-term differentiation of a power series and the Vandermonde determinant are taken from calculus and linear algebra.
Computed: the cumulants of the blood pressures and the check for sums of two draws are exact properties of the 5,247 values. The moments and probabilities for Heyde's density are numerical integrals.
Worked problems
These problems were written for this article; none comes from a textbook. They use the families of the two tables, with numbers chosen so that every answer can be checked exactly. The script tools/experiments/statistics/moment-generating-and-characteristic-functions.py in the site's repository computes every number and asserts it. Decimals are rounded from unrounded values.
Problem 1 (the cumulants of a gamma variable). Let $X$ be gamma with shape $\alpha$ and rate $\lambda$. (a) Find its cumulant generating function and all its cumulants. (b) Find its skewness and excess kurtosis. (c) Apply the result to the mean of 30 independent exponential waiting times with mean 1.
Solution. (a) From the table, $M_X(t) = (1 - t/\lambda)^{-\alpha}$ for $t < \lambda$, so
$$K_X(t) = -\alpha\ln\Big(1 - \frac{t}{\lambda}\Big).$$
Differentiating, $K_X'(t) = \frac{\alpha}{\lambda}(1 - t/\lambda)^{-1}$, $K_X''(t) = \frac{\alpha}{\lambda^2}(1 - t/\lambda)^{-2}$, and each further derivative multiplies by the current exponent's size and by $1/\lambda$: $K_X^{(k)}(t) = \frac{\alpha\,(k - 1)!}{\lambda^k}(1 - t/\lambda)^{-k}$. (Check by induction: differentiating $(1 - t/\lambda)^{-k}$ gives $\frac{k}{\lambda}(1 - t/\lambda)^{-k-1}$, and $(k - 1)!\cdot k = k!$.) At $t = 0$,
$$\kappa_k = \frac{\alpha\,(k - 1)!}{\lambda^k}: \qquad \kappa_1 = \frac{\alpha}{\lambda}, \quad \kappa_2 = \frac{\alpha}{\lambda^2}, \quad \kappa_3 = \frac{2\alpha}{\lambda^3}, \quad \kappa_4 = \frac{6\alpha}{\lambda^4}.$$
(b) The skewness is $\kappa_3/\kappa_2^{3/2} = \frac{2\alpha/\lambda^3}{\alpha^{3/2}/\lambda^3} = 2/\sqrt{\alpha}$, and the excess kurtosis is $\kappa_4/\kappa_2^2 = \frac{6\alpha/\lambda^4}{\alpha^2/\lambda^4} = 6/\alpha$. Neither depends on the rate, which only sets the scale. The exponential distribution ($\alpha = 1$) has skewness 2 and excess kurtosis 6, a kurtosis of $\kappa = 9$. As $\alpha$ grows both the skewness and the excess kurtosis tend to 0, the normal values.
(c) The sum of 30 independent exponential variables with rate 1 is gamma with shape 30 and rate 1, by the closure of the gamma family. The mean is the sum divided by 30, with moment generating function $M_{S}(t/30) = (1 - t/30)^{-30}$: gamma with shape 30 and rate 30. Its cumulants are $\kappa_1 = 1$, $\kappa_2 = 30/30^2 = 1/30 = 0.0333$, skewness $2/\sqrt{30} = 0.3651$ and excess kurtosis $6/30 = 0.2$. The same numbers follow from the corollary on means, with the exponential's skewness $\gamma = 2$ and kurtosis $\kappa = 9$: $\gamma/\sqrt{n} = 2/\sqrt{30}$ and $(\kappa - 3)/n = 6/30$. The script confirms the four cumulants of this gamma distribution by numerical integration of its density.
Problem 2 (an exact tail for a mean of waiting times). Five independent waiting times are exponential with mean 1. Find the exact probability that their mean exceeds 1.5, and compare it with the normal approximation and with the Chernoff bound.
Solution. The mean exceeds 1.5 exactly when the sum $S$ exceeds 7.5. By the closure of the gamma family $S$ is gamma with shape 5 and rate 1, with density $x^4e^{-x}/4!$. For a whole-number shape $n$ the tail has a closed form:
$$P(S > x) = \int_x^\infty\frac{u^{n-1}e^{-u}}{(n - 1)!}\,du = e^{-x}\sum_{k=0}^{n-1}\frac{x^k}{k!}.$$
To verify, differentiate the right side with respect to $x$: the product rule gives $-e^{-x}\sum_{k=0}^{n-1}x^k/k! + e^{-x}\sum_{k=1}^{n-1}x^{k-1}/(k - 1)!$, and the second sum cancels all of the first except its last term, leaving $-e^{-x}x^{n-1}/(n - 1)!$, which is minus the density. Both sides tend to 0 as $x \to \infty$, so they are equal. With $n = 5$ and $x = 7.5$,
$$P(S > 7.5) = e^{-7.5}\Big(1 + 7.5 + \frac{7.5^2}{2} + \frac{7.5^3}{6} + \frac{7.5^4}{24}\Big) = 0.000553 \times 238.77 = 0.1321.$$
Normal approximation. The mean of five has mean 1 and standard deviation $1/\sqrt{5} = 0.4472$, so the threshold is $z = 0.5/0.4472 = 1.118$ standard deviations up, and $1 - \Phi(1.118) = 0.1318$. The two agree to three decimals here. By Problem 1 the mean of five still has skewness $2/\sqrt{5} = 0.894$, so the agreement at this one threshold does not show that the distribution is close to normal: at the threshold 2, the exact tail $P(S > 10) = 0.0293$ is more than twice the normal approximation, 0.0127.
Chernoff bound. By Probability Inequalities, $P(S \ge 7.5) \le e^{-7.5t}M_S(t) = e^{-7.5t}(1 - t)^{-5}$ for $0 < t < 1$. The logarithm $-7.5t - 5\ln(1 - t)$ has derivative $-7.5 + 5/(1 - t)$, zero at $t = 1/3$, where the bound is $e^{-2.5}(2/3)^{-5} = 0.0821 \times 7.594 = 0.6233$. The bound is valid and weak: it is tuned for large deviations, and 7.5 is only 1.118 standard deviations above the mean.
Problem 3 (a combination of independent normal variables). $X$ is normal with mean 10 and standard deviation 3, $Y$ is normal with mean 4 and standard deviation 4, and they are independent. Find the distribution of $W = 2X - Y + 1$ and $P(W > 25)$.
Solution. By the rule for linear changes, $M_{2X}(t) = M_X(2t) = \exp(20t + 9 \cdot 4t^2/2) = \exp(20t + 36t^2/2)$ and $M_{-Y + 1}(t) = e^{t}M_Y(-t) = \exp(t - 4t + 16t^2/2) = \exp(-3t + 16t^2/2)$. The variables $2X$ and $-Y + 1$ are functions of the independent variables $X$ and $Y$, so they are independent, and by the product rule
$$M_W(t) = \exp\big(20t + 36t^2/2\big)\exp\big(-3t + 16t^2/2\big) = \exp\big(17t + 52t^2/2\big).$$
This is the moment generating function of the normal distribution with mean 17 and variance 52, finite for all $t$. By the uniqueness theorem $W$ has that distribution, with standard deviation $\sqrt{52} = 7.211$. Then $P(W > 25) = 1 - \Phi\big((25 - 17)/7.211\big) = 1 - \Phi(1.109) = 0.1336$.
The mean 17 and variance 52 follow from the rules for expected values and variances without any generating function. What the generating function adds is the shape: that $W$ is exactly normal. In the site's notation, with the standard deviation as the second argument, $W$ is $N(17, 7.211)$.
Problem 4 (the sum of two uniform variables, by characteristic functions). $U_1$ and $U_2$ are independent and uniform on $(-\tfrac12, \tfrac12)$. Show that $U_1 + U_2$ has the triangular density $f(x) = 1 - \lvert x\rvert$ on $(-1, 1)$.
Solution. From the table with $c = \tfrac12$, each $U_j$ has $\varphi(t) = \sin(t/2)/(t/2)$ for $t \ne 0$, and by property 5
$$\varphi_{U_1 + U_2}(t) = \Big(\frac{\sin(t/2)}{t/2}\Big)^2 = \frac{4\sin^2(t/2)}{t^2} = \frac{2(1 - \cos t)}{t^2},$$
using $2\sin^2(t/2) = 1 - \cos t$. Now compute the characteristic function of the triangular density directly. The density is even, so the sine part vanishes, and for $t \ne 0$
$$\int_{-1}^{1}(1 - \lvert x\rvert)\cos(tx)\,dx = 2\int_0^1(1 - x)\cos(tx)\,dx = 2\Big[(1 - x)\frac{\sin(tx)}{t}\Big]_0^1 + \frac{2}{t}\int_0^1\sin(tx)\,dx = \frac{2(1 - \cos t)}{t^2},$$
by integration by parts; the bracket is 0 at both ends. At $t = 0$ both functions equal 1. The two characteristic functions agree at every $t$, so by the uniqueness corollary $U_1 + U_2$ has the triangular density. Durrett (2019, Example 3.3.7) runs the argument in the other direction to find the triangular characteristic function. The script checks the identity numerically at $t = 1$, where both sides are 0.9194, and compares the triangular distribution function with a direct computation of $P(U_1 + U_2 \le 0.5) = 0.875$ from the area of the unit square below the line.
Problem 5 (averages of Cauchy variables). $X_1, \ldots, X_n$ are independent with the Cauchy density $1/(\pi(1 + x^2))$. (a) Find the distribution of their mean $\bar{X}_n$. (b) Find $P(\lvert\bar{X}_n\rvert > 1)$ and $P(\lvert\bar{X}_{100}\rvert > 10)$. (c) What would the second probability be for standard normal variables?
Solution. (a) Each $X_j$ has $\varphi(t) = e^{-\lvert t\rvert}$. By property 5 the sum has $e^{-n\lvert t\rvert}$, and by property 4 with $a = 1/n$ the mean has
$$\varphi_{\bar{X}_n}(t) = \Big(e^{-\lvert t/n\rvert}\Big)^n = e^{-\lvert t\rvert}.$$
That is the characteristic function of one Cauchy variable, so by the uniqueness corollary $\bar{X}_n$ has the same Cauchy distribution as a single observation, for every $n$. Averaging does nothing.
(b) The Cauchy distribution function is $F(x) = \tfrac12 + \frac{1}{\pi}\arctan x$, the integral of the density. So $P(\lvert\bar{X}_n\rvert > c) = 1 - \frac{2}{\pi}\arctan c$ for every $n$. With $c = 1$, $\arctan 1 = \pi/4$ and the probability is exactly $\tfrac12$: however many observations are averaged, the mean is more than 1 away from the center of symmetry half the time. With $c = 10$ the probability is $1 - \frac{2}{\pi}\arctan 10 = 0.0635$, for $n = 100$ as for $n = 1$.
(c) The mean of 100 independent standard normal variables is normal with mean 0 and standard deviation $1/\sqrt{100} = 0.1$, by the closure of the normal family and the rule for linear changes. So $\lvert\bar{X}_{100}\rvert > 10$ is an excursion of 100 standard deviations, with probability $2(1 - \Phi(100))$, which is below $10^{-2000}$. The script evaluates its logarithm: the probability is about $10^{-2173.6}$.
The contrast is the subject of The Laws of Large Numbers: the mean of independent observations settles at the population mean when there is one, and the Cauchy distribution has none.
Summary
| Result | Statement | Needs |
|---|---|---|
| Moment series | $M_X(t) = \sum_k E[X^k]t^k/k!$, $M_X^{(k)}(0) = E[X^k]$ | $M_X$ finite near 0 |
| Cumulants | $\kappa_1 = \mu$, $\kappa_2 = \sigma^2$, $\kappa_3 = m_3$, $\kappa_4 = m_4 - 3\sigma^4$ | $M_X$ finite near 0 |
| Linear change | $M_{aX+b}(t) = e^{bt}M_X(at)$; $\varphi_{aX+b}(t) = e^{ibt}\varphi_X(at)$ | |
| Independent sum | $M_{X+Y} = M_XM_Y$; $\varphi_{X+Y} = \varphi_X\varphi_Y$; cumulants add | independence |
| Mean of $n$ | skewness $\gamma/\sqrt{n}$, excess kurtosis $(\kappa - 3)/n$, with $\kappa$ the kurtosis | independent, same distribution |
| Uniqueness | $M_X = M_Y$ near 0, or $\varphi_X = \varphi_Y$ everywhere, implies the same distribution | cited |
| Expansion | $\varphi_X(t) = 1 + itE[X] - t^2E[X^2]/2 + r_2(t)$, $r_2(t)/t^2 \to 0$ | $E[X^2]$ finite |
| Continuity | $\varphi_n \to g$ pointwise, $g$ continuous at 0, implies $X_n \xrightarrow{d} X$ with $\varphi_X = g$ | cited |
What we have learned
A moment generating function, where it is finite near 0, is a power series whose coefficients are the moments, and its logarithm has the cumulants as derivatives. Adding independent variables multiplies the functions and adds the cumulants, which gives the skewness $\gamma/\sqrt{n}$ and excess kurtosis $(\kappa - 3)/n$ of a mean in one line, and shows that the normal, Poisson, binomial and gamma families are closed under addition once uniqueness is granted. Uniqueness is proved here for finite ranges and cited in general, and it concerns the function, not the moments: Heyde's densities share every moment with the lognormal. Characteristic functions exist for every distribution, obey the same rules, determine the distribution, and expand to second order under a finite variance alone. The Cauchy distribution's $e^{-\lvert t\rvert}$ shows that the mean of Cauchy observations is as dispersed as one observation.
Two of these results are the tools of the unit's limit theorems. The next article, The Laws of Large Numbers, asks when and in what sense a sample mean settles at the population mean. After it, Convergence in Distribution and the Proof of the Central Limit Theorem combines the second-order expansion with the continuity theorem.
Every number in this article is computed by tools/experiments/statistics/moment-generating-and-characteristic-functions.py in the site's repository, which downloads the NHANES files from the CDC and checks their checksums. The figure is drawn by tools/figures/moment-generating-and-characteristic-functions.py.
References
- Curtiss, J. H. (1942). A note on the theory of moment generating functions. The Annals of Mathematical Statistics, 13(4), 430–433. https://doi.org/10.1214/aoms/1177731541 Theorem 1 (uniqueness, p. 431) and Theorem 3 (the continuity theorem for moment generating functions, p. 432). record · 2 pages
- Durrett, R. (2019). Probability: Theory and Examples (5th ed.). Cambridge University Press. §3.3: Theorems 3.3.1 and 3.3.2, Examples 3.3.5, 3.3.7 and 3.3.16, Theorems 3.3.11 and 3.3.14 (inversion), Theorem 3.3.17 (continuity theorem), Theorem 3.3.18, Lemma 3.3.19, Theorem 3.3.20, Example 3.3.24, and §3.3.5 (the moment problem). Theorem numbers are those of the author's version of January 11, 2019. record · 4 pages
- Grinstead, C. M., and Snell, J. L. (1997). Introduction to Probability (2nd rev. ed.). American Mathematical Society. §10.1, Theorem 10.1 (a moment generating function determines a distribution with finite range). record · 20 pages
- Heyde, C. C. (1963). On a property of the lognormal distribution. Journal of the Royal Statistical Society, Series B, 25(2), 392–393. https://doi.org/10.1111/j.2517-6161.1963.tb00521.x record
- National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018: Demographic Variables and Sample Weights (DEMO_J) and Blood Pressure (BPX_J) data files. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 67 pages