Of the 5,107 adults examined in the National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018 who make up the running population of this unit, 777 have been told by a doctor that they have diabetes (the other 4,330 include those told only of borderline diabetes; the unit's topic page defines the population). Each adult contributes a 0 or a 1. A week of deaths in Iowa contributes a count, and a body mass index contributes a positive number with a long right tail. The binomial, Poisson and gamma distributions that describe these three kinds of data look unrelated, and each was worked out separately: its mean, its variance, its maximum likelihood estimate. Yet a single regression method, the generalized linear model, fits all three, with one algorithm and one theory of inference. What do the three distributions share that makes this possible, and which of their properties follow from that shared structure alone?
They share a form. A family of distributions is an exponential family when its mass function or density can be written as
$$f(x; \boldsymbol\theta) = h(x)\exp\big\{\boldsymbol\theta^{\mathsf T}\mathbf{T}(x) - A(\boldsymbol\theta)\big\}, \qquad (1)$$
with a vector of statistics $\mathbf{T}(x)$, a vector of parameters $\boldsymbol\theta$ that multiplies it, a nonnegative function $h(x)$ that does not involve the parameter, and a number $A(\boldsymbol\theta)$ that makes the total probability 1. This article proves the properties that the form implies, and it is the home of those properties for the Generalized Linear Models unit: later articles of the unit cite them from here.
First we write the standard families in the form (1) and name its parts. Then we prove that the function $A$ is convex and that its derivatives are the mean, variance and higher cumulants of $\mathbf{T}$. Next we show that the mean determines the member, that maximum likelihood estimation reduces to matching the mean of $\mathbf{T}$ to its sample average, and that in one dimension the relation between variance and mean determines the whole family. Then we compute the Kullback–Leibler divergence between two members. Finally we specialize to the exponential dispersion form on which generalized linear models are built. The article assumes The Likelihood Function, Maximum Likelihood Estimation and Fisher Information and the Cramér–Rao Bound, and uses the moment generating function results of Moment Generating and Characteristic Functions without proving them again.
The form and its parts
The family of distributions in (1) is indexed by $\boldsymbol\theta = (\theta_1, \ldots, \theta_k)$, which is called the natural parameter. The vector $\mathbf{T}(x) = (T_1(x), \ldots, T_k(x))$ is the sufficient statistic, $h$ is the base function, and $A$ is the log-partition function (also called the cumulant function, for a reason the next section proves). The name comes from the normalizing sum or integral,
$$e^{A(\boldsymbol\theta)} = \sum_x h(x)\,e^{\boldsymbol\theta^{\mathsf T}\mathbf{T}(x)} \quad\text{or}\quad e^{A(\boldsymbol\theta)} = \int h(x)\,e^{\boldsymbol\theta^{\mathsf T}\mathbf{T}(x)}\,dx, \qquad (2)$$
which is the "partition function" of statistical physics. The natural parameter space $\Theta$ is the set of all $\boldsymbol\theta$ for which the sum or integral in (2) is finite; every such $\boldsymbol\theta$ gives a distribution. Efron and Hastie (2016, Section 5.5) describe the construction as exponential tilting: start from one distribution, multiply it by $e^{\boldsymbol\theta^{\mathsf T}\mathbf{T}(x)}$, and renormalize.
Two other pages of this collection use the same form in other letters. Sufficient Statistics writes a one-parameter family as $h(x)\exp\{\eta(\theta)T(x) - A(\theta)\}$, with a general parameter $\theta$ and its function $\eta(\theta)$; in (1) the coefficient of $\mathbf{T}$ is itself the parameter, which is what "natural" means. Maximum Entropy Distributions writes the multipliers as $\lambda$ and the normalizer as $Z$, so that $A = \ln Z$. In this unit the letter $\eta$ is kept for the linear predictor of a regression model, and $\boldsymbol\theta$ is the natural parameter.
The standard families, written in the form (1):
- Bernoulli with success probability $p$: $p^x(1 - p)^{1 - x} = \exp\{x\ln\frac{p}{1 - p} + \ln(1 - p)\}$ for $x \in \{0, 1\}$. So $T(x) = x$, $\theta = \ln\frac{p}{1 - p}$ (the log-odds), $h = 1$ and $A(\theta) = -\ln(1 - p) = \ln(1 + e^{\theta})$, because $1 - p = 1/(1 + e^{\theta})$. Every real $\theta$ is allowed: $\Theta = \mathbb{R}$.
- Binomial with $m$ trials, $m$ known: $\binom{m}{x}p^x(1 - p)^{m - x}$, with $T(x) = x$, the same $\theta$, $h(x) = \binom{m}{x}$ and $A(\theta) = m\ln(1 + e^{\theta})$.
- Poisson with mean $\lambda$: $\frac{1}{x!}\exp\{x\ln\lambda - \lambda\}$, with $T(x) = x$, $\theta = \ln\lambda$, $h(x) = 1/x!$ and $A(\theta) = e^{\theta}$.
- Normal with mean $\mu$ and variance $\sigma^2$, both unknown: $\frac{1}{\sqrt{2\pi\sigma^2}}\exp\{-\frac{(x - \mu)^2}{2\sigma^2}\}$ expands to $\exp\{\frac{\mu}{\sigma^2}x - \frac{1}{2\sigma^2}x^2 - \frac{\mu^2}{2\sigma^2} - \frac12\ln(2\pi\sigma^2)\}$, with $\mathbf{T}(x) = (x, x^2)$, $\boldsymbol\theta = (\mu/\sigma^2, -1/(2\sigma^2))$, and $A = \frac{\mu^2}{2\sigma^2} + \frac12\ln(2\pi\sigma^2)$, which in the natural parameters is $-\theta_1^2/(4\theta_2) + \frac12\ln(-\pi/\theta_2)$. (On this site the normal is written $N(\mu, \sigma)$ with the standard deviation second, as The Normal Distribution explains; here it is named in words.)
- Gamma with shape $\alpha$ and rate $\lambda$: $\frac{\lambda^\alpha}{\Gamma(\alpha)}x^{\alpha - 1}e^{-\lambda x} = \frac1x\exp\{-\lambda x + \alpha\ln x + \alpha\ln\lambda - \ln\Gamma(\alpha)\}$ for $x > 0$, with $\mathbf{T}(x) = (x, \ln x)$, $\boldsymbol\theta = (-\lambda, \alpha)$, $h(x) = 1/x$, and $A(\boldsymbol\theta) = \ln\Gamma(\theta_2) - \theta_2\ln(-\theta_1)$ on $\Theta = \{\theta_1 < 0, \theta_2 > 0\}$, as in Efron and Hastie (2016, eqs. 5.52 and 5.53).
A family can be written in the form (1) in more than one way. Adding a constant to $T$ and changing $h$, or writing $T$ in two redundant coordinates, describes the same distributions. The representation is called minimal when no combination of the statistics is constant: there is no nonzero vector $\mathbf{c}$ and number $d$ with $\mathbf{c}^{\mathsf T}\mathbf{T}(X) = d$ with probability 1. Every member of a family (1) has a mass function or density that is positive exactly where $h$ is positive, so an event has probability 0 under one member exactly when it has probability 0 under all of them, and "with probability 1" needs no qualifier. All five representations above are minimal: $x$ is not constant, and neither $x^2$ nor $\ln x$ is an affine function of $x$ on the support.
Example (the uniform distribution is not an exponential family). The uniform distribution on $(0, \theta)$, the counterexample of the Fisher information article, cannot be written as (1). Every member of a family (1) is positive exactly where $h$ is positive, a set that does not depend on the parameter, while the uniform distributions on $(0, 1)$ and $(0, 2)$ have different supports. The same argument excludes every family whose support moves with the parameter.
The log-partition function generates the moments
Here is the idea in one sentence: tilting a member of the family by $e^{\mathbf{t}^{\mathsf T}\mathbf{T}}$ lands on another member, so the moment generating function of $\mathbf{T}$ is a ratio of two normalizers, and its logarithm is a difference of two values of $A$. Before the derivatives, the domain on which $A$ is finite needs a property.
Theorem 1 (convexity). The natural parameter space $\Theta$ is a convex set, and $A$ is a convex function on it: for $\boldsymbol\theta_1, \boldsymbol\theta_2$ in $\Theta$ and $0 < \lambda < 1$,
$$A(\lambda\boldsymbol\theta_1 + (1 - \lambda)\boldsymbol\theta_2) \le \lambda A(\boldsymbol\theta_1) + (1 - \lambda)A(\boldsymbol\theta_2).$$
If the representation is minimal and $\boldsymbol\theta_1 \ne \boldsymbol\theta_2$, the inequality is strict.
The idea: the inequality between a weighted geometric and a weighted arithmetic mean, applied point by point to the two densities, and then summed or integrated.
Proof. For positive numbers $a$, $b$ and $0 < \lambda < 1$, the concavity of the logarithm gives $\ln(\lambda a + (1 - \lambda)b) \ge \lambda\ln a + (1 - \lambda)\ln b$, so
$$a^\lambda b^{1 - \lambda} \le \lambda a + (1 - \lambda)b, \qquad (3)$$
with equality only when $a = b$, because the logarithm is strictly concave (Jensen's inequality and its equality case, Probability Inequalities). Let $f_1$ and $f_2$ be the members at $\boldsymbol\theta_1$ and $\boldsymbol\theta_2$, and put $\boldsymbol\theta = \lambda\boldsymbol\theta_1 + (1 - \lambda)\boldsymbol\theta_2$. Where $h(x) > 0$,
$$f_1(x)^\lambda f_2(x)^{1 - \lambda} = h(x)\exp\big\{\boldsymbol\theta^{\mathsf T}\mathbf{T}(x) - \lambda A(\boldsymbol\theta_1) - (1 - \lambda)A(\boldsymbol\theta_2)\big\}.$$
By (3) the left side is at most $\lambda f_1(x) + (1 - \lambda)f_2(x)$. Summing or integrating over $x$, the right side totals $\lambda + (1 - \lambda) = 1$, so
$$e^{-\lambda A(\boldsymbol\theta_1) - (1 - \lambda)A(\boldsymbol\theta_2)}\sum_x h(x)e^{\boldsymbol\theta^{\mathsf T}\mathbf{T}(x)} \le 1$$
(an integral for densities). The sum is therefore finite, which puts $\boldsymbol\theta$ in $\Theta$, and taking logarithms gives the inequality for $A$.
For strictness, suppose equality holds. Then the nonnegative quantity $\lambda f_1 + (1 - \lambda)f_2 - f_1^\lambda f_2^{1 - \lambda}$ sums or integrates to 0. Divide it by $f_2$: the variable $\lambda r(X) + (1 - \lambda) - r(X)^\lambda$, with $r = f_1/f_2$, is nonnegative and has mean 0 under $f_2$, so it is 0 with probability 1 (a nonnegative variable with mean 0 is 0 with probability 1, as Probability Inequalities proves). By the equality case of (3), $r(X) = 1$ with probability 1, that is, $(\boldsymbol\theta_1 - \boldsymbol\theta_2)^{\mathsf T}\mathbf{T}(X) = A(\boldsymbol\theta_1) - A(\boldsymbol\theta_2)$ with probability 1. That is a constant combination of the statistics with a nonzero coefficient vector, which a minimal representation excludes. $\square$
A point of $\Theta$ is interior if a small ball around it lies in $\Theta$. For the Bernoulli, binomial and Poisson families every point is interior, since $\Theta = \mathbb{R}$; for the gamma family as well, since its $\Theta$ is an open quadrant. The interior of a convex set is convex.
Theorem 2 (the log-partition function generates the cumulants). Let $\boldsymbol\theta$ be an interior point of $\Theta$ and $X$ have the distribution $f(\cdot\,; \boldsymbol\theta)$. For every fixed vector $\mathbf{a}$, the moment generating function of the single variable $\mathbf{a}^{\mathsf T}\mathbf{T}(X)$ is finite for $s$ near 0 and equals
$$E_{\boldsymbol\theta}\big[e^{s\,\mathbf{a}^{\mathsf T}\mathbf{T}(X)}\big] = \exp\{A(\boldsymbol\theta + s\mathbf{a}) - A(\boldsymbol\theta)\}.$$
Consequently the function $s \mapsto A(\boldsymbol\theta + s\mathbf{a})$ has derivatives of every order near $s = 0$, and its derivatives at 0 are the cumulants of $\mathbf{a}^{\mathsf T}\mathbf{T}(X)$. In particular:
- the partial derivatives are the means, $\partial A/\partial\theta_j = E_{\boldsymbol\theta}[T_j]$ (take $\mathbf{a}$ the $j$th unit vector); write $\boldsymbol\mu(\boldsymbol\theta) = E_{\boldsymbol\theta}[\mathbf{T}]$ for the vector of them;
- the second derivative along $\mathbf{a}$ is the variance, $\frac{d^2}{ds^2}A(\boldsymbol\theta + s\mathbf{a})\big|_{s = 0} = \operatorname{Var}_{\boldsymbol\theta}(\mathbf{a}^{\mathsf T}\mathbf{T}) = \mathbf{a}^{\mathsf T}\operatorname{Cov}_{\boldsymbol\theta}(\mathbf{T})\,\mathbf{a}$;
- in one dimension, $A'(\theta) = E_\theta[T]$, $A''(\theta) = \operatorname{Var}_\theta(T)$, and $A'''(\theta) = E_\theta[(T - E_\theta T)^3]$.
Proof. Since $\boldsymbol\theta$ is interior, $\boldsymbol\theta + s\mathbf{a}$ lies in $\Theta$ for all $s$ in some interval $(-\delta, \delta)$. For such $s$, by (2),
$$E_{\boldsymbol\theta}\big[e^{s\,\mathbf{a}^{\mathsf T}\mathbf{T}(X)}\big] = \sum_x h(x)\,e^{(\boldsymbol\theta + s\mathbf{a})^{\mathsf T}\mathbf{T}(x) - A(\boldsymbol\theta)} = e^{A(\boldsymbol\theta + s\mathbf{a})}e^{-A(\boldsymbol\theta)},$$
with an integral in place of the sum for densities. So the cumulant generating function of $\mathbf{a}^{\mathsf T}\mathbf{T}(X)$ is $K(s) = A(\boldsymbol\theta + s\mathbf{a}) - A(\boldsymbol\theta)$, finite on $(-\delta, \delta)$. The theorem on the moment series in Moment Generating and Characteristic Functions shows that a moment generating function finite near 0 has derivatives of every order there, so $K$ does too. Its derivatives at 0 are by definition the cumulants, and the proposition on the first four cumulants in the same article identifies the first three as the mean, the variance and the third central moment. The variance of $\mathbf{a}^{\mathsf T}\mathbf{T}$ is $\mathbf{a}^{\mathsf T}\operatorname{Cov}(\mathbf{T})\mathbf{a}$ by the rule for the variance of a linear combination (Expectation, Variance and Covariance). $\square$
In fact $A$ is infinitely differentiable at interior points, with gradient $\boldsymbol\mu(\boldsymbol\theta)$ and matrix of second derivatives $\operatorname{Cov}_{\boldsymbol\theta}(\mathbf{T})$, which is how Efron and Hastie (2016, eqs. 5.56 and 5.57) state the result. The theorem above proves the directional form, which is all that the arguments below use: each multidimensional question is reduced to a function of one variable along a line.
Example (the Poisson and Bernoulli families). For the Poisson, $A(\theta) = e^{\theta}$: every derivative is $e^{\theta} = \lambda$, so every cumulant of a Poisson variable equals its mean, as the cumulant section of the moment generating function article found from $K(t) = \lambda(e^t - 1)$. For the Bernoulli, $A(\theta) = \ln(1 + e^{\theta})$ gives $A'(\theta) = e^{\theta}/(1 + e^{\theta}) = p$ and $A''(\theta) = p(1 - p)$, the familiar mean and variance. For the 777 diabetic adults among 5,107, the share is $\hat p = 0.1521$, the natural parameter is $\hat\theta = \ln(777/4{,}330) = -1.7179$, $A(\hat\theta) = \ln(5{,}107/4{,}330) = 0.1650$ and $A''(\hat\theta) = \hat p(1 - \hat p) = 0.1290$.
The score and the information need no regularity conditions
The score of one observation in the natural parameter is
$$\frac{\partial}{\partial\theta_j}\ln f(x; \boldsymbol\theta) = T_j(x) - \frac{\partial A}{\partial\theta_j}(\boldsymbol\theta) = T_j(x) - \mu_j(\boldsymbol\theta).$$
By Theorem 2 it has mean zero, and its variance along a direction $\mathbf{a}$ is $\operatorname{Var}(\mathbf{a}^{\mathsf T}\mathbf{T})$: the Fisher information matrix in the natural parameter is $\operatorname{Cov}_{\boldsymbol\theta}(\mathbf{T})$. The Fisher information article proves the two properties of the score under conditions R1 to R3, which allow sums and integrals to be differentiated under the sign. At interior points of an exponential family nothing of that kind is needed: Theorem 2 delivered both facts from the moment generating function.
A second fact will matter for the fitting algorithm of this unit. The second derivative of $\ln f(x;\boldsymbol\theta)$ along a direction $\mathbf{a}$ is $-\frac{d^2}{ds^2}A(\boldsymbol\theta + s\mathbf{a})$, which does not involve $x$. So in the natural parameter the observed information equals the expected information, for every sample. Iteratively Reweighted Least Squares uses this to show that two standard iterations, Newton's method and Fisher scoring, coincide for the canonical link.
The mean determines the member
A member of the family is labeled by its natural parameter, but the quantity a statistician thinks about is the mean. Can the members be labeled by their means instead? Here is the idea: along any line in $\Theta$, the slope of $A$ is a mean of $\mathbf{T}$ and its curvature is a variance, so the mean can only increase in the direction of travel.
Theorem 3 (the mean parametrization). Suppose the representation is minimal, and let $\boldsymbol\theta_1 \ne \boldsymbol\theta_2$ be interior points of $\Theta$. Then
$$(\boldsymbol\theta_1 - \boldsymbol\theta_2)^{\mathsf T}\big(\boldsymbol\mu(\boldsymbol\theta_1) - \boldsymbol\mu(\boldsymbol\theta_2)\big) > 0.$$
In particular $\boldsymbol\mu(\boldsymbol\theta_1) \ne \boldsymbol\mu(\boldsymbol\theta_2)$: the map from the natural parameter to the mean of $\mathbf{T}$ is one to one on the interior. In one dimension the mean $\mu(\theta) = A'(\theta)$ is strictly increasing, with derivative $d\mu/d\theta = \operatorname{Var}_\theta(T) > 0$, and its inverse $\theta(\mu)$ has derivative $d\theta/d\mu = 1/\operatorname{Var}_\theta(T)$.
Proof. Put $\mathbf{a} = \boldsymbol\theta_1 - \boldsymbol\theta_2$ and $g(s) = A(\boldsymbol\theta_2 + s\mathbf{a})$ for $s$ in an open interval containing $[0, 1]$, on which the points stay interior (the interior is convex and open). By Theorem 2 applied at each point $\boldsymbol\theta_2 + s\mathbf{a}$, $g'(s) = \mathbf{a}^{\mathsf T}\boldsymbol\mu(\boldsymbol\theta_2 + s\mathbf{a})$ and $g''(s) = \operatorname{Var}(\mathbf{a}^{\mathsf T}\mathbf{T})$ under that member. Minimality says that $\mathbf{a}^{\mathsf T}\mathbf{T}$ is not constant with probability 1, so the variance is positive and $g'$ is strictly increasing. Hence $g'(1) > g'(0)$, which is $\mathbf{a}^{\mathsf T}\boldsymbol\mu(\boldsymbol\theta_1) > \mathbf{a}^{\mathsf T}\boldsymbol\mu(\boldsymbol\theta_2)$, the claim. In one dimension, $\mu' = A'' = \operatorname{Var}_\theta(T) > 0$, and the derivative of the inverse of a strictly increasing differentiable function with positive derivative is the reciprocal. $\square$
Efron and Hastie (2016, eqs. 5.58 to 5.60) call $\boldsymbol\mu = E\{\mathbf{T}\}$ the expectation parameter and note the same one-to-one relation, with the matrix of derivatives $d\boldsymbol\mu/d\boldsymbol\theta$ equal to the covariance matrix of $\mathbf{T}$.
The variance function
In one dimension, with $T(x) = x$, the variance of a member is a function of its mean: $\operatorname{Var} = A''(\theta(\mu))$. This function is the variance function $V(\mu)$ of the family. For the Poisson it is $V(\mu) = \mu$, for the Bernoulli $V(\mu) = \mu(1 - \mu)$. A one-parameter family of the form $h(x)e^{\theta x - A(\theta)}$, with $T(x) = x$, is called a natural exponential family. Its natural parameter space $\Theta$ is an interval (Theorem 1), which may or may not contain its end points; the theorems below use only the members with $\theta$ in an open interval, for example the interior of $\Theta$. For such families the variance function carries everything.
Theorem 4 (the variance function determines the family). Let two natural exponential families, each restricted to an open interval of natural parameters, have the same set $\Omega$ of means and the same variance function $V$ on it. Then for each $\mu$ in $\Omega$ their members with mean $\mu$ are the same distribution.
The idea: the variance function fixes how fast the natural parameter moves as the mean moves, and that in turn fixes the cumulant generating function of every member.
Proof. Let the families have log-partition functions $A_1$ and $A_2$ and mean maps $\mu_i(\theta) = A_i'(\theta)$, with inverses $\theta_i(\mu)$ on $\Omega$. By Theorem 3, $d\theta_i/d\mu = 1/A_i''(\theta_i(\mu)) = 1/V(\mu)$ for both $i$. Fix $\mu_0$ in $\Omega$ and put $\tau(\mu) = \int_{\mu_0}^{\mu}dm/V(m)$; then $\theta_i(\mu) = \theta_i(\mu_0) + \tau(\mu)$, and $\tau$ is strictly increasing. Now fix a mean $\mu$ and let $X_i$ be the member of family $i$ with that mean. By Theorem 2 its cumulant generating function, for $t$ near 0, is
$$K_i(t) = A_i(\theta_i(\mu) + t) - A_i(\theta_i(\mu)) = \int_0^t A_i'(\theta_i(\mu) + s)\,ds.$$
The integrand is the mean of the member of family $i$ whose natural parameter is $\theta_i(\mu) + s$, that is, the number $\nu$ with $\theta_i(\nu) = \theta_i(\mu) + s$, or equivalently $\tau(\nu) = \tau(\mu) + s$. This condition is the same for $i = 1$ and $i = 2$ (and so is the condition that $\theta_i(\mu) + s$ lies in $\Theta_i$, since $\Theta_i = \theta_i(\mu_0) + \tau(\Omega)$), so the integrands agree for every $s$, and $K_1 = K_2$ on an interval around 0. The moment generating functions of $X_1$ and $X_2$ are then finite and equal near 0, and by the uniqueness theorem stated and cited in Moment Generating and Characteristic Functions, $X_1$ and $X_2$ have the same distribution. $\square$
So a natural exponential family with $V(\mu) = \mu$ on $(0, \infty)$ is the Poisson family, and one with $V(\mu) = \mu^2$ on $(0, \infty)$ is the family of exponential distributions (Worked problem 5). Morris (1982) studied the natural exponential families whose variance function is at most quadratic in the mean; his abstract lists the normal, Poisson, gamma, binomial and negative binomial families and states: "Only one other such family exists."
This theorem is why generalized linear models describe a response by its mean and its variance function. Specifying $V$ specifies the distribution (given a scale parameter, below); and Overdispersion and Quasi-Likelihood shows how far one can get with the mean and variance alone, when no distribution with the observed variance function is assumed.
Maximum likelihood is moment matching
Let $X_1, \ldots, X_n$ be independent with common distribution $f(\cdot\,; \boldsymbol\theta)$ from a family (1). The log-likelihood is
$$\ell(\boldsymbol\theta) = \boldsymbol\theta^{\mathsf T}\sum_{i=1}^n\mathbf{T}(x_i) - nA(\boldsymbol\theta) + \sum_{i=1}^n\ln h(x_i) = n\big[\boldsymbol\theta^{\mathsf T}\bar{\mathbf{t}} - A(\boldsymbol\theta)\big] + \text{const},$$
where $\bar{\mathbf{t}} = \frac1n\sum_i\mathbf{T}(x_i)$ is the sample average of the sufficient statistic. The data enter only through $\bar{\mathbf{t}}$, which is the factorization criterion of Sufficient Statistics at work: $\bar{\mathbf{t}}$ is sufficient, and its dimension $k$ does not grow with $n$. Efron and Hastie (2016, p. 66) put the point this way: however large $n$ is, "the statistician can still compress all the inferential information into a p-dimensional statistic", and they add: "Only exponential families enjoy this property." The converse needs conditions the quotation leaves out, among them a support that does not depend on the parameter: the uniform family on $(0, \theta)$ of the example above has the one-dimensional sufficient statistic $\max_i x_i$ (Sufficient Statistics). It is not proved here.
Theorem 5 (maximum likelihood in an exponential family). Suppose the representation is minimal.
- $\ell$ is strictly concave on $\Theta$.
- An interior point $\hat{\boldsymbol\theta}$ maximizes $\ell$ if and only if $\boldsymbol\mu(\hat{\boldsymbol\theta}) = \bar{\mathbf{t}}$: the fitted mean of the sufficient statistic equals its sample average.
- There is at most one maximizer.
- If $\bar{\mathbf{t}}$ is not the mean of any member with interior parameter, there is no interior maximizer.
- When an interior maximizer exists, the maximum likelihood estimate of the mean $\boldsymbol\mu$ of $\mathbf{T}$ is $\bar{\mathbf{t}}$. As an estimator, $\bar{\mathbf{t}}$ is unbiased with covariance matrix $\operatorname{Cov}(\mathbf{T})/n$. In one dimension its variance $V(\mu)/n$ equals the value of the Cramér–Rao bound for unbiased estimators of $\mu$; when $X$ takes finitely many values, as for the Bernoulli and binomial families, the bound applies to every unbiased estimator, so $\bar t$ has the least variance among them.
Proof. (1) $-nA$ is strictly concave by Theorem 1, and adding the linear function $n\boldsymbol\theta^{\mathsf T}\bar{\mathbf{t}}$ preserves strict concavity.
(2) If $\hat{\boldsymbol\theta}$ is an interior maximizer, each partial derivative of $\ell$ vanishes there, and by Theorem 2 the $j$th one is $n[\bar t_j - \mu_j(\hat{\boldsymbol\theta})]$. Conversely, suppose $\boldsymbol\mu(\hat{\boldsymbol\theta}) = \bar{\mathbf{t}}$. Take any other $\boldsymbol\theta$ in $\Theta$ and put $k(s) = \ell(\hat{\boldsymbol\theta} + s(\boldsymbol\theta - \hat{\boldsymbol\theta}))$ for $0 \le s \le 1$, which is concave by (1). Its derivative at 0 is $n(\boldsymbol\theta - \hat{\boldsymbol\theta})^{\mathsf T}[\bar{\mathbf{t}} - \boldsymbol\mu(\hat{\boldsymbol\theta})] = 0$ (Theorem 2 along that direction). A concave function lies below its tangent line, so $k(1) \le k(0)$, that is, $\ell(\boldsymbol\theta) \le \ell(\hat{\boldsymbol\theta})$.
(3) If $\boldsymbol\theta_1 \ne \boldsymbol\theta_2$ both maximized $\ell$, strict concavity would give their midpoint a larger value.
(4) By (2), an interior maximizer has $\boldsymbol\mu(\hat{\boldsymbol\theta}) = \bar{\mathbf{t}}$.
(5) If $\hat{\boldsymbol\theta}$ is an interior maximizer then, since the mean map is one to one (Theorem 3), by the invariance of maximum likelihood estimates (Maximum Likelihood Estimation) the estimate of $\boldsymbol\mu$ is $\boldsymbol\mu(\hat{\boldsymbol\theta}) = \bar{\mathbf{t}}$. A sample average of independent copies of $\mathbf{T}$ has mean $\boldsymbol\mu$ and covariance $\operatorname{Cov}(\mathbf{T})/n$. In one dimension, the information about $\theta$ in one observation is $I(\theta) = V(\mu)$, and the information about the mean is $I(\theta)(d\theta/d\mu)^2 = V(\mu)\cdot V(\mu)^{-2} = 1/V(\mu)$, by the reparametrization rule of the Fisher information article and Theorem 3. The bound for an unbiased estimator of $\mu$ is therefore $V(\mu)/n$, which is the variance of $\bar t$. The Fisher information article proves the bound under its conditions R1 to R3 for $f$, for estimators $W$ whose mean may also be differentiated under the sum or integral. When $X$ takes finitely many values, as for the Bernoulli and binomial families, R1 to R3 hold (the support does not depend on $\theta$, $f$ is smooth in $\theta$, and R3 is automatic for a finite sum), and $E_\theta[W]$ is a finite sum of terms $W(\mathbf{x})\prod_i f(x_i;\theta)$, each differentiable in $\theta$, so the bound applies to every unbiased $W$. For other families this page does not verify the last condition, and the statement is only the equality of the two numbers. $\square$
Part 5 explains the Fisher information article's remark that the bound is attained only by linear functions of a single statistic in an exponential family: here is the family, and here is the statistic. For the diabetes indicator the estimate is $\hat p = 0.1521$, with variance $p(1 - p)/n$, so its standard deviation is about 0.00503 at $p = \hat p$. The natural parameter is not a mean, and its estimate $\hat\theta = -1.7179$ has no exact variance formula; its large-sample standard error is $1/\sqrt{n\hat p(1 - \hat p)} = 0.0390$ (Asymptotics of the Maximum Likelihood Estimator).
Example (a sample with no estimate). If every one of $n$ Bernoulli observations is 0, then $\bar t = 0$. Every member has mean $e^{\theta}/(1 + e^{\theta})$, strictly between 0 and 1, so by part 4 there is no maximum likelihood estimate of $\theta$. The likelihood $(1 + e^{\theta})^{-n}$ increases toward its supremum 1 as $\theta \to -\infty$, without reaching it. A sample of 40 adults aged 18 to 39 who are not obese, a group in which 12 of 1,040 have diabetes, has a 0.6286 chance of containing no diabetic at all (Worked problem 1). The same thing happens in a regression whenever a combination of the inputs separates the 0s from the 1s, which Deviance and Inference for Generalized Linear Models treats as the failure case of the fitting theory.
For the families of the previous section, the set of possible means can be read from the formulas: $(0, 1)$ for the Bernoulli, $(0, \infty)$ for the Poisson, all real numbers for the normal mean. In each case it is the open interval strictly between the smallest and largest values that $T$ can take, so for these families a maximum likelihood estimate exists exactly when $\bar t$ is not at an end. This page does not state or prove a general version for other families.
Two equations, one Newton iteration: a gamma model for body mass index
The body mass index of the 5,107 adults is positive and skewed to the right, with mean $\bar x = 29.65$ and sample variance 54.55. Consider the gamma family with $\mathbf{T}(x) = (x, \ln x)$ and natural parameter $\boldsymbol\theta = (-\lambda, \alpha)$. By Theorem 5 the maximum likelihood estimate solves two moment equations, one for each coordinate of $\mathbf{T}$:
$$E[X] = \frac{\alpha}{\lambda} = \bar x = 29.65, \qquad E[\ln X] = \psi(\alpha) - \ln\lambda = \overline{\ln x} = 3.3611,$$
where $\psi = \Gamma'/\Gamma$ is the digamma function and $\overline{\ln x}$ the average of the logarithms. The second mean comes from differentiating $A(\boldsymbol\theta) = \ln\Gamma(\theta_2) - \theta_2\ln(-\theta_1)$ in $\theta_2$, as Theorem 2 says it must. Eliminating $\lambda = \alpha/\bar x$ gives one equation, $\ln\alpha - \psi(\alpha) = \ln\bar x - \overline{\ln x}$, which has no closed-form solution.
Newton's method solves the two equations together in the natural parameters. Its matrix is the matrix of second derivatives of $A$,
$$\begin{pmatrix} \theta_2/\theta_1^2 & -1/\theta_1 \\ -1/\theta_1 & \psi'(\theta_2) \end{pmatrix},$$
obtained by differentiating $A$ directly, and each step is $\boldsymbol\theta \leftarrow \boldsymbol\theta + [\nabla^2A(\boldsymbol\theta)]^{-1}(\bar{\mathbf{t}} - \boldsymbol\mu(\boldsymbol\theta))$. Started from the method of moments, which matches the mean and the variance ($\alpha = \bar x^2/s^2 = 16.12$, $\lambda = \bar x/s^2 = 0.5436$), one step gives $\alpha = 17.62$, and three steps reach $\hat\alpha = 17.78$, $\hat\lambda = 0.5996$, unchanged in six significant figures thereafter. The fitted mean is exactly $\bar x$, as part 2 of Theorem 5 requires. The fitted variance is $\hat\alpha/\hat\lambda^2 = 49.45$, not the sample's 54.55: the maximum likelihood fit matches the mean of $\ln x$, not the variance, because $\ln x$ and not $x^2$ is the second sufficient statistic. Its log-likelihood exceeds that of the method-of-moments fit by 12.14.
Example (the misreading "maximum likelihood matches the mean and the variance"). It matches the averages of the sufficient statistics, which are the mean and the variance only for the normal family, where $\mathbf{T} = (x, x^2)$. For the gamma family it matches the arithmetic mean and the geometric mean $e^{\overline{\ln x}}$; for body mass index the fitted variance falls 5.1 below the sample's.
Divergence between two members
How far apart are two members of a family? The Kullback–Leibler divergence of Cross-Entropy and Kullback–Leibler Divergence answers in terms of $A$ alone.
Theorem 6 (divergence as a Bregman divergence). For interior points $\boldsymbol\theta_1$, $\boldsymbol\theta_2$ of $\Theta$,
$$D(f_{\boldsymbol\theta_1} \,\|\, f_{\boldsymbol\theta_2}) = A(\boldsymbol\theta_2) - A(\boldsymbol\theta_1) - (\boldsymbol\theta_2 - \boldsymbol\theta_1)^{\mathsf T}\boldsymbol\mu(\boldsymbol\theta_1). \qquad (4)$$
The right side is the amount by which the convex function $A$ at $\boldsymbol\theta_2$ lies above its tangent plane at $\boldsymbol\theta_1$, a quantity called a Bregman divergence. There is also a point $\boldsymbol\theta^*$ on the segment between $\boldsymbol\theta_1$ and $\boldsymbol\theta_2$ with
$$D(f_{\boldsymbol\theta_1} \,\|\, f_{\boldsymbol\theta_2}) = \tfrac12\operatorname{Var}_{\boldsymbol\theta^*}\big((\boldsymbol\theta_2 - \boldsymbol\theta_1)^{\mathsf T}\mathbf{T}\big), \qquad (5)$$
which in one dimension reads $D = \frac12 A''(\theta^*)(\theta_2 - \theta_1)^2 = \frac12 I(\theta^*)(\theta_2 - \theta_1)^2$.
Proof. Where $h > 0$, $\ln[f_{\boldsymbol\theta_1}(x)/f_{\boldsymbol\theta_2}(x)] = (\boldsymbol\theta_1 - \boldsymbol\theta_2)^{\mathsf T}\mathbf{T}(x) - A(\boldsymbol\theta_1) + A(\boldsymbol\theta_2)$, because $h$ cancels. Taking the expected value under $\boldsymbol\theta_1$ replaces $\mathbf{T}$ by $\boldsymbol\mu(\boldsymbol\theta_1)$, which is (4). For (5), let $g(s) = A(\boldsymbol\theta_1 + s(\boldsymbol\theta_2 - \boldsymbol\theta_1))$. By Taylor's theorem with the Lagrange remainder, $g(1) = g(0) + g'(0) + \frac12 g''(s^*)$ for some $s^*$ in $(0, 1)$. By Theorem 2, $g'(0) = (\boldsymbol\theta_2 - \boldsymbol\theta_1)^{\mathsf T}\boldsymbol\mu(\boldsymbol\theta_1)$ and $g''(s^*)$ is the variance in (5) at $\boldsymbol\theta^* = \boldsymbol\theta_1 + s^*(\boldsymbol\theta_2 - \boldsymbol\theta_1)$. So $D = g(1) - g(0) - g'(0) = \frac12 g''(s^*)$. In one dimension the information in the natural parameter is $A''$, as shown above. $\square$
The approximation $D \approx \frac12 I(\theta)(\theta_2 - \theta_1)^2$ appears in the Fisher information article, and the divergence article proves its limiting form for a finite alphabet. Formula (5) is an exact version for every exponential family in its natural parameter, with the information evaluated at an intermediate point.
Example (two age groups). Among adults aged 60 and over, 270 of 762 who are obese have diabetes ($p_1 = 0.3543$, $\theta_1 = -0.6001$) and 264 of 1,108 who are not ($p_2 = 0.2383$, $\theta_2 = -1.1622$). Formula (4) with the Bernoulli $A$ gives $D(f_{\theta_1} \,\|\, f_{\theta_2}) = 0.03388$ nats, the same as the direct sum $p_1\ln(p_1/p_2) + (1 - p_1)\ln[(1 - p_1)/(1 - p_2)]$. The quadratic with the information at $\theta_1$ gives 0.03615, with the information at $\theta_2$ gives 0.02868; formula (5) holds exactly at the intermediate point where $p = 0.3113$.
The deviance of a generalized linear model, which Deviance and Inference for Generalized Linear Models develops, is built from this divergence: Efron and Hastie (2016, eq. 8.31) define the deviance between two densities as twice their Kullback–Leibler divergence, and show that maximum likelihood fitting of a generalized linear model minimizes the total deviance between the data and the fitted members (their eqs. 8.33 to 8.35).
Where exponential families come from
Maximum Entropy Distributions proves that among all distributions with given averages of the statistics $T_1, \ldots, T_k$, the one closest in Kullback–Leibler divergence to the reference distribution proportional to $h$ (when $h$ has a finite sum or integral) is the member of the family (1) with those averages. Theorem 5 is the same statement seen from the data: the maximum likelihood member is the one whose averages match the sample's. So the exponential family generated by $\mathbf{T}$ is the set of models that commit to the averages of $\mathbf{T}$ and to nothing more. That is one answer to the question why so many classical distributions belong to such families; Efron and Hastie (2016, p. 67) give another, that their mathematical tractability "was greatly facilitated by exponential structure, even if that structure went unrecognized". The same structure gives every exponential family a conjugate family of priors, for which the posterior stays in the family of the prior (Priors, Posteriors and Conjugate Families, Theorem 1).
The exponential dispersion form of generalized linear models
A regression model needs one mean per observation and, often, a scale that is shared by all. Nelder and Wedderburn (1972), in their abstract, describe fitting observations "distributed according to some exponential family" by iterative weighted regression. The form used throughout this unit for such observations, the exponential dispersion form, writes the density or mass function of a response $Y$ as
$$f(y; \theta, \phi) = \exp\Big\{\frac{w\,[\,y\theta - b(\theta)\,]}{\phi} + c(y, \phi/w)\Big\}, \qquad (6)$$
with a natural parameter $\theta$, a cumulant function $b$, a dispersion parameter $\phi > 0$, a known prior weight $w > 0$, and a function $c$ that makes the total probability 1 and does not involve $\theta$. For fixed $\phi$ and $w$, (6) is a natural exponential family in $y$ with natural parameter $w\theta/\phi$: it is the form (1) with $T(y) = y$, $h(y) = e^{c(y, \phi/w)}$ and log-partition function $(w/\phi)\,b(\theta)$.
Theorem 7 (mean and variance in the dispersion form). If $\theta$ is an interior point,
$$E[Y] = \mu = b'(\theta), \qquad \operatorname{Var}(Y) = \frac{\phi}{w}\,b''(\theta) = \frac{\phi}{w}\,V(\mu),$$
where $V(\mu) = b''(\theta(\mu))$ is the variance function and $\theta(\mu)$ is the inverse of $b'$.
Proof. Write $\vartheta = w\theta/\phi$ for the natural parameter of the family (1) and $A(\vartheta) = (w/\phi)\,b(\phi\vartheta/w)$ for its log-partition function. By the chain rule $A'(\vartheta) = b'(\theta)$ and $A''(\vartheta) = (\phi/w)\,b''(\theta)$, and Theorem 2 identifies these as the mean and variance of $Y$. Theorem 3 makes $b'$ strictly increasing, so its inverse $\theta(\mu)$ exists. $\square$
So the variance of a response is the product of three things: the dispersion $\phi$, shared by all observations; the variance function $V(\mu)$, fixed by the family; and $1/w$, known for each observation. Efron and Hastie (2016, eqs. 8.20 to 8.27) set out generalized linear models from the same one-parameter family, with $\phi = 1$ and no weights.
| Family | $\theta(\mu)$ | $b(\theta)$ | $\phi$ | $V(\mu)$ |
|---|---|---|---|---|
| Normal, variance $\sigma^2$ | $\mu$ | $\theta^2/2$ | $\sigma^2$ | $1$ |
| Poisson | $\ln\mu$ | $e^{\theta}$ | $1$ | $\mu$ |
| Binomial proportion, $m$ trials ($w = m$) | $\ln\frac{\mu}{1 - \mu}$ | $\ln(1 + e^{\theta})$ | $1$ | $\mu(1 - \mu)$ |
| Gamma, shape $\nu$ | $-1/\mu$ | $-\ln(-\theta)$ | $1/\nu$ | $\mu^2$ |
| Inverse Gaussian, shape $\lambda$ | $-1/(2\mu^2)$ | $-\sqrt{-2\theta}$ | $1/\lambda$ | $\mu^3$ |
| Negative binomial, shape $k$ known | $\ln\frac{\mu}{\mu + k}$ | $-k\ln(1 - e^{\theta})$ | $1$ | $\mu + \mu^2/k$ |
The rows are derived as follows; the article's script checks each by computing $b'$ and $b''$ numerically and comparing them with the mean and variance of the distribution computed directly.
- Normal. $-\frac{(y - \mu)^2}{2\sigma^2} - \frac12\ln(2\pi\sigma^2) = \frac{y\mu - \mu^2/2}{\sigma^2} - \frac{y^2}{2\sigma^2} - \frac12\ln(2\pi\sigma^2)$, so $\theta = \mu$, $b(\theta) = \theta^2/2$, $\phi = \sigma^2$. Then $b'' = 1$ and $\operatorname{Var}(Y) = \sigma^2$.
- Poisson. $y\ln\mu - \mu - \ln y!$: $\theta = \ln\mu$, $b = e^{\theta}$, $\phi = 1$, $c = -\ln y!$.
- Binomial proportion. If $S$ successes occur in $m$ trials and $Y = S/m$, then $P(Y = y) = \binom{m}{my}\mu^{my}(1 - \mu)^{m - my} = \exp\{m[y\theta - \ln(1 + e^{\theta})] + \ln\binom{m}{my}\}$ with $\theta = \ln\frac{\mu}{1 - \mu}$. So $w = m$, $\phi = 1$, and $\operatorname{Var}(Y) = \mu(1 - \mu)/m$, the variance of a sample proportion. In this unit a binomial response is always the proportion, with the number of trials as its prior weight. For the obese adults aged 60 and over, $m = 762$ and $\mu = 270/762$.
- Gamma with mean $\mu$ and shape $\nu$: the density $\frac{1}{\Gamma(\nu)}(\nu/\mu)^\nu y^{\nu - 1}e^{-\nu y/\mu}$ is $\exp\{\nu[-y/\mu - \ln\mu] + \nu\ln\nu + (\nu - 1)\ln y - \ln\Gamma(\nu)\}$. So $\theta = -1/\mu$, $b(\theta) = \ln\mu = -\ln(-\theta)$, $\phi = 1/\nu$; $b''(\theta) = 1/\theta^2 = \mu^2$, and $\operatorname{Var}(Y) = \mu^2/\nu$. The coefficient of variation $1/\sqrt\nu$ is the same for every mean, which is what a variance function $\mu^2$ says.
- Inverse Gaussian and negative binomial: Worked problems 2 and 4.
The variance functions $1$, $\mu$, $\mu^2$, $\mu^3$ are powers of the mean, and $\mu(1 - \mu)$ and $\mu + \mu^2/k$ are quadratics. Within the dispersion form, Theorem 4 applies for each fixed $\phi$ and $w$: the variance $\phi V(\mu)/w$, not $V$ alone, determines the distribution. With $V(\mu) = \mu$ and $w = 1$, the value $\phi = 1$ gives the Poisson family, and another value gives the multiples $\phi N$ of a Poisson variable $N$ with mean $\mu/\phi$: $P(\phi N = y) = \exp\{[y\ln\mu - \mu]/\phi + c(y, \phi)\}$ on $y = 0, \phi, 2\phi, \ldots$, with mean $\mu$ and variance $\phi\mu$ (Worked problem 5 makes the same point for $\mu^2$). One row of the table needs a remark. For the inverse Gaussian, the normalizing integral is finite also at $\theta = 0$: it is $\int_0^\infty y^{-3/2}e^{-\lambda/(2y)}\,dy$, which the substitution $u = 1/y$ turns into $\int_0^\infty u^{-1/2}e^{-\lambda u/2}\,du < \infty$. So its natural parameter space is $\theta \le 0$, and the member at $\theta = 0$ has no finite mean. The table, the canonical link (with $\eta < 0$) and Theorem 4 use only the interior $\theta < 0$. Whether counts of deaths really have variance equal to their mean is an empirical question, which Overdispersion in Count Data answers for weekly deaths and Overdispersion and Quasi-Likelihood takes up for regression.
What is proved, what is cited, what the experiment shows
Proved on this page: the convexity of the natural parameter space and of the log-partition function, with strictness for a minimal representation (Theorem 1); the moment generating function of $\mathbf{T}$ and the cumulants as derivatives of $A$ along any direction (Theorem 2), with the score's mean and the information as corollaries and observed information equal to expected information in the natural parameter; the one-to-one mean parametrization (Theorem 3); the determination of a natural exponential family by its variance function (Theorem 4); maximum likelihood as moment matching, concavity, uniqueness, non-existence when the average lies outside the possible means, and the equality of the variance of the mean's estimate with the Cramér–Rao bound (with the bound applying to every unbiased estimator when the data take finitely many values) (Theorem 5); the divergence between two members as a Bregman divergence, with an exact quadratic form (Theorem 6); the mean and variance in the dispersion form (Theorem 7); the six rows of the table.
Used from other pages: Jensen's inequality and its equality case, and the fact that a nonnegative variable with mean zero is zero with probability 1 (Probability Inequalities); the moment series and the first four cumulants, and the uniqueness theorem for moment generating functions, which that page states and cites (Moment Generating and Characteristic Functions); the variance of a linear combination; the reparametrization rule for information; invariance of maximum likelihood estimates.
Stated, not proved: that $A$ is infinitely differentiable with Hessian $\operatorname{Cov}(\mathbf{T})$ (Efron and Hastie, 2016, eqs. 5.56 and 5.57; only the directional form is proved and used here); that, among families whose support does not depend on the parameter, only exponential families have sufficient statistics of fixed dimension (Efron and Hastie, 2016, p. 66); Morris's classification of the families with quadratic variance functions (from the paper's abstract).
Computed on the unit's data: the Bernoulli quantities for the diabetes indicator; the gamma fit to body mass index by Newton's method in the natural parameters, against the method of moments; the divergence between two age-and-obesity groups. These describe these 5,107 adults, treated as a finite population as in the other units; the survey's sampling weights are ignored. Every number is computed and checked against this page by tools/experiments/statistics/exponential-families.py.
Worked problems
The problems were written for this article, on the unit's NHANES adults and on the families of the table. Each number is recomputed by the article's script.
Problem 1 (a sample with no maximum likelihood estimate). In the group of adults aged 18 to 39 who are not obese, 12 of 1,040 have diabetes. A random sample of 40 is drawn from the group. What is the probability that the maximum likelihood estimate of the natural parameter of the Bernoulli model does not exist? How large must the sample be for that probability to fall to 0.05 or below?
Solution. The estimate fails to exist exactly when the sample average is 0 or 1, by part 4 of Theorem 5 and the example after it (the same argument with $\theta \to +\infty$ covers an average of 1). Drawing with replacement, the observations are independent Bernoulli variables with $p = 12/1{,}040$, so the probability is
$$(1 - p)^{40} + p^{40} = \Big(\frac{1{,}028}{1{,}040}\Big)^{40} + \Big(\frac{12}{1{,}040}\Big)^{40} = 0.6286,$$
with the second term below $10^{-77}$: the probability of no diabetic in the sample is 0.6286 to four decimals. Drawing without replacement, the probability of no diabetic is $\binom{1{,}028}{40}/\binom{1{,}040}{40} = 0.6230$, a little smaller because each draw of a non-diabetic leaves the remaining group slightly richer in diabetics. For the second question, $(1{,}028/1{,}040)^n \le 0.05$ first holds at $n = 259$ adults, drawing with replacement. A sample of a few dozen from a group with a rare outcome is likely to contain no event at all, and then the log-odds cannot be estimated, however the computation is arranged.
Problem 2 (the negative binomial with known shape). The negative binomial distribution with mean $\mu$ and shape $k$ has $P(Y = y) = \frac{\Gamma(y + k)}{\Gamma(k)\,y!}\big(\frac{\mu}{\mu + k}\big)^y\big(\frac{k}{\mu + k}\big)^k$ for $y = 0, 1, 2, \ldots$ Show that for fixed $k$ it is an exponential dispersion family, find $\theta$, $b$ and $V$, and evaluate them for the Iowa weekly all-cause deaths, for which the Likelihood and Estimation unit estimates $\mu = 570.52$ and $k = 203.49$.
Solution. Write $\theta = \ln\frac{\mu}{\mu + k}$, which is negative. Then $\frac{k}{\mu + k} = 1 - \frac{\mu}{\mu + k} = 1 - e^{\theta}$, and
$$P(Y = y) = \exp\Big\{y\theta + k\ln(1 - e^{\theta}) + \ln\frac{\Gamma(y + k)}{\Gamma(k)\,y!}\Big\}.$$
This is (6) with $w = \phi = 1$, $b(\theta) = -k\ln(1 - e^{\theta})$ on $\theta < 0$, and $c(y) = \ln\frac{\Gamma(y + k)}{\Gamma(k)y!}$, which involves $k$ but not $\theta$; that is why $k$ must be known. Differentiating, $b'(\theta) = \frac{ke^{\theta}}{1 - e^{\theta}} = k\cdot\frac{\mu/(\mu + k)}{k/(\mu + k)} = \mu$, and $b''(\theta) = \frac{ke^{\theta}}{(1 - e^{\theta})^2} = \frac{\mu(\mu + k)}{k} = \mu + \frac{\mu^2}{k}$. So $V(\mu) = \mu + \mu^2/k$, the variance of Overdispersion in Count Data. For Iowa, $\theta = \ln(570.52/774.01) = -0.3050$ and the variance is $570.52 + 570.52^2/203.49 = 2{,}170.1$, nearly four times the Poisson value 570.52. With $k$ unknown the term $\ln\Gamma(y + k)$ mixes the data and the parameter, and the method of this problem does not put the family in the form (1).
Problem 3 (the divergence between two Poisson means). Use Theorem 6 to find $D(\text{Poisson}(\lambda_1) \,\|\, \text{Poisson}(\lambda_2))$, evaluate it for weekly means of 120 and 125 deaths, and compare it with the quadratic approximation.
Solution. Here $A(\theta) = e^{\theta}$, $\theta_j = \ln\lambda_j$ and $\mu(\theta_1) = \lambda_1$. Formula (4) gives
$$D = e^{\theta_2} - e^{\theta_1} - (\theta_2 - \theta_1)e^{\theta_1} = \lambda_2 - \lambda_1 - \lambda_1\ln\frac{\lambda_2}{\lambda_1} = \lambda_1\ln\frac{\lambda_1}{\lambda_2} - \lambda_1 + \lambda_2.$$
For $\lambda_1 = 120$ and $\lambda_2 = 125$ it is $120\ln(120/125) - 120 + 125 = 0.10136$ nats; the script confirms the value by summing $p_1\ln(p_1/p_2)$ over the counts directly. On the scale of the mean, $\theta_2 - \theta_1 \approx (\lambda_2 - \lambda_1)/V(\lambda_1)$ by Theorem 3, so (5) gives $D \approx \frac12(\lambda_2 - \lambda_1)^2/V(\lambda_1) = 25/240 = 0.10417$. So each week of data drawn from the first mean adds 0.10136 to the expected log-likelihood ratio in favor of the first mean over the second; how such a ratio is turned into a test is the subject of The Neyman–Pearson Lemma.
Problem 4 (the inverse Gaussian family). The inverse Gaussian density with mean $\mu$ and shape $\lambda$ is $f(y) = \sqrt{\lambda/(2\pi y^3)}\,\exp\{-\lambda(y - \mu)^2/(2\mu^2y)\}$ for $y > 0$. Put it in the form (6) and find its variance function.
Solution. Expand the exponent: $-\frac{\lambda(y^2 - 2\mu y + \mu^2)}{2\mu^2y} = \lambda\Big[y\cdot\Big(-\frac{1}{2\mu^2}\Big) + \frac1\mu\Big] - \frac{\lambda}{2y}$. With $\theta = -1/(2\mu^2)$ we have $1/\mu = \sqrt{-2\theta}$, so the bracket is $y\theta - b(\theta)$ with $b(\theta) = -\sqrt{-2\theta}$. The form (6) holds with $w = 1$, $\phi = 1/\lambda$ and $c(y, \phi) = -\frac{1}{2\phi y} + \frac12\ln\frac{1}{2\pi\phi y^3}$. Then $b'(\theta) = (-2\theta)^{-1/2} = \mu$ and $b''(\theta) = (-2\theta)^{-3/2} = \mu^3$, so $V(\mu) = \mu^3$ and $\operatorname{Var}(Y) = \mu^3/\lambda$. The script checks the expansion against a library implementation of the density and the variance at $\mu = 3$, $\lambda = 5$, where it is $27/5 = 5.4$.
Problem 5 (the family with variance function $\mu^2$). Which natural exponential family has variance function $V(\mu) = \mu^2$ with means $\Omega = (0, \infty)$?
Solution. By Theorem 4 there is at most one, so it suffices to find its log-partition function. By Theorem 3, $d\theta/d\mu = 1/\mu^2$, so $\theta(\mu) = c - 1/\mu$; choosing the origin of $\theta$ so that $c = 0$ (a shift of $\theta$ only multiplies $h$ by an exponential factor) gives $\mu = -1/\theta$ on $\theta < 0$, and $A'(\theta) = -1/\theta$, so $A(\theta) = -\ln(-\theta)$ up to a constant. The member with mean $\mu$ has $\theta = -1/\mu$ and, by Theorem 2, moment generating function
$$\exp\{A(\theta + t) - A(\theta)\} = \frac{-\theta}{-\theta - t} = \Big(1 - \frac{t}{-\theta}\Big)^{-1} = (1 - \mu t)^{-1}, \qquad t < 1/\mu,$$
the moment generating function of the exponential distribution with mean $\mu$ (Moment Generating and Characteristic Functions gives $(1 - t/\lambda)^{-\alpha}$ for the gamma, here with $\alpha = 1$ and $\lambda = 1/\mu$). So the family is the exponential distributions. The gamma family of the table also has $V(\mu) = \mu^2$, but its variance is $\phi\mu^2$ with $\phi = 1/\nu$: within the dispersion form, the variance function fixes the family only together with the dispersion.
Summary
- An exponential family has $f(x; \boldsymbol\theta) = h(x)\exp\{\boldsymbol\theta^{\mathsf T}\mathbf{T}(x) - A(\boldsymbol\theta)\}$; its support does not depend on $\boldsymbol\theta$.
- $\Theta$ is convex and $A$ is convex, strictly for a minimal representation (Theorem 1).
- $A(\boldsymbol\theta + s\mathbf{a}) - A(\boldsymbol\theta)$ is the cumulant generating function of $\mathbf{a}^{\mathsf T}\mathbf{T}$: $\partial A/\partial\theta_j = E[T_j]$, second derivatives give variances (Theorem 2). The score is $\mathbf{T} - \boldsymbol\mu$; information is $\operatorname{Cov}(\mathbf{T})$, observed equals expected.
- The mean of $\mathbf{T}$ determines the member (Theorem 3); in one dimension $d\mu/d\theta = V(\mu)$, and the variance function determines a natural exponential family (Theorem 4).
- Maximum likelihood: concave log-likelihood, unique estimate, $\boldsymbol\mu(\hat{\boldsymbol\theta}) = \bar{\mathbf{t}}$; no estimate when $\bar{\mathbf{t}}$ is not a possible mean; $\bar t$ has variance equal to the Cramér–Rao bound (Theorem 5).
- $D(f_{\boldsymbol\theta_1}\,\|\,f_{\boldsymbol\theta_2})$ is the Bregman divergence of $A$, exactly $\frac12\operatorname{Var}_{\boldsymbol\theta^*}((\boldsymbol\theta_2 - \boldsymbol\theta_1)^{\mathsf T}\mathbf{T})$ (Theorem 6).
- Dispersion form $\exp\{w[y\theta - b(\theta)]/\phi + c(y, \phi/w)\}$: $E[Y] = b'(\theta)$, $\operatorname{Var}(Y) = \phi V(\mu)/w$ (Theorem 7); $V = 1, \mu, \mu(1 - \mu), \mu^2, \mu^3, \mu + \mu^2/k$ for the six families of the table.
What we have learned
The binomial, Poisson, normal, gamma, inverse Gaussian and negative binomial distributions are tilts of a base function by an exponential of a linear function of the data. Everything a regression method needs from them follows from that form: the mean and the variance are derivatives of one function, the mean labels the member, the log-likelihood is concave in the natural parameter, and the variance is the dispersion times a function of the mean. What the form does not supply is a connection between the mean of each observation and the inputs of a regression. Link Functions and the Structure of a Generalized Linear Model supplies it, and shows why the natural parameter of this article gives the most convenient choice.
References
- Efron, B., and Hastie, T. (2016). Computer Age Statistical Inference: Algorithms, Evidence, and Data Science. Cambridge University Press. Section 5.5, "Exponential Families", pp. 64–68, and Section 8.2, "Generalized Linear Models", pp. 116–120. record · 9 pages
- Morris, C. N. (1982). Natural exponential families with quadratic variance functions. Annals of Statistics, 10(1), 65–80. https://doi.org/10.1214/aos/1176345690 record
- Nelder, J. A., and Wedderburn, R. W. M. (1972). Generalized linear models. Journal of the Royal Statistical Society, Series A, 135(3), 370–384. https://doi.org/10.2307/2344614 record · 5 pages