Asymptotics of the Maximum Likelihood Estimator

Why a maximum likelihood estimate from a large sample is close to the truth and approximately normal, with the variance the Cramér–Rao bound names, proved for a consistent root of the likelihood equation, which is the estimate itself in the standard models. Five regularity conditions stated; the existence of a consistent root of the likelihood equation and its asymptotic normality proved in one dimension from the law of large numbers, the central limit theorem and Slutsky's theorem; the Wald interval of the applied course derived as a corollary; the result checked on a Cauchy model whose likelihood has several maxima, and shown to fail for a uniform distribution and for overdispersed counts; the version for several parameters stated and cited. Worked problems on weekly death counts, an exponential rate, the log odds and a five-observation likelihood with three roots.

A random sample of 40 adults aged 18 to 39 from the National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018 contains 10 with hypertension. Confidence Intervals for a Proportion gives the textbook interval for the population proportion: the estimate plus or minus 1.96 estimated standard errors, $0.25 \pm 1.96\sqrt{0.25 \times 0.75/40}$, or (0.1158, 0.3842). The same recipe, estimate plus or minus 1.96 standard errors, is applied to Poisson means, regression coefficients and the parameters of mixture models. What justifies it for an estimate that is not an average? A theorem: under regularity conditions, the likelihood equation has a root that in a large sample is approximately normal, centered at the true value, with variance $1/(nI(\theta_0))$, where $I$ is the Fisher information. When the equation has only one root, as in the models of the course, that root is the maximum likelihood estimate.

In symbols, for a sample of $n$ independent observations with true parameter $\theta_0$, there are roots $\hat\theta_n$ of the likelihood equation, equal to the maximum likelihood estimate (MLE) in those models, that satisfy

$$\hat\theta_n \xrightarrow{p} \theta_0 \qquad \text{and} \qquad \sqrt{n}\,(\hat\theta_n - \theta_0) \xrightarrow{d} \text{normal with mean 0 and variance } \frac{1}{I(\theta_0)}.$$

The first statement is consistency and the second asymptotic normality. This article proves both for a single parameter, and says exactly which estimate they are proved for.

First we state five regularity conditions and check them on the unit's models. Then we prove that the likelihood equation has a root that converges to the true value. Next we prove a lemma on the curvature of the log-likelihood and, from it, asymptotic normality. The Wald interval follows as a corollary, with the delta method for functions of the parameter. Then an experiment on a Cauchy model tests the theorems where the estimate has no formula and the likelihood has several maxima. Finally we see three ways the conclusion fails and state the result for several parameters. The article assumes Fisher Information and the Cramér–Rao Bound, and uses the weak law of large numbers, the central limit theorem and Slutsky's theorem and the delta method without proving them again.

Likelihood Ratio, Wald and Score Tests builds on the two theorems and the lemma proved here.

What the theorem has to explain

For the proportion nothing new is needed. The estimate $\hat{p}$ is a sample mean of indicators, so the central limit theorem applies to it directly: $\sqrt{n}(\hat{p} - p)$ is approximately normal with variance $p(1-p)$. And $p(1-p)$ is $1/I(p)$, since the information of an indicator is $I(p) = 1/(p(1-p))$. The same holds for a Poisson mean, where $\hat\lambda = \bar{x}$ and $1/I(\lambda) = \lambda$.

Most maximum likelihood estimates are not averages. The estimate of a Cauchy location, of a negative binomial shape or of a mixture's parameters solves an equation that has no closed form (Maximum Likelihood Estimation works such cases). The theorem says that these estimates behave, in large samples, as if they were averages. The proof shows why: near the true value, the equation that defines the estimate is almost linear, and solving a linear equation whose right side is an average gives an average.

Here is the plan in plain words. The estimate solves $U(\theta) = 0$, where $U = \ell'$ is the score of the sample. Expand $U$ around the true value: $0 = U(\hat\theta) \approx U(\theta_0) + (\hat\theta - \theta_0)\,\ell''(\theta_0)$. So

$$\hat\theta - \theta_0 \approx \frac{U(\theta_0)}{-\ell''(\theta_0)}.$$

The numerator is a sum of $n$ independent scores with mean 0 and variance $I(\theta_0)$ each, so the central limit theorem makes it approximately normal with variance $nI(\theta_0)$. The denominator is a sum of $n$ independent terms with mean $I(\theta_0)$ each, so the law of large numbers makes it approximately $nI(\theta_0)$. The ratio is approximately normal with variance $nI/(nI)^2 = 1/(nI)$. Everything below is the work of making "approximately" exact.

The regularity conditions

The notation is the unit's. One observation $X$ has probability mass function or density $f(x;\theta)$, with score $u(x;\theta) = \partial \ln f(x;\theta)/\partial\theta$. The sample has log-likelihood $\ell(\theta) = \sum_i \ln f(X_i;\theta)$, score $U(\theta) = \ell'(\theta)$ and observed information $J(\theta) = -\ell''(\theta)$. The true parameter value is $\theta_0$, and $E_{\theta_0}$ and $P_{\theta_0}$ are computed under it. The likelihood equation is $U(\theta) = 0$.

  • C1 (model). $X_1, \ldots, X_n$ are independent with mass function or density $f(x;\theta_0)$, and the parameter space $\Theta$ is an open interval containing $\theta_0$.
  • C2 (common support). The set $S = \{x : f(x;\theta) > 0\}$ is the same for every $\theta \in \Theta$.
  • C3 (smoothness). For every $x \in S$, $\ln f(x;\theta)$ is three times continuously differentiable in $\theta$ on $\Theta$.
  • C4 (score identities). $E_{\theta_0}[u(X;\theta_0)] = 0$ and $E_{\theta_0}[u(X;\theta_0)^2] = -E_{\theta_0}[\partial u(X;\theta_0)/\partial\theta] = I(\theta_0)$, with $0 < I(\theta_0) < \infty$.
  • C5 (third-derivative bound). There are a number $c > 0$ with $(\theta_0 - c, \theta_0 + c) \subset \Theta$ and a function $M(x)$ such that $\left|\partial^3 \ln f(x;\theta)/\partial\theta^3\right| \le M(x)$ for all $x \in S$ and all $\theta$ with $|\theta - \theta_0| < c$, and $E_{\theta_0}[M(X)] < \infty$.

Each condition has a job. C1 puts the true value in the interior, so that the estimate can approach it from both sides. C2 and C3 make the log-likelihood a smooth function on a fixed set of possible data. C4 holds whenever the sum or integral of $f$ may be differentiated twice under the sign; Fisher Information and the Cramér–Rao Bound proves the two identities under its conditions R1 to R3. C5 controls how fast the curvature of the log-likelihood can change near $\theta_0$, uniformly over the data. The conditions are local: nothing is assumed about $\theta$ far from $\theta_0$, and nothing is concluded there either.

For textbook treatments of this theory see Casella and Berger (2002, Chapter 10, "Asymptotic Evaluations") and van der Vaart (1998, Chapter 5, "M- and Z-Estimators"). We did not have either book open while writing: they are given as further reading, by the chapter titles in their publishers' tables of contents, and no statement on this page rests on them. Every step below is proved from C1 to C5.

Checking the conditions. For the hypertension indicator, $\Theta = (0, 1)$, $S = \{0, 1\}$ and $\ln f = x\ln p + (1-x)\ln(1-p)$. C4 holds because sums over two points can always be differentiated term by term. The third derivative is $2x/p^3 - 2(1-x)/(1-p)^3$, bounded in absolute value by the constant $M = 2/(p_0 - c)^3 + 2/(1 - p_0 - c)^3$ whenever $|p - p_0| < c$, for any $c$ smaller than both $p_0$ and $1 - p_0$.

For a Poisson count, $\Theta = (0, \infty)$, $S = \{0, 1, 2, \ldots\}$ and $\ln f = -\lambda + x\ln\lambda - \ln x!$. The third derivative is $2x/\lambda^3$, at most $M(x) = 2x/(\lambda_0 - c)^3$ for $|\lambda - \lambda_0| < c < \lambda_0$, and $E[M(X)] = 2\lambda_0/(\lambda_0 - c)^3$ is finite.

For a Cauchy location, $\Theta$ is the real line, $S$ is the real line and $\ln f = -\ln\pi - \ln(1 + (x-\theta)^2)$. With $y = x - \theta$ the three derivatives in $\theta$ are

$$\frac{2y}{1+y^2}, \qquad -\frac{2(1-y^2)}{(1+y^2)^2}, \qquad \frac{4y(y^2-3)}{(1+y^2)^3}.$$

All three are bounded functions of $y$; the third is at most 2.91 in absolute value, so $M = 3$ serves for every $c$. The information is $I = 1/2$.

Consistency: a root near the true value

Theorem 1 (a consistent root). Assume C1 to C5. For every $\varepsilon > 0$, the probability that the likelihood equation $U(\theta) = 0$ has a root in $(\theta_0 - \varepsilon, \theta_0 + \varepsilon)$ at which $\ell$ has a local maximum tends to 1 as $n \to \infty$. Consequently there is a sequence of roots $\hat\theta_n$ with $\hat\theta_n \xrightarrow{p} \theta_0$.

Here is the idea. Compare the log-likelihood at the true value with its values a small distance $\varepsilon$ to either side. Per observation, the difference has a first-order term that averages to zero, a second-order term that averages to $-\tfrac12\varepsilon^2 I(\theta_0)$, and a remainder of order $\varepsilon^3$. For small $\varepsilon$ the negative second-order term wins. So with high probability the log-likelihood is higher at $\theta_0$ than at both ends of the interval. A continuous function that is higher in the middle of an interval than at its ends has a local maximum inside, where its derivative is zero.

Proof. Fix $x \in S$ and $\theta$ with $|\theta - \theta_0| < c$, and write $\delta = \theta - \theta_0$. By C3 and Taylor's theorem with the Lagrange form of the remainder, there is a point $\theta^*$ between $\theta_0$ and $\theta$ with

$$\ln f(x;\theta) - \ln f(x;\theta_0) = \delta\,u(x;\theta_0) + \frac{\delta^2}{2}\,\frac{\partial u(x;\theta_0)}{\partial\theta} + \frac{\delta^3}{6}\,\frac{\partial^3 \ln f(x;\theta^*)}{\partial\theta^3}.$$

Sum over the sample and divide by $n$:

$$\frac{\ell(\theta_0 + \delta) - \ell(\theta_0)}{n} = \delta A_n + \frac{\delta^2}{2} B_n + \frac{\delta^3}{6} C_n(\delta), \qquad (1)$$

where

$$A_n = \frac1n\sum_{i=1}^n u(X_i;\theta_0), \qquad B_n = \frac1n\sum_{i=1}^n \frac{\partial u(X_i;\theta_0)}{\partial\theta}, \qquad |C_n(\delta)| \le \bar{M}_n = \frac1n\sum_{i=1}^n M(X_i),$$

the last by C5. Each of $A_n$, $B_n$ and $\bar{M}_n$ is the mean of $n$ independent, identically distributed variables with a finite mean. C4 and C5 give $\partial u/\partial\theta$ and $M$ a finite mean but not necessarily a finite variance, so the weak law proved from Chebyshev's inequality does not apply to them. The weak law for independent observations with only a finite mean does (The Laws of Large Numbers, section "The weak law with only a finite mean"). By it and C4,

$$A_n \xrightarrow{p} 0, \qquad B_n \xrightarrow{p} -I(\theta_0), \qquad \bar{M}_n \xrightarrow{p} m = E_{\theta_0}[M(X)].$$

Write $I = I(\theta_0)$, and let $\varepsilon_0 = \min\{c,\ 3I/(2(m+1))\}$. Take any $\varepsilon$ with $0 < \varepsilon < \varepsilon_0$, and let $G_n$ be the event

$$|A_n| < \frac{\varepsilon I}{8}, \qquad B_n < -\frac{3I}{4}, \qquad \bar{M}_n < m + 1 .$$

Each of the three inequalities holds with probability tending to 1, by the three limits, so $P_{\theta_0}(G_n) \to 1$. On $G_n$, equation (1) with $\delta = \pm\varepsilon$ gives

$$\frac{\ell(\theta_0 \pm \varepsilon) - \ell(\theta_0)}{n} < \varepsilon\cdot\frac{\varepsilon I}{8} - \frac{\varepsilon^2}{2}\cdot\frac{3I}{4} + \frac{\varepsilon^3}{6}(m+1) = -\frac{\varepsilon^2 I}{4} + \frac{\varepsilon^3(m+1)}{6} < 0,$$

the last inequality because $\varepsilon < 3I/(2(m+1))$. So on $G_n$ the log-likelihood is strictly smaller at both $\theta_0 - \varepsilon$ and $\theta_0 + \varepsilon$ than at $\theta_0$.

The function $\ell$ is continuous on the closed interval $[\theta_0 - \varepsilon, \theta_0 + \varepsilon]$, so it attains a maximum there. On $G_n$ the maximum is not at either endpoint, since $\ell(\theta_0)$ is larger than both endpoint values. It is therefore at an interior point, which is a local maximum of $\ell$, and at an interior maximum of a differentiable function the derivative is zero. So on $G_n$ the likelihood equation has a root in $(\theta_0 - \varepsilon, \theta_0 + \varepsilon)$ that is a local maximum of $\ell$. This proves the first claim for $\varepsilon < \varepsilon_0$; for larger $\varepsilon$ the interval only grows.

For the sequence, let $\hat\theta_n$ be the root of the likelihood equation nearest to $\theta_0$ among those in $[\theta_0 - c/2, \theta_0 + c/2]$, taking the smaller if two are equally near, and let $\hat\theta_n = \theta_0$ if there is none. A nearest root exists when any does, because $U$ is continuous, so its roots in a closed interval form a closed set. For $\varepsilon < c/2$, the event $|\hat\theta_n - \theta_0| < \varepsilon$ contains the event that a root lies within $\varepsilon$ of $\theta_0$, whose probability tends to 1. Hence $\hat\theta_n \xrightarrow{p} \theta_0$. $\square$

Three remarks say what has and has not been shown.

The theorem is about a root, not about the global maximum. The likelihood equation can have several roots, and the theorem promises that one of them is near the truth. It does not say that the global maximizer of $\ell$ is that root, or that a global maximizer exists. The consistency of the global maximizer is a different theorem with different conditions, which involve no derivatives. The classical reference is Wald (1949), and it draws the same line. Of the proof in Cramér's textbook of 1946, Wald writes that it "establishes the consistency of some root of the likelihood equation but not necessarily that of the maximum likelihood estimate when the likelihood equation has several roots" (p. 595, footnote 3). His own proof makes "no differentiability assumptions (thus, not even the existence of the likelihood equation is postulated)". It rests on eight assumptions listed on his page 596, among them that the density is continuous in the parameter, that different parameter values give different distributions, that the density tends to zero as the parameter goes to infinity, that certain expected values of the logarithm of the density are finite, and that the parameter space is closed. Under them his Theorem 2 (p. 600) shows that any sequence of estimates at which the likelihood is at least a fixed positive fraction of the likelihood at the true value converges to the true value with probability one. A maximum likelihood estimate, when it exists, is such a sequence, with the fraction equal to 1. The theorem is stated here and not used: nothing below depends on it.

The root of the proof cannot be computed. It was defined as the root nearest to $\theta_0$, which is unknown. Two situations turn the theorem into a statement about a computable estimate.

Corollary 1 (a unique root). Suppose that, for every sample, the likelihood equation has at most one root in $\Theta$. Then that root, when it exists, is the consistent root of Theorem 1. With probability tending to 1 it exists and is a local maximum of $\ell$. If in addition $\ell$ is strictly concave on $\Theta$ for every sample, the root is the global maximizer of $\ell$, that is, the MLE.

Proof. On the event $G_n$ of the proof a root exists in $(\theta_0 - \varepsilon, \theta_0 + \varepsilon)$ and is a local maximum. If roots are unique, it is the only root, so the root in $\Theta$ coincides with $\hat\theta_n$ on $G_n$. A strictly concave function lies strictly below each of its tangent lines except at the point of tangency. At a root the tangent is horizontal, so $\ell(\theta) < \ell(\hat\theta_n)$ for every other $\theta$. Strict concavity also makes $\ell'$ strictly decreasing, so the root is unique. $\square$

This covers the unit's standard models. For the proportion, $\ell''(p) = -x/p^2 - (n-x)/(1-p)^2 < 0$ for every sample with at least one observation. For the Poisson mean, $\ell''(\lambda) = -\sum x_i/\lambda^2 < 0$ unless every count is zero.

Corollary 2 (the root nearest a consistent estimate). Let $[a, b]$ be a closed interval inside $\Theta$ with $a < \theta_0 < b$, and let $\tilde\theta_n$ be any estimator with $\tilde\theta_n \xrightarrow{p} \theta_0$. Let $\check\theta_n$ be a root of the likelihood equation nearest to $\tilde\theta_n$ among the roots in $[a, b]$: any of them, if several are equally near, and $\check\theta_n = \tilde\theta_n$ if there is no root in $[a, b]$. Then $\check\theta_n \xrightarrow{p} \theta_0$.

Proof. A nearest root exists when there is any, because the roots of the continuous function $U$ in a closed bounded interval form a closed bounded set. (In an open parameter space roots could pile up toward an end, with none nearest; the interval $[a, b]$ rules that out.) Take $\varepsilon > 0$ so small that $(\theta_0 - \varepsilon, \theta_0 + \varepsilon)$ lies inside $[a, b]$. Let $E_n$ be the event that the likelihood equation has a root $r_n$ within $\varepsilon$ of $\theta_0$; by Theorem 1, $P_{\theta_0}(E_n) \to 1$. On $E_n$ the root $r_n$ is in $[a, b]$, so $\check\theta_n$ is at least as near to $\tilde\theta_n$ as $r_n$ is. By the triangle inequality, twice,

$$|\check\theta_n - \theta_0| \le |\check\theta_n - \tilde\theta_n| + |\tilde\theta_n - \theta_0| \le |r_n - \tilde\theta_n| + |\tilde\theta_n - \theta_0| \le \varepsilon + 2|\tilde\theta_n - \theta_0| .$$

So $|\check\theta_n - \theta_0| > 3\varepsilon$ requires that $E_n$ fail or that $|\tilde\theta_n - \theta_0| > \varepsilon$, and both have probability tending to 0. $\square$

"With probability tending to 1" is not decoration. For the proportion the likelihood equation has the root $\hat{p} = x/n$ only when $0 < x < n$. A sample with no case has $\ell(p) = n\ln(1-p)$, which increases as $p$ falls toward 0 and has no maximum on the open interval $(0, 1)$. For 40 young adults that happens with probability $(1 - 0.2294)^{40} = 2.98 \times 10^{-5}$; for a sample of 10 it would be 0.0739. The theorem allows for such samples and says only that their probability vanishes.

Why should the true value be favored at all? The proof found a second-order term $-\tfrac12\varepsilon^2 I$, which is the local form of a global fact: the expected log-likelihood $E_{\theta_0}[\ln f(X;\theta)]$ is largest at $\theta = \theta_0$. The Likelihood Function proves that inequality, for mass functions and for densities with $E_{\theta_0}[\,|\ln f(X;\theta_0)|\,]$ finite, as the theorem of its section "Why the true value tends to have the higher likelihood", with equality only when the two parameter values give the same distribution. Theorem 1 did not use it: the proof above is local and needs only C1 to C5. The Fisher information article shows that the gap is approximately $\tfrac12 I(\theta_0)(\theta - \theta_0)^2$ nearby.

The curvature near the true value

The proof of normality divides by the curvature of the log-likelihood at a point near $\theta_0$, not at $\theta_0$ itself. The following lemma says that this makes no difference in the limit.

Lemma (curvature near the true value). Assume C1 to C5. If $\theta_n^*$ are random points with $\theta_n^* \xrightarrow{p} \theta_0$, then

$$\frac{J(\theta_n^*)}{n} = -\frac{\ell''(\theta_n^*)}{n} \xrightarrow{p} I(\theta_0).$$

Proof. The idea is that C5 makes each term of $\ell''$ change by at most $M(x)$ per unit of $\theta$. For $x \in S$ and $|\theta - \theta_0| < c$, the mean value theorem applied to $\partial u(x;\theta)/\partial\theta$, whose derivative is the third derivative of $\ln f$, gives

$$\left|\frac{\partial u(x;\theta)}{\partial\theta} - \frac{\partial u(x;\theta_0)}{\partial\theta}\right| \le |\theta - \theta_0|\,M(x).$$

Summing over the sample, on the event $|\theta_n^* - \theta_0| < c$,

$$\left|\frac{\ell''(\theta_n^*)}{n} - \frac{\ell''(\theta_0)}{n}\right| \le |\theta_n^* - \theta_0|\;\bar{M}_n .$$

Now $\ell''(\theta_0)/n = B_n \xrightarrow{p} -I(\theta_0)$ and $\bar{M}_n \xrightarrow{p} m$, by the finite-mean weak law, as in the proof of Theorem 1. For any $\eta > 0$,

$$P\!\left(\left|\frac{\ell''(\theta_n^*)}{n} + I(\theta_0)\right| > \eta\right) \le P\big(|\theta_n^* - \theta_0| \ge c\big) + P\!\left(|\theta_n^* - \theta_0|\,\bar{M}_n > \frac{\eta}{2}\right) + P\!\left(|B_n + I(\theta_0)| > \frac{\eta}{2}\right).$$

The first and third terms tend to 0. For the second, $|\theta_n^* - \theta_0|\,\bar{M}_n > \eta/2$ requires $\bar{M}_n \ge m + 1$ or $|\theta_n^* - \theta_0| > \eta/(2(m+1))$, and both have probability tending to 0. $\square$

One consequence is immediate. A consistent root has $J(\hat\theta_n)/n \xrightarrow{p} I(\theta_0) > 0$, so with probability tending to 1 the log-likelihood is strictly concave at it. A consistent root is, in large samples, a local maximum and not a minimum or a point of inflection.

Asymptotic normality

Theorem 2 (asymptotic normality). Assume C1 to C5, and let $\hat\theta_n$ be any sequence with $\hat\theta_n \xrightarrow{p} \theta_0$ that solves the likelihood equation with probability tending to 1. Then

$$\sqrt{n}\,(\hat\theta_n - \theta_0) \xrightarrow{d} \text{normal with mean 0 and variance } \frac{1}{I(\theta_0)},$$

and in addition

$$\sqrt{n}\,(\hat\theta_n - \theta_0) - \frac{1}{I(\theta_0)}\cdot\frac{U(\theta_0)}{\sqrt{n}} \xrightarrow{p} 0. \qquad (2)$$

Proof. The score $U$ is continuously differentiable, by C3. By the mean value theorem there is a point $\bar\theta_n$ between $\theta_0$ and $\hat\theta_n$ with

$$U(\hat\theta_n) = U(\theta_0) + (\hat\theta_n - \theta_0)\,\ell''(\bar\theta_n).$$

Since $\bar\theta_n$ lies between $\theta_0$ and $\hat\theta_n$, $|\bar\theta_n - \theta_0| \le |\hat\theta_n - \theta_0|$, so $\bar\theta_n \xrightarrow{p} \theta_0$. Let $D_n = -\ell''(\bar\theta_n)/n$. By the lemma, $D_n \xrightarrow{p} I(\theta_0)$. On the event that $U(\hat\theta_n) = 0$, the display reads

$$\sqrt{n}\,(\hat\theta_n - \theta_0)\,D_n = \frac{U(\theta_0)}{\sqrt{n}}. \qquad (3)$$

The right side is $\sqrt{n}$ times the mean of the $n$ independent variables $u(X_i;\theta_0)$, which have mean 0 and variance $I(\theta_0)$ by C4. By the central limit theorem (Convergence in Distribution and the Proof of the Central Limit Theorem),

$$\frac{U(\theta_0)}{\sqrt{n}} \xrightarrow{d} Z, \qquad Z \text{ normal with mean 0 and variance } I(\theta_0).$$

Define $V_n = U(\theta_0)/(\sqrt{n}\,D_n)$ when $D_n \ne 0$ and $V_n = 0$ otherwise. Since $D_n \xrightarrow{p} I(\theta_0) \ne 0$ and the reciprocal is continuous at a nonzero point, $1/D_n \xrightarrow{p} 1/I(\theta_0)$ (with $1/D_n$ read as 0 on the event $D_n = 0$, whose probability tends to 0). By Slutsky's theorem (the same article, section "Slutsky's theorem"), a sequence converging in distribution to $Z$, multiplied by a sequence converging in probability to the constant $1/I(\theta_0)$, converges in distribution to $Z/I(\theta_0)$. That limit is normal with mean 0 and variance $I(\theta_0)/I(\theta_0)^2 = 1/I(\theta_0)$. By (3), $\sqrt{n}(\hat\theta_n - \theta_0) = V_n$ on the event that $U(\hat\theta_n) = 0$ and $D_n \ne 0$, and the probability of that event tends to 1. Two sequences that are equal with probability tending to 1 differ by a sequence that converges to 0 in probability, so by Slutsky's theorem once more they have the same limit in distribution. This proves the first claim.

For (2), on the same event the left side of (2) equals

$$\frac{U(\theta_0)}{\sqrt{n}}\left(\frac{1}{D_n} - \frac{1}{I(\theta_0)}\right).$$

The first factor converges in distribution to $Z$ and the second converges in probability to 0, so by Slutsky's theorem the product converges in distribution to $0 \cdot Z = 0$. Convergence in distribution to a constant is convergence in probability. $\square$

Statement (2) is the precise form of the claim that the estimate behaves like an average. The error of the estimate, scaled by $\sqrt{n}$, differs negligibly from the scaled mean of the independent variables $u(X_i;\theta_0)/I(\theta_0)$. Likelihood Ratio, Wald and Score Tests uses it to prove that the three test statistics agree in large samples.

One step of the proof was passed over, and the convention of the collection should be stated. The point $\bar\theta_n$ depends on the sample, and treating it as a random variable requires that it can be chosen measurably. We do not prove that; it is part of the measure-theoretic bookkeeping that Random Variables and Distribution Functions sets aside.

The theorem and the Cramér–Rao bound. The limiting variance $1/I(\theta_0)$ is, after division by $n$, the Cramér–Rao bound for unbiased estimators. That is the sense in which the MLE is called asymptotically efficient. Two cautions keep the statement honest. The bound concerns unbiased estimators at a fixed $n$, and the MLE is often biased (Problem 2). So the theorem does not say the MLE attains the bound at any $n$. And nothing proved here shows that no other estimator has a smaller limiting variance; that question belongs to the theory of efficiency (van der Vaart, 1998, Chapter 8, "Efficiency of Estimators").

The Wald interval

The theorem contains the unknown $\theta_0$ in its variance. Replacing it by the estimate gives an interval that can be computed.

Corollary 3 (Wald interval). Under the conditions of Theorem 2, for $0 < \alpha < 1$ let $z = z_{1-\alpha/2}$ be the standard normal quantile. Then

$$P_{\theta_0}\!\left(\hat\theta_n - \frac{z}{\sqrt{J(\hat\theta_n)}} \le \theta_0 \le \hat\theta_n + \frac{z}{\sqrt{J(\hat\theta_n)}}\right) \to 1 - \alpha .$$

If the information $I(\theta)$ is continuous at $\theta_0$, the same holds with $J(\hat\theta_n)$ replaced by $nI(\hat\theta_n)$.

Proof. By the lemma with $\theta_n^* = \hat\theta_n$, $J(\hat\theta_n)/n \xrightarrow{p} I(\theta_0) > 0$, so $\sqrt{J(\hat\theta_n)/n} \xrightarrow{p} \sqrt{I(\theta_0)}$ by the continuity of the square root. (On the event of vanishing probability where $J(\hat\theta_n) \le 0$ the interval is undefined; count it as not covering.) By Theorem 2 and Slutsky's theorem,

$$\sqrt{J(\hat\theta_n)}\,(\hat\theta_n - \theta_0) = \sqrt{\frac{J(\hat\theta_n)}{n}}\cdot\sqrt{n}\,(\hat\theta_n - \theta_0) \xrightarrow{d} \sqrt{I(\theta_0)}\cdot\frac{Z}{I(\theta_0)},$$

which is standard normal. The interval covers $\theta_0$ exactly when this quantity lies in $[-z, z]$. The standard normal distribution function is continuous, so the probability of that event converges to $\Phi(z) - \Phi(-z) = 1 - \alpha$. If $I$ is continuous at $\theta_0$, then $I(\hat\theta_n) \xrightarrow{p} I(\theta_0)$, and the same argument applies with $nI(\hat\theta_n)$. $\square$

The quantity $1/\sqrt{J(\hat\theta_n)}$, or $1/\sqrt{nI(\hat\theta_n)}$, is the standard error of the MLE.

Back to the 40 adults. $I(p) = 1/(p(1-p))$, so the standard error is $\sqrt{\hat{p}(1-\hat{p})/n} = 0.06847$, the margin is $1.96 \times 0.06847 = 0.1342$, and the interval is (0.1158, 0.3842). This is the Wald interval of Confidence Intervals for a Proportion, to the digit. For the indicator model the observed and expected versions coincide, since $J(\hat{p}) = x/\hat{p}^2 + (n-x)/(1-\hat{p})^2 = n/(\hat{p}(1-\hat{p}))$.

Example (what "coverage tends to 0.95" does not promise). The corollary is a limit for a fixed $\theta_0$ as $n$ grows. It gives no rate and no guarantee at any particular $n$. The course article computes the exact coverage of this interval for $p = 0.2294$ and $n = 40$: 91.09%, not 95%. It also shows the coverage rising and falling irregularly as $n$ grows, and collapsing when $p$ is near 0 or 1, where the probability of a sample with no case is not negligible. The theorem is true and the interval is still poor at $n = 40$; a limit theorem licenses an approximation without measuring its error.

Functions of the parameter

Corollary 4 (delta method). Under the conditions of Theorem 2, let $g$ be differentiable at $\theta_0$ with $g'(\theta_0) \ne 0$. Then

$$\sqrt{n}\,\big(g(\hat\theta_n) - g(\theta_0)\big) \xrightarrow{d} \text{normal with mean 0 and variance } \frac{[g'(\theta_0)]^2}{I(\theta_0)}.$$

This is the delta method applied to Theorem 2. The limiting variance, divided by $n$, is the Cramér–Rao bound for estimating $\tau(\theta) = g(\theta)$. It also agrees with the change-of-parameter rule for the information. If $\eta = g(\theta)$ is used as the parameter, its information is $I(\theta)/[g'(\theta)]^2$, so applying Theorem 2 to $\hat\eta = g(\hat\theta)$ directly gives the same variance. The limit theorem does not depend on the parametrization. The Wald interval does, as Problem 3 shows.

An experiment where the estimate has no formula

For the proportion and the Poisson mean, Theorem 2 is the central limit theorem and nothing more. A fair test needs a model whose estimate is not an average. The Cauchy location model is a hard case. Its sample mean does not converge at all. Its likelihood equation is a polynomial equation of degree $2n - 1$, with up to $2n - 1$ real roots. And conditions C1 to C5 hold, with $I = 1/2$. The theorems predict that a consistent root has $n\operatorname{Var}(\hat\theta_n) \approx 2$ and that the interval $\hat\theta_n \pm 1.96\sqrt{2/n}$ covers 95% of the time, in the limit.

The experiment draws samples with $\theta_0 = 0$, at four sample sizes, 2,000 samples for each of 30 seeds. The estimate is the root of the likelihood equation reached from the sample median by the iteration $\theta \leftarrow \sum_i w_i x_i/\sum_i w_i$ with weights $w_i = 1/(1 + (x_i - \theta)^2)$, whose fixed points are exactly the roots. The median is a consistent starting point, in the spirit of Corollary 2, though the root reached is not guaranteed to be the nearest one. In every sample the root reached had positive observed information, so it was a local maximum. The table gives medians over the 30 seeds, with interquartile ranges.

$n$ $n \times$ mean squared error of the root of the sample median Coverage, expected information Coverage, observed information
5 5.32 (5.02 to 5.88) 5.69 (5.23 to 6.25) 0.878 (0.873 to 0.882) 0.864 (0.856 to 0.868)
20 2.32 (2.27 to 2.37) 2.79 (2.69 to 2.87) 0.932 (0.928 to 0.936) 0.938 (0.935 to 0.941)
80 2.08 (2.02 to 2.09) 2.54 (2.48 to 2.58) 0.946 (0.941 to 0.949) 0.948 (0.946 to 0.952)
320 2.03 (1.99 to 2.09) 2.51 (2.44 to 2.56) 0.947 (0.943 to 0.951) 0.949 (0.945 to 0.951)

Four observations, all of this experiment.

  • The scaled error of the root falls toward the predicted 2: from 5.32 at $n = 5$ to 2.03 at $n = 320$.
  • The sample median, the natural simple estimate, levels off near 2.5, about a quarter above the bound. The likelihood root uses the data better, as the Fisher information article anticipated.
  • Coverage of both intervals rises toward 0.95 and is within a percentage point of it from $n = 80$. At $n = 5$ it is 0.86 to 0.88: the normal approximation is poor in small samples.
  • At $n = 5$ the likelihood equation had more than one root in 34.9% of samples (interquartile range 34.4% to 35.3%). Of the 60,000 samples, 39,145 had one root, 15,831 three, 4,251 five, 662 seven and 111 nine, the largest number possible. The root reached from the median was the global maximizer in 98.5% of samples (98.3% to 98.6%).
Left: the Cauchy log-likelihood for the five observations −0.5, 0, 0.4, 7.5, 8 (marked on the axis). Notice the two local maxima, near 0.10 and near 7.51, with a local minimum between them: the likelihood equation has three roots, and only the one near 0.10 is the global maximum. Right: histograms of √n times the estimate over 2,000 samples of size 5 (outline) and size 320 (filled), with the limiting normal density of variance 2. Notice that at n = 320 the histogram follows the curve, while at n = 5 the tails are heavier: 49 of the 2,000 scaled estimates fall outside the plotted range of −6 to 6, and none do at n = 320.
Left: the Cauchy log-likelihood for the five observations −0.5, 0, 0.4, 7.5, 8 (marked on the axis). Notice the two local maxima, near 0.10 and near 7.51, with a local minimum between them: the likelihood equation has three roots, and only the one near 0.10 is the global maximum. Right: histograms of √n times the estimate over 2,000 samples of size 5 (outline) and size 320 (filled), with the limiting normal density of variance 2. Notice that at n = 320 the histogram follows the curve, while at n = 5 the tails are heavier: 49 of the 2,000 scaled estimates fall outside the plotted range of −6 to 6, and none do at n = 320.

Where the conclusion fails

Each condition excludes something real. Three failures show what.

A support that moves: rate $n$ and a limit that is not normal. Let the observations be uniform on $(0, \theta)$, so that C2 fails. The likelihood is $\theta^{-n}$ for $\theta \ge M = \max_i X_i$ and 0 below, so the MLE is $M$. It is always below $\theta$. A worked problem of Convergence in Distribution and the Proof of the Central Limit Theorem proves that for the uniform distribution on $(0, 1)$ the scaled shortfall $n(1 - M)$ converges in distribution to an exponential variable with mean 1, from

$$P\big(n(1 - M) > t\big) = \left(1 - \frac{t}{n}\right)^n \to e^{-t} .$$

Dividing every observation by $\theta$ turns a uniform sample on $(0, \theta)$ into one on $(0, 1)$, so $n(\theta - M)/\theta$ has the same limit. The error of the MLE is of order $1/n$, not $1/\sqrt{n}$, and its limit is exponential, not normal. A Wald interval makes no sense here: $\ell''(\theta) = n/\theta^2$ is positive, so the "observed information" is negative. Problem 5 builds the correct interval.

A wrong model: the estimate converges, the standard error does not. The theorems assume that the data come from the model. Iowa's weekly deaths from all causes have mean 570.52 over the 313 weeks and variance 2,196.66, which is 3.85 times the mean (Overdispersion in Count Data). Fit the Poisson model anyway. The MLE is still the sample mean, and the sample mean still converges to the mean weekly count. But the Poisson standard error, $\sqrt{\hat\lambda/n} = 1.35$, assumes the variance equals the mean. The standard error computed from the sample variance, the one the central limit theorem supports for independent counts, is $\sqrt{2{,}196.66/313} = 2.65$, larger by the factor $\sqrt{3.85} = 1.96$. A nominal 95% Wald interval from the Poisson model is half as wide as it should be. The information identity C4 is a property of the model, and it gives the right variance only when the model is right. Even 2.65 rests on independence between weeks, which counts with winter peaks do not have.

A true value on the boundary. C1 puts $\theta_0$ inside an open interval. Suppose instead that the true value is at an end of the parameter space, such as a mixture weight that is truly 0. The estimate can then err in one direction only, so its limiting distribution cannot be a normal distribution centered at the truth. Model-Based Clustering and the EM Algorithm meets this situation when it asks how many components a mixture has.

Several parameters

The statement for a vector parameter $\boldsymbol\theta = (\theta_1, \ldots, \theta_k)$ replaces the information by the information matrix $I(\boldsymbol\theta)$ of the Fisher information article and the reciprocal by the matrix inverse. The conditions are of the same kind as C1 to C5: an open parameter set, partial derivatives up to the third order and an invertible information matrix. Under them there is a sequence of roots of the $k$ likelihood equations $\partial\ell/\partial\theta_j = 0$ with

$$\hat{\boldsymbol\theta}_n \xrightarrow{p} \boldsymbol\theta_0, \qquad \sqrt{n}\,(\hat{\boldsymbol\theta}_n - \boldsymbol\theta_0) \xrightarrow{d} \text{multivariate normal with mean } \mathbf{0} \text{ and covariance matrix } I(\boldsymbol\theta_0)^{-1}.$$

This is stated here and not proved. A proof follows the same plan, with the matrix of second derivatives in place of $\ell''$ and the central limit theorem for random vectors in place of the one-dimensional theorem. The reference for the general theory is van der Vaart (1998, Chapter 5, "M- and Z-Estimators"); we cite the chapter by its title and by the publisher's summary of it, which describes an introduction to the consistency and asymptotic normality of these estimators with maximum likelihood estimators as a special case. We did not have its text open, and give no theorem number.

In practice the statement is used as follows. Let $J$ be the matrix of minus the second derivatives of $\ell$. The standard error of the $j$th component is the square root of the $j$th diagonal entry of $J(\hat{\boldsymbol\theta})^{-1}$. A Wald interval is the estimate plus or minus $z$ standard errors. For a normal sample with both parameters unknown, the information matrix is diagonal with entries $1/\sigma^2$ and $2/\sigma^2$. The standard errors are therefore $\hat\sigma/\sqrt{n}$ for the mean and $\hat\sigma/\sqrt{2n}$ for the standard deviation. The EM Algorithm computes standard errors this way for the three parameters of a mixture, and the coefficient tests of logistic regression are Wald tests of this kind.

What is proved, what is cited, what the experiment shows

Proved on this page, in one dimension under C1 to C5:

  • the existence of a consistent root of the likelihood equation that is a local maximum (Theorem 1);
  • its identification with the MLE when the root is unique or the log-likelihood strictly concave (Corollary 1);
  • the consistency of the root nearest a consistent estimate (Corollary 2);
  • the curvature lemma, and asymptotic normality with the linear representation (Theorem 2);
  • the Wald interval (Corollary 3).

Used from other pages: the weak law of large numbers for observations with a finite mean, the central limit theorem, Slutsky's theorem and the delta method (the Probability Theory unit), and the score identities (Fisher Information and the Cramér–Rao Bound).

Stated and cited, not proved: the version for several parameters; the consistency of the global maximizer; any optimality of the limiting variance among all estimators. The measurability of the intermediate point in the mean value theorem is assumed.

Observed in the experiment: for Cauchy samples of 5 to 320 observations, the scaled error of the likelihood root and the coverage of its Wald intervals approach the predicted values. At $n = 5$ they are far from them. These are results of one model at four sample sizes, 30 seeds each. Every number on the page is produced by tools/experiments/statistics/asymptotics-of-the-mle.py.

Worked problems

These problems are posed for this article on the unit's data and on small exact cases; the unit has no course assignments. Every number is recomputed by the article's script.

Problem 1 (a Poisson mean). Iowa recorded 38,395 cancer deaths in the 313 weeks of 2014 to 2019. Treating the weekly counts as independent Poisson counts with mean $\lambda$, find the MLE, its standard error and the 95% Wald interval. Which of the article's results justify each step?

Solution. $\hat\lambda = 122.67$, with standard error 0.626 and interval (121.44, 123.89). The log-likelihood is $\ell(\lambda) = -n\lambda + \sum x_i \ln\lambda$ plus a constant, with $U(\lambda) = -n + \sum x_i/\lambda$ and $\ell''(\lambda) = -\sum x_i/\lambda^2 < 0$. The likelihood equation has the single root $\hat\lambda = \bar{x} = 38{,}395/313 = 122.67$, and $\ell$ is strictly concave, so by Corollary 1 this root is the MLE and the consistent root. C1 to C5 were checked for the Poisson model above, so Theorem 2 applies with $I(\lambda) = 1/\lambda$. The information is continuous in $\lambda$, so Corollary 3 gives the standard error $1/\sqrt{nI(\hat\lambda)} = \sqrt{\hat\lambda/n} = \sqrt{122.67/313} = 0.626$ and the interval $122.67 \pm 1.96 \times 0.626$. Here the observed information gives the same number: $J(\hat\lambda) = \sum x_i/\hat\lambda^2 = n/\hat\lambda$. For a year of 52 weeks, Corollary 4 with $g(\lambda) = 52\lambda$ gives an expected 6,379 deaths with standard error $52 \times 0.626 = 32.6$.

Problem 2 (a biased estimate that is still asymptotically normal). Lifetimes $X_1, \ldots, X_n$ are independent and exponential with rate $\rho$, density $\rho e^{-\rho x}$ for $x > 0$.

  • (a) Find the MLE and check C1 to C5.
  • (b) State its limiting distribution, and confirm it with the delta method.
  • (c) Show that $E[\hat\rho] = n\rho/(n-1)$, and reconcile the bias with Theorem 2.

Solution. (a) $\ln f = \ln\rho - \rho x$, so $u = 1/\rho - x$, $\partial u/\partial\rho = -1/\rho^2$ and the third derivative of $\ln f$ is $2/\rho^3$. The log-likelihood $\ell(\rho) = n\ln\rho - \rho\sum x_i$ has $\ell'' = -n/\rho^2 < 0$ and the single root $\hat\rho = 1/\bar{x}$, the MLE by Corollary 1. C1: $\Theta = (0, \infty)$ is open. C2: the support is $(0, \infty)$ for every $\rho$. C3: $\ln f$ is smooth in $\rho$. C4: $E[u] = 1/\rho - E[X] = 0$ since the mean lifetime is $1/\rho$, and $E[u^2] = \operatorname{Var}(X) = 1/\rho^2 = -E[\partial u/\partial\rho]$, so $I(\rho) = 1/\rho^2$. C5: for $|\rho - \rho_0| < c < \rho_0$ the third derivative is at most the constant $M = 2/(\rho_0 - c)^3$.

(b) By Theorem 2, $\sqrt{n}(\hat\rho - \rho)$ converges in distribution to a normal variable with mean 0 and variance $1/I(\rho) = \rho^2$. The delta method gives the same: $\bar{X}$ has mean $1/\rho$ and variance $1/(n\rho^2)$, and $g(t) = 1/t$ has $g'(1/\rho) = -\rho^2$, so the limiting variance of $\sqrt{n}(1/\bar{X} - \rho)$ is $\rho^4 \cdot (1/\rho^2) = \rho^2$.

(c) The sum $T = \sum X_i$ of $n$ independent exponential lifetimes has density $\rho^n t^{n-1}e^{-\rho t}/(n-1)!$ for $t > 0$. This is proved by induction with the convolution formula (Random Variables and Distribution Functions). It is the exponential density for $n = 1$. If it holds for $n$, the density of the sum of $n + 1$ lifetimes at $t$ is

$$\int_0^t \frac{\rho^n s^{n-1}e^{-\rho s}}{(n-1)!}\,\rho e^{-\rho(t-s)}\,ds = \frac{\rho^{n+1}e^{-\rho t}}{(n-1)!}\int_0^t s^{n-1}\,ds = \frac{\rho^{n+1}t^n e^{-\rho t}}{n!}.$$

Then, for $n \ge 2$,

$$E\!\left[\frac1T\right] = \int_0^\infty \frac1t\cdot\frac{\rho^n t^{n-1}e^{-\rho t}}{(n-1)!}\,dt = \frac{\rho^n}{(n-1)!}\cdot\frac{(n-2)!}{\rho^{n-1}} = \frac{\rho}{n-1},$$

using $\int_0^\infty t^{k}e^{-\rho t}\,dt = k!/\rho^{k+1}$ with $k = n - 2$. So $E[\hat\rho] = E[n/T] = n\rho/(n-1)$: the MLE overestimates the rate, by the factor 1.111 when $n = 10$. There is no conflict with Theorem 2. The bias is $\rho/(n-1)$, of order $1/n$, while the standard deviation of the limit is $\rho/\sqrt{n}$. Their ratio is $\sqrt{n}/(n-1)$: 0.35 at $n = 10$ and 0.032 at $n = 1{,}000$. Multiplied by $\sqrt{n}$, the bias tends to 0, so it does not appear in the limiting distribution. Asymptotic normality around the true value is compatible with a bias at every finite $n$.

Problem 3 (two Wald intervals from the same data). For the 40 adults with 10 cases, compute the 95% Wald interval for the log odds $\eta = \ln\frac{p}{1-p}$ and transform it back to an interval for $p$. Compare with the Wald interval for $p$, and say which is correct.

Solution. The back-transformed interval is (0.1401, 0.4054), against (0.1158, 0.3842) on the $p$ scale. The MLE of a function of the parameter is the function of the MLE (Maximum Likelihood Estimation proves this invariance), so $\hat\eta = \ln(0.25/0.75) = -1.0986$. The information about $\eta$ in one observation is $p(1-p)$, by the change-of-parameter rule, so the standard error is $1/\sqrt{n\hat{p}(1-\hat{p})} = 1/\sqrt{7.5} = 0.3651$. Corollary 4 gives the same: $g'(p) = 1/(p(1-p))$, and $0.06847/(0.25 \times 0.75) = 0.3651$. The interval for $\eta$ is $-1.0986 \pm 1.96 \times 0.3651$, that is $(-1.8143, -0.3829)$. Applying $p = 1/(1 + e^{-\eta})$ to both ends gives (0.1401, 0.4054).

Both intervals are justified by the same theorem, and both have coverage tending to 95% as $n$ grows. At $n = 40$ they differ by 0.02 at each end, and the theorem does not choose between them. The Wald interval is not invariant under a change of parameter, because it is symmetric on whichever scale it is computed. A scale on which the estimate is closer to normal gives a better interval at a given $n$. The interval from the log odds cannot leave $(0, 1)$ and is longer on the side away from 0, as the skewness of $\hat{p}$ requires. Likelihood Ratio, Wald and Score Tests derives intervals from the likelihood ratio and the score, which do not depend on the scale in this way.

Problem 4 (a likelihood equation with three roots). Five observations, $-0.5$, $0$, $0.4$, $7.5$ and $8$, are modeled as a sample from a Cauchy distribution with location $\theta$.

  • (a) Find the roots of the likelihood equation and classify them.
  • (b) Which is the MLE, and what are its standard errors from the observed and the expected information?
  • (c) What does one step of the iteration $\theta \leftarrow \theta + U(\theta)/(nI)$ from the sample median give?

Solution. (a) There are three roots: 0.0963, 4.7554 and 7.5055. The score is $U(\theta) = \sum_i 2(x_i - \theta)/(1 + (x_i - \theta)^2)$; clearing denominators gives a polynomial of degree 9, whose real roots the script finds and then refines. The observed information $J(\theta) = \sum_i 2(1 - (x_i-\theta)^2)/(1 + (x_i-\theta)^2)^2$ is 4.103 at the first root, −0.555 at the second and 2.874 at the third. So the first and third are local maxima and the second is a local minimum, between the two clusters of observations.

(b) The log-likelihood, $-5\ln\pi - \sum_i \ln(1 + (x_i - \theta)^2)$, is −14.298 at the first root, −19.822 at the second and −18.108 at the third. The first root is the MLE, $\hat\theta = 0.0963$; the likelihood there is 45.2 times the likelihood at the other maximum. Its standard error from the observed information is $1/\sqrt{4.103} = 0.494$. The expected information is $nI = 5/2$, the same wherever the estimate falls, giving $\sqrt{2/5} = 0.632$. The two differ because the observed information reflects this sample: three observations close together near the estimate determine it more sharply than the average sample of five does.

(c) The sample median is 0.4, where $U(0.4) = -1.1492$. One step gives $0.4 + (-1.1492)/2.5 = -0.0597$, already much closer to the root 0.0963 than the median was. The sample mean, 3.08, lies in the empty stretch between the clusters and is near none of the roots. An iteration started at the mean, or near 7.5, could end at the wrong maximum. Started at the median, which is a consistent estimate, it ends at the right one here. This is Corollary 2 at work in a sample far too small for any limit theorem.

Problem 5 (an interval for the uniform model). For a sample of $n$ from the uniform distribution on $(0, \theta)$ with maximum $M$, find an exact 95% confidence interval for $\theta$ of the form $[M, kM]$, and evaluate $k$ for $n = 5$.

Solution. $k = 0.05^{-1/n}$, which is 1.821 for $n = 5$. Since $M \le \theta$ always, the lower limit $M$ never misses. The upper limit misses when $kM < \theta$, that is when $M < \theta/k$, which has probability $(1/k)^n$ because all $n$ observations must fall below $\theta/k$. Setting $(1/k)^n = 0.05$ gives $k = 0.05^{-1/n}$, and the interval $[M, kM]$ covers $\theta$ with probability exactly 0.95 for every $\theta$ and every $n$. For $n = 5$ and $M = 0.9$ the interval is $[0.9, 1.639]$. Its length is of order $1/n$: since $0.05^{-1/n} = e^{(\ln 20)/n}$ and $\ln 20 = 2.996$, the upper limit is about $M(1 + 3/n)$ for large $n$. This is the interval the exponential limit of the section above leads to. It has nothing in common with a Wald interval: it lies entirely on one side of the estimate, and it shrinks like $1/n$.

Summary

  • Conditions C1 to C3: independent observations and an interior true value; a support that does not depend on the parameter; three continuous derivatives of $\ln f$.
  • Conditions C4 and C5: the score identities with $0 < I(\theta_0) < \infty$; a third derivative bounded, on an interval around $\theta_0$ inside $\Theta$, by a function with finite mean.
  • Theorem 1: with probability tending to 1 the likelihood equation has a root near $\theta_0$ that is a local maximum, and a sequence of roots converges in probability to $\theta_0$. If the root is unique it is that sequence, and if $\ell$ is strictly concave it is the MLE.
  • Lemma: $J(\theta_n^*)/n \xrightarrow{p} I(\theta_0)$ whenever $\theta_n^* \xrightarrow{p} \theta_0$.
  • Theorem 2: for a consistent root, $\sqrt{n}(\hat\theta_n - \theta_0)$ converges in distribution to a normal variable with mean 0 and variance $1/I(\theta_0)$, and differs negligibly from $U(\theta_0)/(\sqrt{n}\,I(\theta_0))$.
  • Wald interval: $\hat\theta_n \pm z_{1-\alpha/2}/\sqrt{J(\hat\theta_n)}$, or with $nI(\hat\theta_n)$, has coverage tending to $1 - \alpha$. For a proportion it is $\hat{p} \pm z\sqrt{\hat{p}(1-\hat{p})/n}$.
  • For a function $g$, the limiting variance is $[g'(\theta_0)]^2/I(\theta_0)$; Wald intervals computed on different scales differ at finite $n$.
  • The conclusion fails for the uniform distribution on $(0,\theta)$, where $n(\theta - M)/\theta$ has an exponential limit; under a wrong model, where the standard error is wrong; and at a boundary of the parameter space.
  • Several parameters: the limiting covariance matrix is $I(\boldsymbol\theta_0)^{-1}$ (cited).

What we have learned

Near the true value the likelihood equation is nearly linear, with a slope that the law of large numbers fixes at $-nI(\theta_0)$ and a right side that the central limit theorem makes normal. Solving it gives an estimate that is consistent and asymptotically normal with variance $1/(nI(\theta_0))$, the value the Cramér–Rao bound singled out. The interval of the applied course, estimate plus or minus 1.96 standard errors, is this theorem with the information evaluated at the estimate. The theorem is about a root of the likelihood equation near the truth, it is a statement about limits, and it assumes the model. The Cauchy experiment, the uniform distribution and the overdispersed death counts mark those three boundaries.

The same expansion, carried one term further in the log-likelihood itself, gives the large-sample distribution of the likelihood ratio statistic. Likelihood Ratio, Wald and Score Tests takes that step. For models whose likelihood equation cannot be solved directly because part of the data is missing, The EM Algorithm gives an iteration that finds a root.

References

  • Casella, G., and Berger, R. L. (2002). Statistical Inference (2nd ed.). Duxbury. Chapter 10, "Asymptotic Evaluations". record · 11 pages
  • van der Vaart, A. W. (1998). Asymptotic Statistics. Cambridge University Press. Chapter 5, "M- and Z-Estimators", and Chapter 8, "Efficiency of Estimators". record · 8 pages
  • Wald, A. (1949). Note on the consistency of the maximum likelihood estimate. The Annals of Mathematical Statistics, 20(4), 595–601. https://doi.org/10.1214/aoms/1177729952 record · 2 pages