Maximum Likelihood Estimation

Choosing the parameter value under which the observed data are most probable. The estimators of the applied course rederived as maximum likelihood estimates, with the bias of the normal variance estimate; the invariance property proved for any function of the parameter; maxima that calculus does not find, likelihoods with no maximum and with many; and the negative binomial fitted to weekly deaths and to word counts, where no formula exists, with a proof that the maximum exists when the variance exceeds the mean. Worked problems on a count of zero, a Cauchy likelihood with two peaks, the uniform model and functions of the fitted negative binomial.

Iowa recorded 178,573 deaths from all causes in the 313 weeks of 2014 to 2019, a mean of 570.52 a week with a variance of 2,196.66 (divisor $n - 1$). Overdispersion in Count Data shows that the Poisson distribution cannot describe these counts and proposes the negative binomial, which has two parameters, a mean $\mu$ and a shape $k$. Which values of $\mu$ and $k$ should be used, and how are they found when no formula gives them? The values under which the 313 observed counts are most probable are $\mu = 570.52$ and $k = 203.49$. The first is the sample mean. The second solves an equation that has no solution in closed form and is found by iteration, in two steps from a reasonable start.

The maximum likelihood estimate (MLE) of a parameter is the value $\hat\theta$ at which the likelihood function of the observed data is largest. The rule that produces it from any dataset, a function of the random sample, is the maximum likelihood estimator; the estimate is its value on the data at hand. The Likelihood Function defines the likelihood $L(\theta)$ and the log-likelihood $\ell(\theta) = \ln L(\theta)$, and this article assumes it.

First we rederive the estimators of the applied course as maximum likelihood estimates. Then we prove the invariance property: the MLE of a function of the parameter is the function of the MLE. Next we collect the cases where differentiation fails, where the maximum is not unique and where there is none. Then we fit the negative binomial to the Iowa deaths and to word counts, where the estimate must be computed. Finally we connect the method to the loss functions that classifiers and language models minimize, and ask how good the estimates are.

The estimators of the course, rederived

The applied course estimates a proportion by the sample proportion, a Poisson mean by the sample mean, and a normal mean and variance by $\bar{x}$ and $s^2$. Where do these choices come from? All are maximum likelihood estimates except $s^2$, and $s^2$ is the maximum likelihood estimate of the variance multiplied by $n/(n - 1)$.

Here is the method. The likelihood and its logarithm peak at the same place, so we maximize $\ell$. Where $\ell$ is differentiable, its derivative

$$U(\theta) = \ell'(\theta)$$

is called the score, and an interior maximum must satisfy the likelihood equation $U(\theta) = 0$. A solution of that equation is only a candidate: it could be a minimum, or one of several local maxima. In each case below the score is positive to the left of the solution and negative to the right, so $\ell$ rises up to the solution and falls after it. That makes the solution the maximum over the whole parameter space, and no second-derivative test is needed.

A binomial proportion

For $x$ successes in $n$ trials, $\ell(p) = x\ln p + (n - x)\ln(1 - p)$, dropping the binomial coefficient, which does not involve $p$. For $0 < x < n$ the score is

$$U(p) = \frac{x}{p} - \frac{n - x}{1 - p} = \frac{x - np}{p(1 - p)}.$$

The denominator is positive on $0 < p < 1$, so the score has the sign of $x - np$: positive for $p < x/n$, zero at $p = x/n$, negative beyond. Hence

$$\hat{p} = \frac{x}{n}.$$

The MLE of a proportion is the sample proportion. For the 10 hypertensive adults among the 40 young adults of the previous article, $\hat{p} = 0.25$. The score at $p = 0.20$ is $12.50$, at 0.25 it is $0.00$, and at 0.30 it is $-9.52$.

A Poisson mean

For independent Poisson counts $x_1, \ldots, x_n$, $\ell(\lambda) = (\sum x_i)\ln\lambda - n\lambda$ plus a term free of $\lambda$. If at least one count is positive,

$$U(\lambda) = \frac{\sum x_i}{\lambda} - n = \frac{n(\bar{x} - \lambda)}{\lambda},$$

positive for $\lambda < \bar{x}$ and negative for $\lambda > \bar{x}$. Hence $\hat\lambda = \bar{x}$. For Iowa's weekly cancer deaths $\hat\lambda = 122.67$, and for Bortkiewicz's horse-kick deaths $\hat\lambda = 122/200 = 0.61$, the value The Poisson Distribution fitted without saying why.

A normal mean and variance

For independent normal observations with mean $\mu$ and variance $\sigma^2$,

$$\ell(\mu, \sigma^2) = -\frac{n}{2}\ln(2\pi\sigma^2) - \frac{1}{2\sigma^2}\sum_{i=1}^{n}(x_i - \mu)^2.$$

There are two parameters, and we maximize in two stages. For any fixed $\sigma^2$, $\ell$ is largest where $\sum(x_i - \mu)^2$ is smallest. Since $\sum(x_i - \mu)^2 = \sum(x_i - \bar{x})^2 + n(\bar{x} - \mu)^2$, that is at $\mu = \bar{x}$, whatever $\sigma^2$ is. So $\hat\mu = \bar{x}$. With $\mu = \bar{x}$ and $S = \sum(x_i - \bar{x})^2$, the derivative in $\sigma^2$ is

$$\frac{\partial\ell}{\partial\sigma^2} = -\frac{n}{2\sigma^2} + \frac{S}{2\sigma^4} = \frac{S - n\sigma^2}{2\sigma^4},$$

positive for $\sigma^2 < S/n$ and negative beyond (assuming the observations are not all equal, so that $S > 0$). Hence

$$\hat\mu = \bar{x}, \qquad \hat\sigma^2 = \frac{1}{n}\sum_{i=1}^{n}(x_i - \bar{x})^2.$$

Every pair $(\mu, \sigma^2)$ satisfies $\ell(\mu, \sigma^2) \le \ell(\bar{x}, \sigma^2) \le \ell(\bar{x}, \hat\sigma^2)$, so this is the maximum over the whole plane.

The divisor is $n$, not the $n - 1$ of the sample variance $s^2$. For Student's ten sleep differences, $\bar{x} = 1.58$ and $S = 13.616$, so $\hat\sigma^2 = 1.3616$ against $s^2 = 1.5129$, and $\hat\sigma = 1.1669$ against $s = 1.2300$. The log-likelihood is $-15.733$ at the MLE and $-15.759$ at $(\bar{x}, s^2)$, lower as it must be.

Example (the maximum likelihood estimate of a variance is biased). The estimator $\hat\sigma^2$ is too small on average, by the factor $(n - 1)/n$. The split of the sum of squares, applied to the random variables with $\mu$ their true mean, gives $\sum(X_i - \bar{X})^2 = \sum(X_i - \mu)^2 - n(\bar{X} - \mu)^2$. The first sum has expected value $n\sigma^2$. The second term has expected value $n\operatorname{Var}(\bar{X}) = \sigma^2$. So

$$E\Big[\sum(X_i - \bar{X})^2\Big] = (n - 1)\sigma^2, \qquad E[\hat\sigma^2] = \frac{n - 1}{n}\,\sigma^2, \qquad E[s^2] = \sigma^2.$$

For $n = 10$ the factor is $0.9$. The argument used no normality, only independent observations with a common mean and variance. Maximum likelihood does not promise unbiased estimators. What it does promise, in large samples, is the subject of Asymptotics of the Maximum Likelihood Estimator.

Other models on this site

Model Parameter Maximum likelihood estimate
Binomial, $x$ of $n$ $p$ $x/n$
Poisson $\lambda$ $\bar{x}$
Normal $\mu$, $\sigma^2$ $\bar{x}$, $\frac{1}{n}\sum(x_i - \bar{x})^2$
Exponential waiting times, density $\lambda e^{-\lambda x}$ $\lambda$ $1/\bar{x}$
Tokens from a vocabulary (multinomial) $\theta_t$ for each term $T_t/\sum_{t'} T_{t'}$, the relative frequencies
Documents containing a term (Bernoulli model) $p_t$ the fraction of documents that contain $t$
Uniform on $[0, \theta]$ $\theta$ the largest observation

For the exponential row, $\ell(\lambda) = n\ln\lambda - \lambda\sum x_i$ and $U(\lambda) = n/\lambda - \sum x_i$, positive below $\lambda = n/\sum x_i = 1/\bar{x}$ and negative above. The multinomial row has one parameter per term and the constraint that the parameters sum to 1. Naive Bayes derives it with a Lagrange multiplier, and Language Models for Retrieval uses it as the document model. The Bernoulli row is the binomial row applied term by term, as in the Bernoulli model. The uniform row is treated below, because differentiation does not find it.

Invariance

The sample gives $\hat{p} = 0.25$ for the proportion with hypertension. What is the maximum likelihood estimate of the odds of hypertension, $p/(1 - p)$? It is $0.25/0.75 = 0.3333$, the odds computed from $\hat{p}$. No new maximization is needed, and this is true of every function of the parameter.

Here is the idea in one sentence. A one-to-one function only renames the parameter values, so the best-supported value keeps its place under the new names; for other functions, each new value inherits the likelihood of the best old value it comes from.

Theorem (invariance of the maximum likelihood estimate). Let $\hat\theta$ maximize $L(\theta)$ over $\Theta$, and let $g$ be any function on $\Theta$. For each value $\eta$ in the range of $g$, define the induced likelihood

$$L^{*}(\eta) = \sup\{L(\theta) : \theta \in \Theta,\ g(\theta) = \eta\}.$$

Then $g(\hat\theta)$ maximizes $L^{*}$. If $g$ is one-to-one, $L^{*}(\eta) = L(g^{-1}(\eta))$, the likelihood of the model written in terms of $\eta$.

Proof. For every $\eta$, the set $\{\theta : g(\theta) = \eta\}$ is a subset of $\Theta$, so $L^{*}(\eta) \le \sup_{\theta \in \Theta} L(\theta) = L(\hat\theta)$. The value $\hat\theta$ itself belongs to the set $\{\theta : g(\theta) = g(\hat\theta)\}$, so $L^{*}(g(\hat\theta)) \ge L(\hat\theta)$. Together, $L^{*}(g(\hat\theta)) = L(\hat\theta) \ge L^{*}(\eta)$ for every $\eta$. If $g$ is one-to-one, the set $\{\theta : g(\theta) = \eta\}$ has the single member $g^{-1}(\eta)$, and the supremum is $L(g^{-1}(\eta))$. $\square$

For a one-to-one $g$ the theorem is a statement about the model alone. For other functions it rests on the definition of $L^{*}$, which is the natural one: a value $\eta$ is given the likelihood of the best-supported parameter value consistent with it, the profile likelihood of the previous article. The formulation is that of Zehna (1966), a note of one page. Zehna observes that the proofs then given in textbooks assumed, openly or tacitly, that the function is one-to-one, an assumption under which invariance could not be used for "even as common a case as the variance, $p(1 - p)$, of a Bernoulli random variable". For an arbitrary function $u$ of the parameter he defines the supremum above, calls it "the likelihood function induced by $u$", and proves the theorem by the inequality used here.

Examples.

  • Log odds. The MLE of $\ln\frac{p}{1 - p}$ for the 40 adults is $\ln(0.3333) = -1.0986$. This is why logistic regression can be fitted on the log odds scale and its results read back as probabilities.
  • A probability from a fitted model. Under the Poisson model the probability of a corps-year with no horse-kick death is $e^{-\lambda}$, so its MLE is $e^{-0.61} = 0.5434$, or 108.7 of the 200 corps-years. The observed share is $109/200 = 0.545$. Every "expected count" in the course's fitted tables is a maximum likelihood estimate by this theorem.
  • A standard deviation. The MLE of $\sigma$ is $\sqrt{\hat\sigma^2} = 1.1669$ for the sleep data.
  • A function that is not one-to-one. The variance of a single trial is $\eta = p(1 - p)$, and $p$ and $1 - p$ give the same $\eta$. Its MLE is $0.25 \times 0.75 = 0.1875$. The induced likelihood at $\eta = 0.1875$ is the larger of $L(0.25) = 0.1444$ and $L(0.75) = 4.14 \times 10^{-11}$.

Example (a common misstatement). "Unbiasedness carries over to functions in the same way." It does not. $s^2$ is unbiased for $\sigma^2$, but $s$ is not unbiased for $\sigma$: unless $s$ is constant, $E[s]^2 < E[s^2] = \sigma^2$, because $\operatorname{Var}(s) = E[s^2] - E[s]^2 > 0$. Being the maximum likelihood estimate is preserved by every function. Being unbiased is preserved by functions of the form $a + b\theta$, by linearity. A strictly convex or strictly concave function of an unbiased estimator that is not constant is biased, by Jensen's inequality (Probability Inequalities), as $s$ is. Stigler (2007) recounts that Fisher, in his paper of 1912, was so taken by the invariance of the estimates that he called the criterion "absolute": the estimate of the square of a constant was the square of the estimate.

When calculus does not find the maximum

The likelihood equation finds interior maxima of differentiable likelihoods. Four things can go wrong, and each occurs in models used on this site.

The maximum is on the boundary

Suppose none of the 40 adults had hypertension. Then $L(p) = (1 - p)^{40}$, which is 1.0000 at $p = 0$, 0.6690 at $p = 0.01$, 0.1285 at $p = 0.05$ and 0.0148 at $p = 0.10$. It decreases on the whole interval, so the maximum is at the endpoint: $\hat{p} = 0 = x/n$. The formula $x/n$ still holds, but the score $-40/(1 - p)$ is never zero; the likelihood equation has no solution.

The estimate itself is the difficulty. An estimated probability of exactly zero says the event cannot happen. In text models that is fatal: a term that never occurred in a class's training documents gets probability zero, and one such term makes the probability of a whole document zero. Probability for Information Retrieval shows the infinite log odds that result, and Language Models for Retrieval the documents that drop out of a ranking. The remedy on those pages, smoothing, replaces the MLE by an estimate that is never zero. Naive Bayes derives add-one smoothing as the mean of a posterior distribution, the same likelihood combined with a prior.

The likelihood is not differentiable at its maximum

Let $x_1, \ldots, x_n$ be independent and uniform on $[0, \theta]$. The joint density is $\theta^{-n}$ when every observation lies in $[0, \theta]$ and zero otherwise, so

$$L(\theta) = \begin{cases} 0, & \theta < x_{(n)}, \\ \theta^{-n}, & \theta \ge x_{(n)}, \end{cases}$$

where $x_{(n)}$ is the largest observation. On $\theta \ge x_{(n)}$ the likelihood is decreasing, so the maximum is at the smallest permitted value: $\hat\theta = x_{(n)}$. The derivative of $\theta^{-n}$ is never zero, and at the maximum the likelihood jumps. The estimate is found by reading the likelihood, not by differentiating it.

The estimator is biased downward, since the largest observation can never exceed $\theta$. The maximum is at most $t$ exactly when all $n$ observations are, so $P(X_{(n)} \le t) = (t/\theta)^n$ for $0 \le t \le \theta$, with density $nt^{n-1}/\theta^n$, and

$$E[X_{(n)}] = \int_0^\theta t\cdot\frac{n t^{n - 1}}{\theta^n}\,dt = \frac{n}{n + 1}\,\theta.$$

The factor is 0.7500 for $n = 3$, 0.9091 for $n = 10$ and 0.9756 for $n = 40$.

The maximum is not unique

Let the observations be uniform on $[\theta, \theta + 1]$. The joint density is 1 when every observation lies in the interval, that is, when $x_{(n)} - 1 \le \theta \le x_{(1)}$ with $x_{(1)}$ the smallest observation, and 0 otherwise. Every $\theta$ in that range maximizes the likelihood. There is a whole interval of maximum likelihood estimates, and the method gives no reason to prefer one. A second kind of non-uniqueness, separate peaks of equal height, is the subject of the second worked problem.

There is no maximum

The likelihood can grow without bound. Fit a mixture of two normal distributions with equal weights to the ten sleep differences: each observation has density $\tfrac{1}{2}f_1(x) + \tfrac{1}{2}f_2(x)$, where $f_1$ is normal with mean $\mu_1$ and standard deviation $\sigma_1$ and $f_2$ normal with mean $\mu_2$ and standard deviation $\sigma_2$. Put $\mu_1$ exactly on one observation, say the largest, 4.6, and let $f_2$ be any normal density. The observation 4.6 contributes at least $\tfrac{1}{2}f_1(4.6) = 1/(2\sigma_1\sqrt{2\pi})$, and every other observation at least $\tfrac{1}{2}f_2(x_i) > 0$. So

$$L \ge \frac{1}{2\sigma_1\sqrt{2\pi}}\prod_{x_i \ne 4.6}\tfrac{1}{2}f_2(x_i) \longrightarrow \infty \quad \text{as } \sigma_1 \to 0.$$

With $f_2$ fixed at mean 1.58 and standard deviation 1.1669, the log-likelihood is $-18.96$ for $\sigma_1 = 1$, then $-16.86$, $-14.56$, $-12.25$ and $-9.95$ as $\sigma_1$ falls to 0.1, 0.01, 0.001 and 0.0001. It passes the single normal's maximum of $-15.73$ and keeps rising by about $\ln 10 = 2.30$ for each factor of ten. A spike on one data point is not a sensible fit, and the likelihood rewards it without limit. Stigler (2007) describes this example, for the five-parameter mixture in which the weight is also free, as part of the folklore of the 1950s. In practice the answer taken from such a model is a local maximum, sought with the EM algorithm, as in model-based clustering; The EM Algorithm shows what the algorithm does and does not guarantee.

Logistic regression has the same defect when the classes can be separated perfectly. Classification Loss Functions shows a fit where "the likelihood keeps rising as the coefficients grow" and no finite coefficients maximize it.

No closed form: the negative binomial

Back to Iowa's weekly deaths from all causes. The negative binomial distribution with mean $\mu$ and shape $k$, derived in Overdispersion in Count Data as a Poisson count whose rate has a gamma distribution, has probabilities

$$P(X = x) = \frac{\Gamma(x + k)}{x!\,\Gamma(k)}\Big(\frac{k}{k + \mu}\Big)^{k}\Big(\frac{\mu}{k + \mu}\Big)^{x}, \qquad x = 0, 1, 2, \ldots$$

and variance $\mu + \mu^2/k$. (This is that article's formula with its $\theta$ written as $\mu/k$.) Small $k$ means strong overdispersion, and as $k \to \infty$ the distribution tends to the Poisson. That article estimated $k$ by matching the variance: setting $\mu + \mu^2/k$ equal to the sample variance gives $k = \bar{x}^2/(s^2 - \bar{x}) = 200.2$. That is a method of moments estimate, obtained by equating moments of the model to moments of the sample. What does maximum likelihood give?

The log-likelihood

The gamma functions can be removed. Since $\Gamma(k + 1) = k\,\Gamma(k)$, the ratio $\Gamma(x + k)/\Gamma(k)$ is the product $k(k + 1)\cdots(k + x - 1)$ of $x$ factors (and 1 when $x = 0$). The same product shows the Poisson limit. In $\frac{\Gamma(x + k)}{\Gamma(k)}\big(\frac{\mu}{k + \mu}\big)^{x} = \mu^x\prod_{j=0}^{x - 1}\frac{k + j}{k + \mu}$ each of the $x$ ratios tends to 1 as $k \to \infty$, and $\big(\frac{k}{k + \mu}\big)^{k} = \big(1 + \frac{\mu}{k}\big)^{-k}$ tends to $e^{-\mu}$, so $P(X = x)$ tends to $e^{-\mu}\mu^x/x!$. For observations $x_1, \ldots, x_n$,

$$\ell(\mu, k) = \sum_{i=1}^{n}\sum_{j=0}^{x_i - 1}\ln(k + j) + nk\ln\frac{k}{k + \mu} + \Big(\sum_{i=1}^{n} x_i\Big)\ln\frac{\mu}{k + \mu} - \sum_{i=1}^{n}\ln(x_i!).$$

The mean has a closed form

Differentiate in $\mu$ with $k$ fixed. Using $\frac{\partial}{\partial\mu}\ln\frac{k}{k + \mu} = -\frac{1}{k + \mu}$ and $\frac{\partial}{\partial\mu}\ln\frac{\mu}{k + \mu} = \frac{1}{\mu} - \frac{1}{k + \mu} = \frac{k}{\mu(k + \mu)}$,

$$\frac{\partial\ell}{\partial\mu} = -\frac{nk}{k + \mu} + \frac{k\sum x_i}{\mu(k + \mu)} = \frac{nk(\bar{x} - \mu)}{\mu(k + \mu)}.$$

This is positive for $\mu < \bar{x}$ and negative for $\mu > \bar{x}$, for every $k > 0$. So $\hat\mu = \bar{x} = 570.52$ whatever $k$ is, and the problem reduces to one parameter: maximize the profile log-likelihood $\ell_{\text{p}}(k) = \ell(\bar{x}, k)$.

The shape does not

The derivative of the profile log-likelihood is

$$U(k) = \ell_{\text{p}}'(k) = \sum_{i=1}^{n}\sum_{j=0}^{x_i - 1}\frac{1}{k + j} + n\ln\frac{k}{k + \bar{x}}.$$

(The derivative of $nk\ln k - (nk + n\bar{x})\ln(k + \bar{x})$ is $n\ln k + n - n\ln(k + \bar{x}) - n$, and the two $n$ cancel.) The equation $U(k) = 0$ mixes a sum of reciprocals with a logarithm, and it cannot be solved for $k$ by algebra. It must be solved numerically.

Newton's method solves $U(k) = 0$ by repeatedly replacing $U$ with its tangent line. From a current value $k$, the tangent line $U(k) + U'(k)(k' - k)$ is zero at

$$k' = k - \frac{U(k)}{U'(k)}, \qquad U'(k) = -\sum_{i=1}^{n}\sum_{j=0}^{x_i - 1}\frac{1}{(k + j)^2} + \frac{n\bar{x}}{k(k + \bar{x})},$$

and $k'$ becomes the new current value. Starting from the moments estimate 200.16:

Step $k$ $U(k)$ $\ell(\bar{x}, k)$
0 200.16 0.0070 $-1{,}645.66$
1 203.41 0.0002 $-1{,}645.65$
2 203.49 0.0000 $-1{,}645.65$

The maximum likelihood estimates are $\hat\mu = 570.52$ and $\hat{k} = 203.49$. A start much further away also arrives, in more steps:

Step 0 1 2 3 4 5 6
$k$ 50.00 83.56 126.32 168.43 195.34 203.01 203.49
$U(k)$ 2.0056 0.8430 0.3159 0.0957 0.0179 0.0010 0.0000
$\ell(\bar{x}, k)$ $-1{,}722.49$ $-1{,}678.26$ $-1{,}655.27$ $-1{,}647.18$ $-1{,}645.72$ $-1{,}645.65$ $-1{,}645.65$
Profile log-likelihood of the shape k for Iowa's 313 weekly all-cause death counts, on a logarithmic axis for k, with the Newton iterates from the start k = 50 marked. The dashed line is the Poisson log-likelihood, which the curve approaches as k grows. Notice that the peak stands more than 200 units of log-likelihood above the Poisson line, that the curve descends toward that line as k grows, and that the iterates climb from the left and close in on the peak.
Profile log-likelihood of the shape k for Iowa's 313 weekly all-cause death counts, on a logarithmic axis for k, with the Newton iterates from the start k = 50 marked. The dashed line is the Poisson log-likelihood, which the curve approaches as k grows. Notice that the peak stands more than 200 units of log-likelihood above the Poisson line, that the curve descends toward that line as k grows, and that the iterates climb from the left and close in on the peak.

The fitted model has variance $\hat\mu + \hat\mu^2/\hat{k} = 2{,}170.06$, close to the sample variance 2,196.66 but not equal to it: maximum likelihood does not match the variance, it weighs the whole shape of the distribution. Its log-likelihood is $-1{,}645.65$ against $-1{,}877.08$ for the Poisson model at $\lambda = 570.52$, a gain of 231.43. The Poisson model is the limit of the negative binomial as $k \to \infty$, so the negative binomial's maximum can never be lower. How large a gain is needed before the extra parameter is justified is a question for Likelihood Ratio, Wald and Score Tests. In this case the two estimates of $k$, 200.16 and 203.49, nearly agree.

Does a maximum exist?

The table shows Newton's method arriving somewhere. It does not show that the place is the maximum, or that one exists. The second question has a clean answer.

Theorem (existence of the negative binomial estimate). Let $x_1, \ldots, x_n$ be non-negative whole numbers with mean $\bar{x}$ and variance $\hat\sigma^2 = \frac{1}{n}\sum(x_i - \bar{x})^2$ (divisor $n$). If $\hat\sigma^2 > \bar{x}$, the profile log-likelihood $\ell_{\text{p}}(k)$ attains its maximum over $k > 0$ at a finite value $\hat{k}$, and $U(\hat{k}) = 0$.

Here is the plan. We show that $U$ is positive for all small enough $k$ and negative for all large enough $k$. Then $\ell_{\text{p}}$ rises at the left end and falls at the right end, and its maximum over the closed interval between is the maximum over all $k > 0$.

Proof. Small $k$. Let $m$ be the number of observations that are at least 1. Since $\hat\sigma^2 > \bar{x} \ge 0$, the observations are not all zero, so $m \ge 1$. Each such observation contributes the term $1/k$ (for $j = 0$) to the double sum, and all other terms are positive, so

$$U(k) \ge \frac{m}{k} + n\ln k - n\ln(k + \bar{x}) = \frac{1}{k}\big[m + nk\ln k - nk\ln(k + \bar{x})\big].$$

As $k \to 0^{+}$, both $k\ln k$ and $k\ln(k + \bar{x})$ tend to 0, so the bracket tends to $m > 0$. Hence there is an $a > 0$ with $U(k) > 0$ for all $0 < k \le a$.

Large $k$. Two elementary inequalities hold for $u \ge 0$. First, $\frac{1}{1 + u} \le 1 - u + u^2$, because $(1 + u)(1 - u + u^2) = 1 + u^3 \ge 1$. Second, $\ln(1 + u) \ge u - \frac{u^2}{2}$, because the difference of the two sides is 0 at $u = 0$ and has derivative $\frac{1}{1 + u} - 1 + u = \frac{u^2}{1 + u} \ge 0$. Apply the first with $u = j/k$ and the second with $u = \bar{x}/k$:

$$\frac{1}{k + j} \le \frac{1}{k} - \frac{j}{k^2} + \frac{j^2}{k^3}, \qquad n\ln\frac{k}{k + \bar{x}} = -n\ln\Big(1 + \frac{\bar{x}}{k}\Big) \le -\frac{n\bar{x}}{k} + \frac{n\bar{x}^2}{2k^2}.$$

Sum the first over $j = 0, \ldots, x_i - 1$ and over $i$. The number of terms is $\sum x_i = n\bar{x}$, and $\sum_i\sum_{j < x_i} j = \sum_i x_i(x_i - 1)/2$. Writing $B = \sum_i\sum_{j < x_i} j^2$,

$$U(k) \le \frac{n\bar{x}}{k} - \frac{\sum x_i(x_i - 1)}{2k^2} + \frac{B}{k^3} - \frac{n\bar{x}}{k} + \frac{n\bar{x}^2}{2k^2} = -\frac{1}{2k^2}\Big[\sum x_i^2 - n\bar{x}^2 - \sum x_i\Big] + \frac{B}{k^3}.$$

The bracket is $n\hat\sigma^2 - n\bar{x}$. So

$$U(k) \le -\frac{n(\hat\sigma^2 - \bar{x})}{2k^2} + \frac{B}{k^3},$$

which is negative whenever $k > 2B/[n(\hat\sigma^2 - \bar{x})]$. Fix any $b$ beyond that bound, with $b > a$; then $U(k) < 0$ for all $k \ge b$.

Conclusion. The function $\ell_{\text{p}}$ is continuous on $[a, b]$, so it attains a maximum there at some $\hat{k}$. It is increasing on $(0, a]$ and decreasing on $[b, \infty)$, so $\ell_{\text{p}}(\hat{k}) \ge \ell_{\text{p}}(a) \ge \ell_{\text{p}}(k)$ for $k < a$ and $\ell_{\text{p}}(\hat{k}) \ge \ell_{\text{p}}(b) \ge \ell_{\text{p}}(k)$ for $k > b$. So $\hat{k}$ maximizes $\ell_{\text{p}}$ over all $k > 0$. The maximum is not at an endpoint: $U(a) > 0$ means $\ell_{\text{p}}$ takes larger values just to the right of $a$, and $U(b) < 0$ means it takes larger values just to the left of $b$. At an interior maximum of a differentiable function the derivative is zero, so $U(\hat{k}) = 0$. $\square$

The Iowa deaths satisfy the condition, with $\hat\sigma^2 = 2{,}189.64$ against $\bar{x} = 570.52$. For them the bound of the proof is $2B/[n(\hat\sigma^2 - \bar{x})] = 7.78 \times 10^{4}$, far to the right of the root at 203.49. The bound is crude, and the theorem needs only that it be finite.

The proof gave an upper bound for $U$. A lower bound of the same form turns it into a limit. Two more elementary inequalities hold for $u \ge 0$: $\frac{1}{1 + u} \ge 1 - u$, because $(1 + u)(1 - u) = 1 - u^2 \le 1$; and $\ln(1 + u) \le u - \frac{u^2}{2} + \frac{u^3}{3}$, because the difference of the two sides is 0 at $u = 0$ and has derivative $\frac{1}{1 + u} - 1 + u - u^2 = -\frac{u^3}{1 + u} \le 0$. With $u = j/k$ and $u = \bar{x}/k$, and the same sums as in the proof,

$$U(k) \ge \frac{n\bar{x}}{k} - \frac{\sum x_i(x_i - 1)}{2k^2} - \frac{n\bar{x}}{k} + \frac{n\bar{x}^2}{2k^2} - \frac{n\bar{x}^3}{3k^3} = -\frac{n(\hat\sigma^2 - \bar{x})}{2k^2} - \frac{n\bar{x}^3}{3k^3}.$$

Together with the upper bound, and for any non-negative whole numbers whatever their mean and variance,

$$-\frac{n}{2}(\hat\sigma^2 - \bar{x}) - \frac{n\bar{x}^3}{3k} \;\le\; k^2U(k) \;\le\; -\frac{n}{2}(\hat\sigma^2 - \bar{x}) + \frac{B}{k}.$$

So $k^2U(k)$ tends to $-\frac{n}{2}(\hat\sigma^2 - \bar{x})$ as $k \to \infty$: unless mean and variance are equal, the sign of the score far to the right is the sign of mean minus variance. For the Iowa deaths the limit is $-253{,}393$, and the script finds $k^2U(k) = -81{,}344$, $-221{,}930$ and $-249{,}988$ at $k = 10^3$, $10^4$ and $10^5$.

Two things are not proved here. The first is uniqueness: that $U$ has only one root. Levin and Reeds (1977) prove unimodality results of this kind for the negative binomial likelihood. For the data of this article the script confirms it numerically: on a grid of 2,000 values of $k$ from 0.01 to $10^6$, $U$ changes sign exactly once. The second is the converse, that when $\hat\sigma^2 \le \bar{x}$ no finite maximum exists. Part of it is proved here: if $\hat\sigma^2 < \bar{x}$, the lower bound shows $U(k) > 0$ for all $k > 2\bar{x}^3/[3(\bar{x} - \hat\sigma^2)]$, so far to the right the likelihood increases toward the Poisson limit. That $U$ is positive for every $k$, so that there is no maximum anywhere, is not proved here, and neither bound decides the case $\hat\sigma^2 = \bar{x}$. The simulation below met two samples with $\hat\sigma^2 < \bar{x}$ and found $U$ positive throughout the range examined in both.

Word counts

The second place this site meets overdispersed counts is text. In the Cranfield collection of 1,400 aerodynamics abstracts, the stemmed term flutter occurs 282 times, a mean of 0.2014 per document with a variance of 1.2951 (divisor $n$). The 2-Poisson Model shows that a Poisson distribution cannot describe these counts and fits a mixture of two. The negative binomial is the other standard answer, and it is fitted exactly as for the deaths: $\hat\mu = \bar{x}$, and $\hat{k}$ from the score equation.

flutter, 1,400 documents Parameters Maximized $\ell$ Documents with 0 With 6 or more
Observed 1,344 22
Poisson 1 $-1{,}045.86$ 1,144.6 0.0001
Negative binomial, $\hat{k} = 0.0156$ 2 $-389.51$ 1,343.6 15.0
2-Poisson mixture 3 $-367.60$

The estimate $\hat{k} = 0.0156$ is extreme: the variance $\mu + \mu^2/k$ is dominated by its second term. The Poisson model expects 1,144.6 documents without the term and essentially none with six or more occurrences. The negative binomial expects 1,343.6 and 15.0, against the observed 1,344 and 22. The mixture row uses the parameters reported in The 2-Poisson Model, and its log-likelihood is higher still. For the term shock (821 occurrences) the order of the last two reverses: $-2{,}044.41$ for the Poisson, $-1{,}143.68$ for the negative binomial with $\hat{k} = 0.0978$, and $-1{,}177.15$ for the mixture, although the mixture has one more parameter. Of the 1,400 documents, 1,160 have no occurrence of shock; the Poisson model expects 778.8 and the negative binomial 1,157.5. Six or more occurrences are found in 41 documents, and the negative binomial expects 41.6. Models with different numbers of parameters cannot be ranked by maximized likelihood alone, since adding parameters can only raise it for nested models. Prediction Error and Model Choice takes that up.

Maximum likelihood as the minimization of a loss

Most models elsewhere on this site are fitted by minimizing a loss, not by maximizing a likelihood. In the common cases the two are the same computation.

Least squares. Suppose $y_i$ is normal with mean $\beta_0 + \beta_1x_i$ and variance $\sigma^2$, independently for each $i$. Then

$$\ell(\beta_0, \beta_1, \sigma^2) = -\frac{n}{2}\ln(2\pi\sigma^2) - \frac{1}{2\sigma^2}\sum_{i=1}^{n}(y_i - \beta_0 - \beta_1x_i)^2.$$

For every $\sigma^2$, the log-likelihood is largest where the sum of squared residuals is smallest. So the maximum likelihood estimates of the coefficients under normal errors are the least squares estimates, the same two-stage argument as for the normal mean. Linear Models develops least squares itself.

Log loss and cross-entropy. For outcomes $y_i \in \{0, 1\}$ with modeled probabilities $p_i$, the average log loss is $-\ell/n$, as The Likelihood Function showed. Minimizing it is maximum likelihood, and that is how logistic regression is fitted: its likelihood equations have no closed form either, and are solved by the same Newton iteration, in several dimensions. The training loss of a language model, the average of $-\ln P(\text{token} \mid \text{context})$, is $-\ell/n$ for the chain of conditional probabilities. Cross-Entropy and Kullback–Leibler Divergence proves that maximizing the likelihood is the same as minimizing the divergence from the empirical distribution of the data to the model.

Penalties. A loss with a penalty term, or an estimate with added counts, is not a maximum likelihood estimate. Add-one smoothing lowers the likelihood of the training tokens by a factor of 4.42 in the example of the previous article. Such estimates maximize the likelihood times a prior density, or average over a posterior distribution, and belong to Bayesian Inference.

How good is the estimate?

Maximum likelihood is a recipe. Nothing so far says the recipe is good. The examples already show it is not always unbiased ($\hat\sigma^2$, the uniform maximum) and not always sensible in small samples (the zero count). Its justification is its behavior as the sample grows. Under regularity conditions the likelihood equation has a root that converges to the true value and is approximately normal, with the smallest variance any well-behaved estimator can have. That root is the maximum likelihood estimate itself when it is the only root, as in the closed-form models of this article. Fisher Information and the Cramér–Rao Bound and Asymptotics of the Maximum Likelihood Estimator prove those statements and state the conditions.

The conditions matter. Here is a model in which the MLE converges to the wrong value.

Example (Neyman and Scott's pairs). Let $X_{1j}$ and $X_{2j}$, for $j = 1, \ldots, n$, be independent normal variables, the two members of pair $j$ having mean $\mu_j$, and all having variance $\sigma^2$. Each pair has its own mean, so there are $n + 1$ parameters for $2n$ observations. By the two-stage argument above, the likelihood is maximized by $\hat\mu_j = (x_{1j} + x_{2j})/2$ for each pair, which leaves the sum of squares $\sum_j\big[(x_{1j} - \hat\mu_j)^2 + (x_{2j} - \hat\mu_j)^2\big] = \sum_j (x_{1j} - x_{2j})^2/2$, and then by

$$\hat\sigma^2 = \frac{1}{2n}\sum_{j=1}^{n}\frac{(x_{1j} - x_{2j})^2}{2} = \frac{1}{4n}\sum_{j=1}^{n}(x_{1j} - x_{2j})^2.$$

Since $X_{1j} - X_{2j}$ has mean 0 and variance $2\sigma^2$, $E[\hat\sigma^2] = \frac{1}{4n}\cdot n\cdot 2\sigma^2 = \sigma^2/2$ for every $n$. By the law of large numbers, $\hat\sigma^2$ converges to $\sigma^2/2$ as the number of pairs grows. More data do not help, because each new pair brings a new parameter. The script drew 2,000 pairs with $\sigma^2 = 4$ for each of 30 seeds; the estimate had median 2.02 (interquartile range 1.95 to 2.05). Stigler (2007) traces the example to a problem that Abraham Wald described in a letter to Neyman in 1938. Neyman and Scott (1948) published the simplified version given here, and Stigler calls it a warning of what may happen in models with many parameters. It is the bias of $\hat\sigma^2$ from the first section, with $n = 2$ in every pair and no pooling to dilute it.

The sampling distribution of $\hat{k}$. How do the maximum likelihood and moments estimates of the shape compare? The script simulated negative binomial samples from two fitted models and estimated $k$ both ways, with 1,000 samples for each of 30 seeds. Each entry is the median over the seeds, with the interquartile range in parentheses.

Setting True $k$ Median estimate, ML Median estimate, moments Spread, ML Spread, moments
Deaths, $n = 313$ 203.5 205.0 (204.3 to 205.5) 204.3 (203.4 to 204.6) 30.5 (29.9 to 31.5) 30.5 (29.6 to 31.2)
Deaths, $n = 26$ 203.5 222.8 (221.4 to 224.3) 211.2 (209.5 to 212.9) 126.4 (121.1 to 128.8) 117.9 (113.0 to 120.0)
flutter, $n = 1{,}400$ 0.0156 0.0158 (0.0157 to 0.0159) 0.0174 (0.0173 to 0.0176) 0.0036 (0.0035 to 0.0037) 0.0061 (0.0059 to 0.0063)

"Spread" is the interquartile range of the 1,000 estimates within a run. For the deaths, with a large mean and mild overdispersion, the two methods are indistinguishable at $n = 313$. With half a year of data ($n = 26$) both were centered above the true value, maximum likelihood more so, and the moments estimate was, if anything, the less variable. For the word counts, with a tiny mean and extreme overdispersion, the moments estimate was centered about 12% too high and its spread was 1.7 times that of maximum likelihood. These are observations of three settings. They agree with the theory of the next articles in one respect: maximum likelihood was never clearly worse in the large samples, and in one setting it was much better. Among the 30,000 samples of size 26, two had a variance no larger than their mean, and in both the variance was strictly smaller and the score was positive at every $k$ examined (60 values from 0.01 to $10^7$), so no finite estimate was found.

What is proved, what is cited, what the experiment shows

Proved on this page:

  • the maximum likelihood estimates for the binomial, Poisson, normal, exponential and uniform models, each as a global maximum;
  • the bias of $\hat\sigma^2$ and of the uniform maximum;
  • the invariance theorem for an arbitrary function;
  • the unboundedness of the mixture likelihood;
  • $\hat\mu = \bar{x}$ for the negative binomial, the existence of $\hat{k}$ when the variance (divisor $n$) exceeds the mean, and the limit of $k^2U(k)$;
  • the equivalence of least squares and maximum likelihood under normal errors;
  • the limit $\sigma^2/2$ in the Neyman–Scott model.

Cited or left to other articles: the multinomial estimate (Naive Bayes); uniqueness of the negative binomial maximum (Levin and Reeds, 1977, cited here from the paper's abstract); the large-sample properties of the estimator; and the history (Stigler, 2007). Convergence of Newton's method was observed, not proved.

Shown by the experiments: the Newton iterations and the single sign change of the score for these datasets; the fits to the Cranfield terms; and the sampling distributions of the two estimates of $k$ in three settings.

Worked problems

These problems are posed on the data of this unit and on small exact cases. None comes from a course assignment or a textbook. The script named under "In Python" checks every number.

Problem 1 (no events in the sample). Suppose a sample of 40 young adults contained no one with hypertension. Find the maximum likelihood estimate of $p$, the 1/8 likelihood interval, and the largest $p$ under which such a sample has probability at least 0.05. Compare with the rule of three.

Solution. With $x = 0$ the likelihood is $L(p) = (1 - p)^{40}$, decreasing on $[0, 1]$, so $\hat{p} = 0$, on the boundary. The relative likelihood is $L(p)/L(0) = (1 - p)^{40}$. It is at least $1/8$ when $1 - p \ge 8^{-1/40} = 0.9493$, so the 1/8 likelihood interval is $[0,\ 0.0507]$. The sample has probability at least 0.05 when $(1 - p)^{40} \ge 0.05$, that is, when $p \le 1 - 0.05^{1/40} = 0.0722$. The rule of three of Confidence Intervals for a Proportion approximates that bound by $3/n = 0.075$.

The point estimate 0 and the interval say different things. The estimate, read alone, says hypertension does not occur. The interval says the sample is compatible with a prevalence of up to about 5% (at the cutoff 1/8) or 7% (at probability 0.05). This is the gap that smoothing fills in text models. Adding half an event and half a non-event, for instance, gives $(0 + 0.5)/(40 + 1) = 0.0122$, a value inside both intervals and not zero.

Problem 2 (a likelihood with two peaks). The Cauchy density with location $\theta$ is $f(x; \theta) = 1/\big[\pi(1 + (x - \theta)^2)\big]$. Two observations are $-a$ and $a$. Find all solutions of the likelihood equation and the maximum likelihood estimate, for $a = 0.5$ and for $a = 2$.

Solution. The likelihood is $L(\theta) = 1/[\pi^2 D(\theta)]$ with

$$D(\theta) = \big(1 + (\theta - a)^2\big)\big(1 + (\theta + a)^2\big) = (\theta^2 + 1 + a^2)^2 - 4a^2\theta^2,$$

using $(u - v)(u + v) = u^2 - v^2$ with $u = \theta^2 + 1 + a^2$ and $v = 2a\theta$. Maximizing $L$ is minimizing $D$, and

$$D'(\theta) = 4\theta(\theta^2 + 1 + a^2) - 8a^2\theta = 4\theta\,(\theta^2 + 1 - a^2).$$

For $a = 0.5$, the factor $\theta^2 + 1 - a^2 = \theta^2 + 0.75$ is always positive, so $D'$ has the sign of $\theta$: $D$ falls until $\theta = 0$ and rises after. The likelihood equation has the single solution $\theta = 0$, and $\hat\theta = 0$, midway between the observations, with $L(0) = 0.06485$.

For $a = 2$, $D'(\theta) = 4\theta(\theta^2 - 3)$ is zero at $\theta = 0$ and at $\theta = \pm\sqrt{3} = \pm 1.7321$. Its sign is negative for $\theta < -\sqrt{3}$, positive on $(-\sqrt{3}, 0)$, negative on $(0, \sqrt{3})$ and positive beyond. So $D$ has minima at $\pm\sqrt{3}$ and a local maximum at 0, and the likelihood has two peaks of equal height with a dip between them:

$$L(\pm\sqrt{3}) = \frac{1}{16\pi^2} = 0.00633, \qquad L(0) = \frac{1}{25\pi^2} = 0.00405,$$

since $D(\pm\sqrt{3}) = 8^2 - 48 = 16$ and $D(0) = 25$. The likelihood equation has three solutions. The midpoint 0, the only solution when the observations were close together, is now a local minimum, where the likelihood is smaller than at either peak by the factor 1.5625. There are two maximum likelihood estimates, $\sqrt{3}$ and $-\sqrt{3}$. Each sits near one observation and treats the other as an outlier, which the heavy tails of the Cauchy distribution make cheap. The complete answer: one solution and one MLE when $a \le 1$ (at $a = 1$ the three roots coincide at 0), three solutions and two MLEs when $a > 1$. A numerical method started near 0 would stop at a stationary point that is not a maximum.

Problem 3 (correcting the uniform estimate). Three observations, 1.2, 2.4 and 1.3, are modeled as uniform on $[0, \theta]$. Give the maximum likelihood estimate, an unbiased estimate based on it, and the mean squared error of each as a multiple of $\theta^2$.

Solution. The MLE is the largest observation, $\hat\theta = 2.4$. By the article, $E[X_{(n)}] = \frac{n}{n + 1}\theta = 0.75\,\theta$ for $n = 3$, so $\frac{n + 1}{n}X_{(n)} = \frac{4}{3}X_{(n)}$ is unbiased, and its value here is $3.2$.

For the mean squared errors we need the second moment. With the density $nt^{n - 1}/\theta^n$,

$$E[X_{(n)}^2] = \int_0^\theta t^2\cdot\frac{nt^{n - 1}}{\theta^n}\,dt = \frac{n}{n + 2}\theta^2, \qquad \operatorname{Var}(X_{(n)}) = \Big[\frac{n}{n + 2} - \frac{n^2}{(n + 1)^2}\Big]\theta^2 = \frac{n\,\theta^2}{(n + 1)^2(n + 2)}.$$

For $n = 3$ the variance is $\frac{3}{80}\theta^2 = 0.0375\,\theta^2$. The MLE has bias $-\theta/4$, so its mean squared error is variance plus squared bias, $0.0375\,\theta^2 + 0.0625\,\theta^2 = 0.1000\,\theta^2$. The corrected estimator has no bias and variance $\frac{16}{9}\times 0.0375\,\theta^2 = 0.0667\,\theta^2$, which is its mean squared error. Correcting the bias lowers the mean squared error here by a third. Both errors shrink like $1/n^2$, not the $1/n$ of the earlier models: the variance of the maximum, $n\theta^2/[(n + 1)^2(n + 2)]$, is of order $\theta^2/n^2$. That unusually fast rate is a sign that this model lies outside the large-sample theory: the set of possible observations depends on $\theta$, which the regularity conditions of Asymptotics of the Maximum Likelihood Estimator exclude.

Problem 4 (what the fitted negative binomial says about Iowa). Using $\hat\mu = 570.52$ and $\hat{k} = 203.49$, give the maximum likelihood estimates of the standard deviation of a week's deaths, of the standard deviation and coefficient of variation of the underlying weekly rate, and of the probability of a week with 650 or more deaths. Compare the last with the Poisson model and with the data.

Solution. Each quantity is a function of $(\mu, k)$, so by the invariance theorem its MLE is the function evaluated at $(\hat\mu, \hat{k})$.

The variance of a week's count is $\mu + \mu^2/k$, estimated as 2,170.1, a standard deviation of 46.6 deaths. The sample standard deviation is 46.9, and the Poisson model's is $\sqrt{570.52} = 23.9$.

By Overdispersion in Count Data, the rate has variance $\mu^2/k$, estimated as 1,599.5, a standard deviation of 40.0 deaths a week. Its coefficient of variation is $1/\sqrt{k}$, estimated as $1/\sqrt{203.49} = 0.0701$, or 7.0%. That article's moments calculation gave 0.0707.

The probability of 650 or more deaths is one minus the sum of the fitted probabilities of 0 to 649. Under the fitted negative binomial it is 0.0489, about 15.3 of the 313 weeks. Under the Poisson model with mean 570.52 it is 0.0006, about one week in thirty years. The data have 18 such weeks, a share of 0.0575. The negative binomial is close; the Poisson is wrong by two orders of magnitude, which is the practical cost of a model whose variance is too small.

Summary

Result Statement
Maximum likelihood estimate the $\hat\theta$ that maximizes $L(\theta)$, equivalently $\ell(\theta)$
Likelihood equation $U(\theta) = \ell'(\theta) = 0$ at an interior maximum of a differentiable likelihood
Binomial, Poisson $\hat{p} = x/n$, $\hat\lambda = \bar{x}$
Normal $\hat\mu = \bar{x}$, $\hat\sigma^2 = \frac{1}{n}\sum(x_i - \bar{x})^2$, with $E[\hat\sigma^2] = \frac{n - 1}{n}\sigma^2$
Uniform on $[0, \theta]$ $\hat\theta = x_{(n)}$, with $E[\hat\theta] = \frac{n}{n + 1}\theta$
Invariance the MLE of $g(\theta)$ is $g(\hat\theta)$, for any function $g$
Negative binomial $\hat\mu = \bar{x}$; $\hat{k}$ solves $\sum_i\sum_{j < x_i}\frac{1}{k + j} + n\ln\frac{k}{k + \bar{x}} = 0$, and exists if $\hat\sigma^2 > \bar{x}$
Newton's method $k' = k - U(k)/U'(k)$
Failures boundary maxima, jumps, ties, unbounded likelihoods, inconsistent estimates with many parameters

What we have learned

The estimators of the applied course are maximum likelihood estimates or close relatives of them: the sample proportion, the sample mean, and, in place of $s^2$, the variance with divisor $n$, which is biased by the factor $(n - 1)/n$. The estimate of any function of the parameter is the function of the estimate. Differentiation finds the maximum only in regular cases; a zero count, the uniform model, a Cauchy sample and a normal mixture each defeat it in a different way. Where no formula exists, as for the shape of the negative binomial, the score equation is solved by iteration: $\hat{k} = 203.49$ for Iowa's weekly deaths, a maximum that exists because their variance exceeds their mean.

Every closed form in this article depended on the data only through one or two totals: the number of successes, the sum of the counts, the mean and the sum of squares. Sufficient Statistics shows that this is no accident, and that those totals hold everything the sample can say about the parameter.

In Python

import numpy as np
from scipy import optimize, special

def score(k, x):                      # derivative in k of the profile log-likelihood
    return np.sum(special.digamma(x + k) - special.digamma(k)) + len(x) * np.log(k / (k + x.mean()))

x = np.array([...])                   # the 313 weekly counts
k_hat = optimize.brentq(score, 1.0, 1e5, args=(x,))    # about 203.49; mu_hat is x.mean()

special.digamma is the derivative of $\ln\Gamma$; the difference digamma(x + k) - digamma(k) equals the sum $\sum_{j < x} 1/(k + j)$ of the article. scipy.stats.norm.fit and the other fit methods return maximum likelihood estimates, so norm.fit gives the standard deviation with divisor $n$. Every number in this article is computed by tools/experiments/statistics/maximum-likelihood-estimation.py in the site's repository, with the data loaded by ledata.py beside it.

References