Likelihood and Estimation

The theory of estimation and testing that grows from one function, the probability of the observed data read as a function of the parameter. Eight articles, from the definition of the likelihood through maximum likelihood, sufficiency, Fisher information, the large-sample behavior of the estimator, the three classical tests and the EM algorithm, with every result proved in one dimension or cited, on NHANES hypertension counts, NCHS weekly deaths and word counts from the Cranfield collection.

The applied statistics course on this site estimates a proportion by the sample proportion and a mean by the sample mean, builds intervals from standard errors, and tests hypotheses with z, t and chi-square statistics. It rarely says why those choices and not others. This unit supplies the reason. Nearly all of them follow from one object, the likelihood function: the probability of the data that were observed, computed under each possible value of the unknown parameter and read as a function of that parameter.

The unit does three things with the likelihood. It estimates: the parameter value that makes the data most probable is the maximum likelihood estimate, and the estimators of the course turn out to be instances. It measures precision: the curvature of the logarithm of the likelihood at its peak is the Fisher information, which bounds the variance of every unbiased estimator and gives the standard error of the maximum likelihood estimate in large samples. And it tests: ratios of likelihoods give the most powerful tests of one value against another, and the likelihood ratio, Wald and score tests are three large-sample approximations to one another, two of which the course's intervals for a proportion invert.

The same function is the training criterion of most models elsewhere on this site. Naive Bayes and language models for retrieval estimate term probabilities by maximum likelihood and then repair its zeros. Logistic regression maximizes a likelihood by Newton's method. The bigram language model and its successors minimize a negative log-likelihood per token, the log loss of Classification Loss Functions. Mixture models, in model-based clustering and the 2-Poisson model, climb a likelihood with the EM algorithm. The articles here are written to be the common reference for those pages.

What the unit assumes

The unit assumes the applied course through Inference on Proportions: the binomial, Poisson and normal distributions, expectation and variance (Random Variables, Expectation and Variance), confidence intervals and tests. From the Probability Theory unit it takes, by link and without proving them again, the laws of large numbers, the central limit theorem and Slutsky's theorem, the delta method, conditional expectation, and the inequalities of Jensen and Cauchy–Schwarz (Probability Inequalities). Calculus in one variable is used throughout, and partial derivatives where a model has two parameters.

What is proved and what is cited

Results are proved in one dimension, for a single real parameter, under conditions stated on the page where they are used. Proofs that condition on a statistic are written for discrete data, where conditional probabilities are ratios of probabilities; the continuous versions need measure theory and are cited, with one continuous case worked by a limit in Sufficient Statistics. Versions for several parameters are stated and cited. The large-sample theorems are proved for a root of the likelihood equation that converges to the true value, which is the maximum likelihood estimate when the root is unique; consistency of the global maximizer in general is cited (Wald, 1949, whose theorem is stated in Asymptotics of the Maximum Likelihood Estimator). The unit does not prove the general measure-theoretic factorization theorem, completeness beyond the binomial and Poisson families, or the convergence of the numerical methods it uses, apart from one worked example of the EM algorithm, and each article says so where the matter arises. The history told in the articles is taken from Stigler (2007) and from the abstract of Aldrich (1997), both of which describe Fisher (1922), the paper that, in the words of Aldrich's abstract, "introduced the method of maximum likelihood". The articles quote Fisher's paper itself for the name of the method and for its definitions of likelihood and sufficiency.

Three datasets run through the unit. The first is the sample of 40 young adults from NHANES 2017 to 2018, 10 of them with hypertension, that the course's article on intervals for a proportion works with (binomial). The second is the weekly number of deaths in Iowa from 2014 to 2019: deaths from cancer, which fit a Poisson model, and deaths from all causes, which need the negative binomial, whose estimate has no closed form. The third is the number of occurrences of a term in each of the 1,400 abstracts of the Cranfield collection. Every number is computed by a script in tools/experiments/statistics/ in the site's repository, and the scripts assert each value the articles show.

Articles

In reading order:

  • The Likelihood Function: what a likelihood is and is not. Only its ratios carry meaning, which is proved for sampling plans, for changes of parameter and for changes of measurement scale; the log-likelihood, likelihood intervals and the profile likelihood; and the forms the likelihood takes for classifiers and language models.
  • Maximum Likelihood Estimation: the estimators of the course rederived as maximum likelihood estimates; the invariance property proved for any function of the parameter; boundary maxima, likelihoods with several peaks and with none; and the negative binomial fitted by Newton's method, with a proof that its maximum exists when the variance exceeds the mean.
  • Sufficient Statistics: when a sample can be replaced by a summary with nothing lost. The factorization theorem and the criterion for minimal sufficiency proved for discrete data, the Rao–Blackwell theorem proved, completeness proved for the binomial and Poisson families, and the Lehmann–Scheffé theorem on best unbiased estimators.
  • Fisher Information and the Cramér–Rao Bound: the score and its mean of zero, the two forms of the information, and the lower bound on the variance of an unbiased estimator, proved by the Cauchy–Schwarz inequality.
  • Asymptotics of the Maximum Likelihood Estimator: a consistent root of the likelihood equation and its asymptotic normality, proved in one dimension under stated regularity conditions, with the Wald interval of the course as a corollary.
  • The Neyman–Pearson Lemma: the most powerful test of one simple hypothesis against another is a likelihood ratio test, proved with the randomized rule that discrete data need and the sense in which the best test is unique; uniformly most powerful one-sided tests under a monotone likelihood ratio; and a proof that no uniformly most powerful two-sided test of a normal mean exists.
  • Likelihood Ratio, Wald and Score Tests: the three tests on one example, Wilson's interval as an inverted score test, and Wilks's theorem on the chi-squared limit of the likelihood ratio statistic, proved in one dimension.
  • The EM Algorithm: maximum likelihood when part of the data is missing or hidden; the proof, by Jensen's inequality, that each iteration cannot lower the likelihood, worked on a two-component mixture.

Where the unit leads

Generalized Linear Models fits regression models by the maximum likelihood method of this unit, and Bayesian Inference combines the same likelihood with a prior distribution. Information Theory reads the average negative log-likelihood as a cross-entropy, and Prediction Error and Model Choice asks how to compare maximized likelihoods of models of different size.

References