A random sample of 40 adults aged 18 to 39 from the National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018 contains 10 with hypertension. This is the sample that Confidence Intervals for a Proportion works with. The proportion of young adults with hypertension in the population, $p$, is unknown to someone who holds only the sample. Which values of $p$ does the sample support, and how strongly does it prefer one value to another? It supports $p = 0.25$ best, and it supports each other value in proportion to the probability that value gives to what was observed: $p = 0.25$ makes 10 in 40 about 7.35 times as probable as $p = 0.40$ does.
The likelihood function is the probability of the observed data, computed under each value of the parameter in turn and read as a function of the parameter. The data are held fixed and the parameter varies. That reverses the usual reading of a probability formula, in which the parameter is fixed and the data vary. This article is the home of the definition for the whole site. The pages on Naive Bayes, language models for retrieval, logistic regression and the bigram language model all maximize a likelihood, and the definitions here are written to serve them.
First we build the likelihood for the 40 adults and tabulate it. Then we separate what a likelihood means from what it does not: it is not a probability distribution for the parameter, and only its ratios carry meaning. Next come the log-likelihood, likelihood intervals, and the likelihood for Poisson and normal data. Finally we write the likelihood for data that are not identically distributed or not independent, which is the form classifiers and language models use, and show why the true parameter value tends to receive the higher likelihood.
The article assumes the binomial, Poisson and normal distributions of the course, and the rules for expected values in Random Variables, Expectation and Variance. Logarithms are natural, written $\ln$.
From a probability to a likelihood
Treat the 40 adults as independent draws from a large population in which a proportion $p$ has hypertension. The number with hypertension, $X$, is then a binomial count, and the probability of the observed value 10 is
$$P_p(X = 10) = \binom{40}{10} p^{10} (1 - p)^{30}.$$
The subscript on $P_p$ says which value of the parameter the probability is computed under. As a statement about data, this formula is used with $p$ known: for $p = 0.25$ it gives the probabilities of 0, 1, ..., 40 hypertensive adults, and those 41 numbers add to 1. The analyst's position is the opposite one. The count 10 is known and $p$ is not. So fix the count at 10 and let $p$ run over its possible values:
$$L(p) = \binom{40}{10} p^{10} (1 - p)^{30}, \qquad 0 \le p \le 1.$$
This is the likelihood function of the sample. It is the same formula with the other argument held fixed.
| $p$ | $L(p)$ | $L(p)/L(0.25)$ | $\ln L(p) - \ln L(0.25)$ |
|---|---|---|---|
| 0.05 | 0.0000 | 0.0001 | $-9.003$ |
| 0.10 | 0.0036 | 0.0249 | $-3.693$ |
| 0.15 | 0.0373 | 0.2584 | $-1.353$ |
| 0.20 | 0.1075 | 0.7443 | $-0.295$ |
| 0.2294 | 0.1377 | 0.9539 | $-0.047$ |
| 0.25 | 0.1444 | 1.0000 | 0.000 |
| 0.30 | 0.1128 | 0.7815 | $-0.247$ |
| 0.35 | 0.0571 | 0.3952 | $-0.928$ |
| 0.40 | 0.0196 | 0.1361 | $-1.994$ |
| 0.50 | 0.0008 | 0.0053 | $-5.232$ |
The row for 0.2294 is the true proportion: the sample was drawn from the 1,696 young adults of the survey, 389 of whom have hypertension. The likelihood is largest at $p = 0.25$, the sample proportion; Maximum Likelihood Estimation proves that it must be. The true value has a likelihood 0.9539 times the largest, so the sample barely distinguishes the two. The value 0.40 has a likelihood 0.1361 times the largest, a ratio of 1 to 7.35. The value 0.05 is nearly ruled out.
In general, let the data $\mathbf{x} = (x_1, \ldots, x_n)$ be the observed values of random variables $\mathbf{X} = (X_1, \ldots, X_n)$. A statistical model gives their joint probability mass function (for discrete data) or joint density (for continuous data), $f(\mathbf{x}; \theta)$, for each value of a parameter $\theta$ in a set $\Theta$ called the parameter space. The semicolon separates the data from the parameter, which in this unit is an unknown constant and not a random variable. The likelihood function is
$$L(\theta) = L(\theta; \mathbf{x}) = f(\mathbf{x}; \theta), \qquad \theta \in \Theta,$$
with $\mathbf{x}$ fixed at what was observed. When the observations are independent and identically distributed (i.i.d.), each with mass function or density $f(x; \theta)$, the joint function is a product, and
$$L(\theta) = \prod_{i=1}^{n} f(x_i; \theta).$$
What a likelihood is not
The table invites three misreadings. Each is settled by a short calculation on the 40 adults.
It is not a probability distribution for the parameter
Example (a common misstatement). "The likelihood $L(0.25) = 0.1444$ is the probability that $p$ is 0.25." It is the probability of observing 10 in 40 if $p$ is 0.25. The column $L(p)$ is not a distribution over $p$, and the quickest proof is that its total is not 1. The area under the likelihood is $1/41 = 0.0244$:
$$\int_0^1 \binom{n}{x} p^x (1 - p)^{n - x}\, dp = \frac{1}{n + 1}.$$
Proof. Write $I(x)$ for the integral. Integrate by parts with $u = (1 - p)^{n - x}$ and $dv = p^x\,dp$. For $x < n$ the boundary term $\big[p^{x+1}(1 - p)^{n-x}/(x + 1)\big]_0^1$ is zero, so
$$\int_0^1 p^x (1 - p)^{n - x}\, dp = \frac{n - x}{x + 1} \int_0^1 p^{x + 1} (1 - p)^{n - x - 1}\, dp.$$
Since $\binom{n}{x}\frac{n - x}{x + 1} = \binom{n}{x + 1}$, this says $I(x) = I(x + 1)$. Repeating the step, $I(x) = I(n) = \int_0^1 p^n\,dp = 1/(n + 1)$. $\square$
A function whose total is 0.0244 could be rescaled to total 1. The deeper trouble is that the total depends on how the parameter is written. Describe the same population by its log odds, $\varphi = \ln\frac{p}{1 - p}$, which runs over the whole real line. Every value of $p$ corresponds to one value of $\varphi$, and the likelihood of a value is the same number whichever name the value goes by. The area is not the same. Since $d\varphi = dp/[p(1 - p)]$,
$$\int_{-\infty}^{\infty} L\, d\varphi = \binom{n}{x}\int_0^1 p^{x - 1}(1 - p)^{n - x - 1}\,dp = \frac{n}{x(n - x)}$$
for $0 < x < n$. The last step uses the integral just proved, with $n - 2$ in place of $n$ and $x - 1$ in place of $x$: it gives $\int_0^1 p^{x-1}(1-p)^{n-x-1}dp = 1\big/\big[(n - 1)\binom{n-2}{x-1}\big]$, and $\binom{n}{x} = \frac{n(n-1)}{x(n-x)}\binom{n-2}{x-1}$. For the 40 adults the area is $40/300 = 0.1333$ on the log odds scale against 0.0244 on the scale of $p$. Areas under a likelihood, and so statements such as "the likelihood puts 90% of its mass between two values", have no meaning that survives a change of scale. A probability distribution for a parameter needs something more than the likelihood, a prior distribution, and that is the subject of Bayesian Inference.
Its size alone means nothing
The largest likelihood in the table is 0.1444. Is that small? The question has no answer, because the number depends on how finely the data are described. The sample was drawn one adult at a time, and the outcomes in the order drawn were
0000010100 1110110000 0001001000 0001000000
with 1 for hypertension. Under independent draws, the probability of this exact sequence is $p^{10}(1 - p)^{30}$, with no binomial coefficient, because it is one particular sequence and not the set of all $\binom{40}{10} = 847{,}660{,}528$ sequences with ten ones. At $p = 0.25$ that probability is $1.70 \times 10^{-10}$. The same data, described in more detail, have a likelihood smaller by a factor of 847,660,528, at every value of $p$.
Now suppose the survey team had instead decided to keep drawing adults until the tenth with hypertension appeared, and that this took 40 draws. The data look the same, but the sampling plan differs: the last draw must be a success, and the other nine successes fall anywhere among the first 39 draws. The probability of that outcome is
$$\binom{39}{9} p^{10} (1 - p)^{30},$$
which at $p = 0.25$ is $211{,}915{,}132 \times 1.70 \times 10^{-10} = 0.0361$. Three descriptions give three likelihoods at $p = 0.25$: 0.1444, $1.70 \times 10^{-10}$ and 0.0361. None is the right size, because size is not what a likelihood conveys.
Only ratios carry meaning
What the three descriptions agree on is every ratio. Each likelihood is a constant times $p^{10}(1 - p)^{30}$, and the constant does not involve $p$, so it cancels when two values of $p$ are compared:
$$\frac{L(0.40)}{L(0.25)} = \frac{0.40^{10} \times 0.60^{30}}{0.25^{10} \times 0.75^{30}} = 0.1361$$
under all three. The likelihood ratio $L(\theta_1)/L(\theta_2)$ says how many times more probable the observed data are under $\theta_1$ than under $\theta_2$. It is the quantity with a direct meaning, and it is why likelihoods that differ by a factor free of the parameter are treated as the same likelihood. Many authors define the likelihood only up to such a factor and write $L(p) \propto p^{10}(1 - p)^{30}$.
Three facts make ratios the stable part. Each is proved here in the generality the unit needs.
Theorem (ratios do not depend on the description). Let two likelihoods for the same parameter satisfy $L_2(\theta) = c\,L_1(\theta)$ for all $\theta$, with $c > 0$ not depending on $\theta$. Then $L_2(\theta_1)/L_2(\theta_2) = L_1(\theta_1)/L_1(\theta_2)$ for all $\theta_1, \theta_2$ with $L_1(\theta_2) > 0$.
Proof. $L_2(\theta_1)/L_2(\theta_2) = cL_1(\theta_1)/[cL_1(\theta_2)]$, and $c$ cancels. $\square$
Theorem (ratios do not depend on the name of the parameter). Let $\varphi = g(\theta)$ be a one-to-one function, and let $L^{*}(\varphi) = L(g^{-1}(\varphi))$ be the likelihood written in terms of $\varphi$. Then $L^{*}(g(\theta_1))/L^{*}(g(\theta_2)) = L(\theta_1)/L(\theta_2)$.
Proof. The model assigns the same probability to the data whether a parameter value is called $\theta$ or $g(\theta)$, so $L^{*}(g(\theta)) = L(\theta)$ for every $\theta$, and the ratios agree term by term. $\square$
Theorem (ratios do not depend on the scale of measurement). Let $X$ have density $f(x; \theta)$ on an interval, and let $Y = h(X)$, where $h$ is continuously differentiable on the interval with a derivative that is never zero (so $h$ is strictly increasing or strictly decreasing). Then the likelihood from observing $y = h(x)$ equals the likelihood from observing $x$ times a factor that does not depend on $\theta$.
Proof. By the change of variable formula for densities, proved in Random Variables and Distribution Functions, the density of $Y$ is $f_Y(y; \theta) = f(h^{-1}(y); \theta)\,\big|\tfrac{d}{dy}h^{-1}(y)\big|$. At $y = h(x)$ the first factor is $f(x; \theta)$, and the second depends on $y$ alone. $\square$
So recording blood pressure in kilopascals instead of millimeters of mercury, or analyzing the logarithm of a concentration instead of the concentration, changes every likelihood by the same factor and no likelihood ratio.
The second sampling plan above deserves one more remark. That the two plans give proportional likelihoods is a theorem. That inferences should therefore agree is a position, known as the likelihood principle; Berger and Wolpert (1988) devote a monograph to it. The methods of the applied course do not all obey it. A p-value is a probability of data that were not observed, and it depends on the plan. For the null hypothesis $p = 0.40$ against smaller values, the probability of 10 or fewer successes in 40 draws is 0.0352, while under the second plan the probability of needing 40 or more draws for 10 successes is 0.0205. The likelihood ratio is 0.1361 under both.
The log-likelihood
Likelihoods of large samples are products of many small numbers. The log-likelihood turns the product into a sum:
$$\ell(\theta) = \ln L(\theta) = \sum_{i=1}^{n} \ln f(x_i; \theta)$$
for i.i.d. data. The logarithm is strictly increasing, so $\ell$ and $L$ rise and fall together and peak at the same place. A ratio of likelihoods becomes a difference of log-likelihoods, and a constant factor in $L$ becomes a constant added to $\ell$. For the 40 adults,
$$\ell(p) = \ln\binom{40}{10} + 10 \ln p + 30 \ln(1 - p),$$
with $\ln\binom{40}{10} = 20.5580$. At $p = 0.25$ the other two terms come to $-22.4934$, so $\ell(0.25) = -1.9354$, the logarithm of 0.1444. The constant 20.5580 is usually dropped, since it plays no part in any comparison. The last column of the table is $\ell(p) - \ell(0.25)$, and it is the same with or without the constant.
There are three reasons to work with $\ell$. The first is arithmetic: sums are easier to differentiate than products, which the next article relies on. The second is statistical: $\ell$ is a sum of independent terms, so the laws of large numbers and the central limit theorem apply to it directly, and Asymptotics of the Maximum Likelihood Estimator builds on that. The third is numerical, and the Poisson example below shows it: a likelihood can be too small for a computer to represent while its logarithm is an ordinary number.
Likelihood intervals
A likelihood ratio compares two values. To summarize the whole function, divide by its largest value. The relative likelihood of $\theta$ is $L(\theta)/L(\hat\theta)$, where $\hat\theta$ is the value at which $L$ is largest. It lies between 0 and 1, and it is the third column of the table. A likelihood interval is the set of parameter values whose relative likelihood is at least a chosen cutoff $c$:
$$\{\theta : L(\theta)/L(\hat\theta) \ge c\} = \{\theta : \ell(\hat\theta) - \ell(\theta) \le -\ln c\}.$$
With $c = 1/8$ the interval holds every value under which the data are at least one eighth as probable as under the best-supported value, that is, every value within $\ln 8 = 2.079$ of the top of the log-likelihood. For the 40 adults the interval runs from 0.130 to 0.403, found by solving $\ell(0.25) - \ell(p) = \ln 8$ numerically on each side of 0.25. It contains the true proportion 0.2294.
The choice of cutoff is a convention until the interval is given a frequency interpretation. That interpretation exists. In large samples, twice the drop in log-likelihood at the true value behaves like a chi-squared variable with one degree of freedom, a result that Likelihood Ratio, Wald and Score Tests proves. The 95th percentile of that distribution is the square of the normal quantile, $1.959964^2 = 3.8415$, so the cutoff $c = e^{-3.8415/2} = e^{-1.9207} = 0.1465$ gives an interval with approximate 95% coverage. For the 40 adults it is (0.134, 0.397). The course's intervals from the same sample are (0.116, 0.384) for the Wald interval and (0.142, 0.402) for Wilson's. The likelihood interval, like Wilson's, extends further above 0.25 than below it, following the asymmetry of the curve. The Wald interval is symmetric by construction.
More data sharpen the likelihood. With 100 hypertensive adults in 400, the same proportion, the 1/8 interval is (0.208, 0.296) and the 0.1465 interval is (0.209, 0.294). The width fell by a factor of about 3.1, close to $\sqrt{10}$. The reason is the curvature of $\ell$ at its peak. Two derivatives of $\ell(p) = x\ln p + (n - x)\ln(1 - p)$ give $\ell''(p) = -x/p^2 - (n - x)/(1 - p)^2$, which at $p = x/n$ equals $-n/[\hat{p}(1 - \hat{p})]$. For $\hat{p} = 0.25$ that is $-213.3$ when $n = 40$ and $-2{,}133.3$ when $n = 400$. Near the peak, $\ell(p) \approx \ell(\hat{p}) - \tfrac{1}{2}\,|\ell''(\hat{p})|\,(p - \hat{p})^2$, so the distance at which the log-likelihood has dropped by a fixed amount shrinks like $1/\sqrt{n}$. The quantity $-\ell''(\hat\theta)$ is the observed information, the subject of Fisher Information and the Cramér–Rao Bound.
Poisson data: weekly deaths
The second running dataset of the unit is the number of deaths from cancer in Iowa in each of the 313 weeks of 2014 to 2019, which Overdispersion in Count Data shows to behave as a Poisson count. The counts range from 97 to 160 and total 38,395. Model them as i.i.d. Poisson with mean $\lambda$, so that each has probability $e^{-\lambda}\lambda^{x}/x!$. The likelihood and log-likelihood are
$$L(\lambda) = \prod_{i=1}^{n} \frac{e^{-\lambda}\lambda^{x_i}}{x_i!} = \frac{e^{-n\lambda}\,\lambda^{\sum x_i}}{\prod x_i!}, \qquad \ell(\lambda) = \Big(\sum_{i=1}^{n} x_i\Big)\ln\lambda - n\lambda - \sum_{i=1}^{n}\ln(x_i!).$$
The 313 counts enter only through their total. The last sum does not involve $\lambda$ and drops out of every comparison. Here $n = 313$ and $\sum x_i = 38{,}395$.
The log-likelihood is largest at the sample mean $\lambda = 38{,}395/313 = 122.67$, where it equals $-1{,}188.85$. The likelihood itself is $e^{-1188.85}$, about $10^{-516.3}$. Standard double-precision arithmetic cannot represent a positive number that small, and a program that multiplies the 313 probabilities returns exactly zero. The log-likelihood is an unremarkable number. This is the practical reason every implementation works in logarithms, as Naive Bayes does when it adds log probabilities over the tokens of a document.
Differences in log-likelihood have a closed form, because the factorials cancel:
$$\ell(\lambda) - \ell(\hat\lambda) = \Big(\sum x_i\Big)\ln\frac{\lambda}{\hat\lambda} - n(\lambda - \hat\lambda), \qquad \hat\lambda = 122.67.$$
| $\lambda$ | $\ell(\lambda) - \ell(\hat\lambda)$ | Relative likelihood |
|---|---|---|
| 120 | $-9.21$ | 0.0001 |
| 122 | $-0.57$ | 0.5650 |
| 123 | $-0.14$ | 0.8688 |
| 124 | $-2.25$ | 0.1056 |
| 125 | $-6.85$ | 0.0011 |
Six years of weekly counts pin the mean down tightly: the 1/8 likelihood interval is (121.40, 123.95). A weekly mean of 120 or of 125, each about 2% from the best-supported value, makes the data less than a hundredth as probable.
Continuous data and the normal likelihood
For continuous data the model gives a density, and a density is not a probability. The probability of any exact value is zero. Why may a density stand in for it?
Here is the idea in one sentence. Every measurement is recorded to some finite precision, the probability of a recorded value is close to the density times the width of the recording interval, and that width does not depend on the parameter.
The formal version. Suppose $x$ is recorded to the nearest multiple of $\delta$, so that the recorded value $x$ means that the underlying value lay in $(x - \delta/2,\ x + \delta/2)$. The probability of that event is $\int_{x - \delta/2}^{x + \delta/2} f(u; \theta)\,du$, which for a density that is continuous at $x$ is approximately $f(x; \theta)\,\delta$ when $\delta$ is small. For $n$ independent values the probability of the recorded data is approximately $\delta^n\prod f(x_i; \theta)$. The factor $\delta^n$ is free of $\theta$. By the first theorem above it does not affect any likelihood ratio, and the likelihood is taken to be $\prod f(x_i; \theta)$.
The approximation can be checked. Student's ten patients, used throughout the course's unit on confidence intervals, gained
$$1.2,\ 2.4,\ 1.3,\ 1.3,\ 0.0,\ 1.0,\ 1.8,\ 0.8,\ 4.6,\ 1.4$$
hours of sleep under one drug compared with another, recorded to 0.1 hour. Compare the parameter values $(\mu, \sigma) = (1.0, 1.2)$ with the best-supported values found below. Using the exact probability that each difference falls in its recording interval of width 0.1, the log of the likelihood ratio is $-1.1752$. Using densities, it is $-1.1757$. The two agree to three decimals.
Model the ten differences as i.i.d. normal with mean $\mu$ and variance $\sigma^2$. (On this site the normal distribution is written $N(\mu, \sigma)$, with the standard deviation as the second argument.) Each has density $\frac{1}{\sqrt{2\pi\sigma^2}}\exp\!\big(-\frac{(x - \mu)^2}{2\sigma^2}\big)$, so
$$\ell(\mu, \sigma^2) = -\frac{n}{2}\ln(2\pi\sigma^2) - \frac{1}{2\sigma^2}\sum_{i=1}^{n}(x_i - \mu)^2.$$
The parameter now has two components, and the likelihood is a surface over the $(\mu, \sigma)$ plane. The sum of squares can be split around the sample mean $\bar{x}$:
$$\sum_{i=1}^{n}(x_i - \mu)^2 = \sum_{i=1}^{n}(x_i - \bar{x})^2 + n(\bar{x} - \mu)^2,$$
because the cross term $2(\bar{x} - \mu)\sum(x_i - \bar{x})$ is zero. The data therefore enter the likelihood only through $\bar{x}$ and $\sum(x_i - \bar{x})^2$. For the sleep data $\bar{x} = 1.58$ and $\sum(x_i - \bar{x})^2 = 13.616$. The surface peaks at $\mu = 1.58$ and $\sigma^2 = 13.616/10 = 1.3616$ (so $\sigma = 1.1669$), where $\ell = -15.733$; the next article derives those values. Some comparisons:
| $(\mu, \sigma^2)$ | $\ell - \ell_{\max}$ | Relative likelihood |
|---|---|---|
| (1.0, 1.44) | $-1.176$ | 0.3086 |
| (0.0, 1.44) | $-8.676$ | 0.0002 |
| (1.58, 4.0) | $-2.090$ | 0.1237 |
The profile likelihood
Often one component is of interest and the other is not. For the sleep data the question is whether the mean difference $\mu$ is zero, and $\sigma^2$ is a nuisance parameter. A likelihood for $\mu$ alone is obtained by giving each $\mu$ the benefit of the doubt: the profile likelihood of $\mu$ is the largest value of the likelihood over $\sigma^2$ with $\mu$ held fixed,
$$L_{\text{p}}(\mu) = \max_{\sigma^2 > 0} L(\mu, \sigma^2).$$
For fixed $\mu$, write $S(\mu) = \sum(x_i - \mu)^2$. Then $\ell = -\frac{n}{2}\ln(2\pi\sigma^2) - S(\mu)/(2\sigma^2)$, and its derivative in $\sigma^2$ is $-\frac{n}{2\sigma^2} + \frac{S(\mu)}{2\sigma^4}$. That is positive for $\sigma^2 < S(\mu)/n$ and negative for $\sigma^2 > S(\mu)/n$, so the maximum is at $\sigma^2 = S(\mu)/n$. This is the dashed curve of the figure. Substituting,
$$\ln L_{\text{p}}(\mu) = -\frac{n}{2}\ln\!\Big(\frac{2\pi S(\mu)}{n}\Big) - \frac{n}{2}.$$
The relative profile likelihood is then a ratio of sums of squares. With the split of $S(\mu)$ above,
$$\frac{L_{\text{p}}(\mu)}{L_{\text{p}}(\bar{x})} = \Big(\frac{S(\mu)}{S(\bar{x})}\Big)^{-n/2} = \Big(1 + \frac{n(\bar{x} - \mu)^2}{\sum(x_i - \bar{x})^2}\Big)^{-n/2} = \Big(1 + \frac{t^2}{n - 1}\Big)^{-n/2},$$
where $t = (\bar{x} - \mu)/(s/\sqrt{n})$ is the one-sample t statistic of The Chi-Squared and t Distributions, with $s^2 = \sum(x_i - \bar{x})^2/(n - 1)$. The profile likelihood of a normal mean is a decreasing function of $|t|$. Ranking values of $\mu$ by profile likelihood and ranking them by the t statistic are the same ranking.
For the sleep data and $\mu = 0$: $s = 1.2300$ and $t = 4.062$, so $1 + t^2/9 = 2.8334$ and the relative profile likelihood is $2.8334^{-5} = 0.0055$. With $\mu$ fixed at zero the best $\sigma^2$ is 3.8580, nearly three times the unrestricted 1.3616, and even so the likelihood is 0.0055 times its value at the peak. The t-test of the course rejects $\mu = 0$ for the same data.
Likelihoods beyond identical, independent observations
The product $\prod f(x_i; \theta)$ needs observations that are independent and share one distribution. Most models elsewhere on this site drop one condition or the other. The definition $L(\theta) = f(\mathbf{x}; \theta)$ covers them all; only the form of the joint probability changes.
Independent but not identically distributed. When the observations are independent and observation $i$ has its own mass function or density $f_i(x; \theta)$, the joint probability is still a product, $L(\theta) = \prod_i f_i(x_i; \theta)$. In logistic regression the outcome $y_i$ is 1 with probability $p_i$ and 0 with probability $1 - p_i$, where $p_i$ depends on that case's predictors and on the coefficients. One trial's mass function is $p_i^{y_i}(1 - p_i)^{1 - y_i}$, and so
$$\ell = \sum_{i=1}^{n}\big[y_i \ln p_i + (1 - y_i)\ln(1 - p_i)\big].$$
The negative of this sum, divided by $n$, is the log loss of Classification Loss Functions. Minimizing average log loss and maximizing the likelihood are the same operation.
Dependent observations. For any random variables, the multiplication rule of probability gives the joint mass function as a chain of conditional ones:
$$f(x_1, \ldots, x_n; \theta) = f(x_1; \theta)\,f(x_2 \mid x_1; \theta)\cdots f(x_n \mid x_1, \ldots, x_{n-1}; \theta).$$
This holds with no assumption at all. A model for dependent data is a rule for the conditional factors. The bigram language model assumes each token depends on the past only through the token before it, so the $i$th factor is $P(x_i \mid x_{i-1})$ and the log-likelihood of a text is $\sum_i \ln P(x_i \mid x_{i-1})$. Larger language models keep the chain and let each factor depend on a longer context. Their training loss, the average of $-\ln P(x_i \mid \text{context})$ over tokens, is the negative log-likelihood per token.
Counts over a vocabulary. Naive Bayes and the query likelihood model of Language Models for Retrieval treat the tokens of a text as independent draws from one distribution over the vocabulary, with probability $\theta_t$ for term $t$. If term $t$ occurs $T_t$ times, the probability of the token sequence is
$$L(\theta) = \prod_{t} \theta_t^{T_t}, \qquad \ell(\theta) = \sum_t T_t \ln\theta_t,$$
a likelihood with one parameter per term, constrained to sum to 1. The textbook example of Naive Bayes has eight training tokens in the class China: Chinese five times and Beijing, Shanghai and Macao once each, in a vocabulary that also contains Tokyo and Japan. At the relative frequencies $5/8, 1/8, 1/8, 1/8, 0, 0$ the likelihood is $(5/8)^5(1/8)^3 = 1.86 \times 10^{-4}$. At the add-one estimates $6/14, 2/14, 2/14, 2/14, 1/14, 1/14$ it is $4.22 \times 10^{-5}$, smaller by a factor of 4.42. Smoothing lowers the likelihood of the training tokens. It is adopted for a reason the likelihood of the training data cannot see: the relative frequencies give probability zero to any future document that contains Tokyo.
The average negative log-likelihood per token is $1.0735$ nats at the relative frequencies and $1.2593$ at the add-one estimates. Cross-Entropy and Kullback–Leibler Divergence shows that this average is a cross-entropy, and that its exponential, here $e^{1.0735} = 2.926$, is the perplexity that language model pages report.
Why the true value tends to have the higher likelihood
The likelihood is used to rank parameter values. What guarantees that the ranking favors the truth? In one sample, nothing does: the 40 adults give $p = 0.25$ a higher likelihood than the true 0.2294. On average, in the logarithm, the truth wins.
Take one Bernoulli trial with true success probability $p_0$, and compute the expected log-likelihood of an arbitrary value $p$:
$$E_{p_0}[\ln f(X; p)] = p_0 \ln p + (1 - p_0)\ln(1 - p).$$
Its derivative in $p$ is $\frac{p_0}{p} - \frac{1 - p_0}{1 - p} = \frac{p_0 - p}{p(1 - p)}$, positive for $p < p_0$ and negative for $p > p_0$. So the expected log-likelihood is uniquely largest at $p = p_0$. With $p_0 = 0.2294$:
| $p$ | 0.10 | 0.20 | 0.2294 | 0.25 | 0.30 | 0.40 |
|---|---|---|---|---|---|---|
| $E_{p_0}[\ln f(X; p)]$ | $-0.6093$ | $-0.5411$ | $-0.5385$ | $-0.5397$ | $-0.5510$ | $-0.6038$ |
The same is true of every model, discrete or continuous.
Theorem (the expected log-likelihood is largest at the true value). Let $X$ have mass function or density $f(x; \theta_0)$, with $E_{\theta_0}\big[\,|\ln f(X; \theta_0)|\,\big] < \infty$, and let $\theta$ be any other parameter value. Then
$$E_{\theta_0}[\ln f(X; \theta)] \le E_{\theta_0}[\ln f(X; \theta_0)],$$
where the left side may be $-\infty$. Equality holds only if $f(X; \theta) = f(X; \theta_0)$ with probability 1 under $\theta_0$, in which case $\theta$ and $\theta_0$ give the same distribution.
Here is the idea in one sentence. The ratio $f(X; \theta)/f(X; \theta_0)$ has mean at most 1 under $\theta_0$, because $f(\cdot\,; \theta)$ has total mass 1, and the logarithm of a quantity with mean at most 1 has mean at most 0.
Proof. Under $\theta_0$ the event $f(X; \theta_0) > 0$ has probability 1, so the ratio $Y = f(X; \theta)/f(X; \theta_0)$ is defined with probability 1, and $Y \ge 0$. For discrete data its mean is
$$E_{\theta_0}[Y] = \sum_{x:\,f(x; \theta_0) > 0} f(x; \theta_0)\,\frac{f(x; \theta)}{f(x; \theta_0)} = \sum_{x:\,f(x; \theta_0) > 0} f(x; \theta) \le 1,$$
and for densities the sums are integrals over the set where $f(x; \theta_0) > 0$. Next, $\ln y \le y - 1$ for every $y \ge 0$, with equality only at $y = 1$ (and $\ln 0 = -\infty$): the function $y - 1 - \ln y$ has derivative $1 - 1/y$, negative on $(0, 1)$ and positive beyond, so its smallest value is 0, at $y = 1$. Hence $\ln Y \le Y - 1$. In particular the positive part of $\ln Y$ is at most $Y$, which has a finite mean, so $E_{\theta_0}[\ln Y]$ is defined, as a number or as $-\infty$, and
$$E_{\theta_0}[\ln Y] \le E_{\theta_0}[Y] - 1 \le 0.$$
Since $\ln f(X; \theta) = \ln Y + \ln f(X; \theta_0)$ and the last term has a finite mean, the inequality of the theorem follows. Now suppose equality holds, so that $E_{\theta_0}[\ln Y] = 0$. The variable $Y - 1 - \ln Y$ is non-negative, and its mean is $E_{\theta_0}[Y] - 1 - 0 \le 0$. A non-negative variable cannot have a negative mean, so the mean is exactly zero, and a non-negative variable with mean zero is zero with probability 1 (the lemma that opens Probability Inequalities). So $Y = 1$ with probability 1: $f(X; \theta) = f(X; \theta_0)$. Then $f(\cdot\,; \theta)$ agrees with $f(\cdot\,; \theta_0)$ on a set that carries all the probability under $\theta_0$, and there its total mass is $E_{\theta_0}[Y] = 1$, which leaves none for anywhere else. The two distributions are the same. $\square$
In a model where different parameter values give different distributions, the true value is therefore the unique maximizer of the expected log-likelihood. The result is known as Gibbs' inequality. Cross-Entropy and Kullback–Leibler Divergence proves it for a finite alphabet, where the gap between the two sides is the Kullback–Leibler divergence, and the argument through Jensen's inequality given there leads to the same place. The log-likelihood of a sample of $n$ is a sum of $n$ such terms, so by the law of large numbers $\ell(\theta)/n$ settles near its expected value, which is largest at the truth. Turning that into a proof that the maximizer of the likelihood converges to the true value needs more conditions, and that result is cited, not proved, in Asymptotics of the Maximum Likelihood Estimator. That article proves, by a local argument that does not use this inequality, that the likelihood equation has a root converging to the true value. The root is the maximizer when it is the only one or when the log-likelihood is strictly concave.
In samples of 40 the advantage is small. The log-likelihood at 0.2294 exceeds the log-likelihood at 0.25 exactly when the count is 9 or fewer, an event of probability 0.5620 under the true value. The script drew 10,000 binomial samples of 40 for each of 30 seeds, and the share in which the true value had the higher likelihood had median 0.5614 (interquartile range 0.5573 to 0.5646). The truth beats a nearby wrong value in a little over half of the samples of this size. The expected log-likelihoods differ by $40 \times (0.5397 - 0.5385) = 0.05$ in a sample of 40, and by ten times that in a sample of 400.
Likelihood in Bayes' rule and in diagnostic tests
Two other uses of the word on this site are the same object.
In Bayes' rule, the posterior probability of a hypothesis is proportional to its prior probability times the probability of the data under it. The second factor, read across hypotheses, is the likelihood. Naive Bayes scores a class $c$ by $P(c)\prod_k P(t_k \mid c)$, a prior times the likelihood of the document's tokens under the class. The factor free of the hypothesis, there $P(d)$, is dropped for the reason given above. Bayesian Inference develops the same rule for a continuous parameter.
In Likelihood Ratios for Diagnostic Tests, the parameter has two values, diseased and not diseased, and the datum is a test result. The positive likelihood ratio, sensitivity divided by one minus specificity, is $P(\text{positive} \mid \text{diseased})/P(\text{positive} \mid \text{not diseased})$: the ratio $L(\theta_1)/L(\theta_2)$ of this article for a parameter space of two points. The Neyman–Pearson Lemma proves that a test based on this ratio is the most powerful one for choosing between two such values.
Where the idea came from
Maximizing the probability of the observations is old. Stigler (2007) traces it to Lambert and to Daniel Bernoulli in the eighteenth century and to Gauss's 1809 derivation of least squares, where it appears as the most probable value under a uniform prior distribution. What those treatments lacked, in Stigler's account, was a reasoned defense: the value was called most probable because it maximized the only probability in sight. He also records the objection Karl Pearson put to the young R. A. Fisher in 1916, drawing on a footnote by Kirstine Smith: for continuous data the quantity maximized is a density and not a probability. Fisher answered with a theory. He first presented the numerical procedure in 1912 and introduced the method of maximum likelihood in his paper of 1922 (Aldrich, 1997). The name is the paper's own. Its sixth section, on the solution of problems of estimation, says: "Such a method is, I believe, provided by the Method of Maximum Likelihood, although I am not satisfied as to the mathematical rigour of any proof which I can put forward to that effect" (Fisher, 1922, p. 323). The paper opens with a list of definitions, and the one for likelihood keeps it apart from probability: "The likelihood that any parameter (or set of parameters) should have any assigned value (or set of values) is proportional to the probability that if this were so, the totality of observations should be that observed" (Fisher, 1922, p. 310). Aldrich's account lists likelihood, sufficiency, efficiency and information among the concepts Fisher developed around the method in those ten years. The first of them, likelihood as something distinct from probability, is the point of the section "What a likelihood is not" above. The other three are the subjects of Sufficient Statistics and Fisher Information and the Cramér–Rao Bound.
What is proved, what is cited, what the experiment shows
Proved on this page:
- the area under the binomial likelihood on the scales of $p$ and of the log odds;
- that likelihood ratios are unchanged by a constant factor, by a one-to-one change of parameter and by a monotone change of measurement scale;
- the curvature of the binomial log-likelihood;
- the reduction of the normal likelihood to $\bar{x}$ and $\sum(x_i - \bar{x})^2$;
- the profile likelihood of a normal mean and its identity with the t statistic;
- the maximization of the expected log-likelihood at the true value, for the Bernoulli model by calculus and for any model with mass functions or densities, with its case of equality.
Cited or left to other articles: the change of variable formula for densities; the chi-squared calibration of the cutoff 0.1465, which is a large-sample result; and the history. The replacement of a probability by a density was argued for small recording intervals and continuous densities and checked on the sleep data. It was not proved as a limit theorem.
The seeded experiment shows one thing: in binomial samples of 40 at $p_0 = 0.2294$, the true value had a higher likelihood than 0.25 in about 56% of samples, in agreement with the exact probability 0.5620.
Worked problems
These problems are posed on the data of this unit and of the course pages it builds on. None comes from a course assignment or a textbook. The script named under "In Python" checks every number.
Problem 1 (the likelihood of the horse-kick rate). Bortkiewicz's table for ten Prussian army corps over 20 years has 200 corps-years with 122 deaths from horse kicks (The Poisson Distribution). Model the 200 counts as i.i.d. Poisson with mean $\lambda$. Write the log-likelihood, compare $\lambda = 0.5$ and $\lambda = 0.75$ with $\lambda = 0.61$, and find the 1/8 likelihood interval.
Solution. With $n = 200$ and $\sum x_i = 122$, the log-likelihood is $\ell(\lambda) = 122\ln\lambda - 200\lambda - \sum\ln(x_i!)$. The sample mean is $122/200 = 0.61$, and the difference from its log-likelihood is
$$\ell(\lambda) - \ell(0.61) = 122\ln\frac{\lambda}{0.61} - 200(\lambda - 0.61).$$
For $\lambda = 0.5$: $122\ln(0.5/0.61) = -24.260$ and $-200(0.5 - 0.61) = 22.0$, so the difference is $-2.260$ and the relative likelihood is $e^{-2.260} = 0.1044$. For $\lambda = 0.75$: $122\ln(0.75/0.61) = 25.207$ and $-200(0.75 - 0.61) = -28.0$, so the difference is $-2.793$ and the relative likelihood is 0.0612. The data are about ten times as probable under 0.61 as under 0.5, and about sixteen times as probable as under 0.75. The 1/8 interval is the set where the difference is at least $-\ln 8 = -2.079$. Solving numerically on each side of 0.61 gives (0.504, 0.730). The value 0.5 lies just outside it and 0.75 further outside, as the two relative likelihoods, both below 0.125, already showed.
Problem 2 (combining two samples). Suppose a second, independent sample of 60 young adults from the same population contained 13 with hypertension. Show that the log-likelihood of the two samples together is the sum of their log-likelihoods. Find where it peaks, and give the 1/8 interval.
Solution. The samples are independent, so the probability of both is the product of their probabilities, and the logarithm of a product is a sum:
$$\ell_{12}(p) = \big[10\ln p + 30\ln(1 - p)\big] + \big[13\ln p + 47\ln(1 - p)\big] = 23\ln p + 77\ln(1 - p),$$
dropping the two binomial coefficients. This is the log-likelihood of a single sample of 100 with 23 successes. Evidence from independent samples adds on the log-likelihood scale:
| $p$ | $\ell_1(p)$ | $\ell_2(p)$ | $\ell_{12}(p)$ | Relative likelihood, combined |
|---|---|---|---|---|
| 0.20 | $-22.789$ | $-31.410$ | $-54.199$ | 0.7622 |
| 0.25 | $-22.493$ | $-31.543$ | $-54.036$ | 0.8970 |
| 0.30 | $-22.740$ | $-32.415$ | $-55.155$ | 0.2930 |
The derivative $23/p - 77/(1 - p) = (23 - 100p)/[p(1 - p)]$ is positive below $p = 23/100 = 0.23$ and negative above it, so $\ell_{12}$ peaks there, at the value $-53.928$. The first sample alone peaked at 0.25 and the second at $13/60$; together they peak at the pooled proportion. The 1/8 interval for the combined sample is (0.152, 0.322), against (0.130, 0.403) for the first sample alone. The value 0.30 had relative likelihood 0.7815 in the first sample and has 0.2930 after the second.
Problem 3 (the odds scale). Express the comparison of $p = 0.40$ with $p = 0.25$ for the 40 adults, and the 1/8 interval, in terms of the odds $\omega = p/(1 - p)$. Then find the area under the likelihood as a function of $\omega$.
Solution. The odds are a one-to-one function of $p$, with $p = \omega/(1 + \omega)$. The values 0.25 and 0.40 are the odds $0.3333$ and $0.6667$. By the second theorem of the article the likelihood ratio is unchanged: $L^{*}(0.6667)/L^{*}(0.3333) = 0.1361$. The interval transforms endpoint by endpoint, because the set of well-supported values is the same set under either name: $0.130/0.870 = 0.150$ and $0.403/0.597 = 0.676$, so the 1/8 interval for the odds is (0.150, 0.676).
The area does change. Since $dp = d\omega/(1 + \omega)^2$ and $(1 + \omega)^{-2} = (1 - p)^2$, we have $d\omega = dp/(1 - p)^2$ and
$$\int_0^\infty L^{*}(\omega)\,d\omega = \binom{n}{x}\int_0^1 p^{x}(1 - p)^{n - x - 2}\,dp = \binom{n}{x}\cdot\frac{1}{(n - 1)\binom{n - 2}{x}} = \frac{n}{(n - x)(n - x - 1)},$$
using the integral of the article with $n - 2$ in place of $n$, and $\binom{n}{x} = \frac{n(n - 1)}{(n - x)(n - x - 1)}\binom{n - 2}{x}$. For $n = 40$ and $x = 10$ this is $40/(30 \times 29) = 0.0460$. The three scales give areas 0.0244 for $p$, 0.1333 for the log odds and 0.0460 for the odds, and one likelihood ratio, 0.1361.
Problem 4 (a likelihood with a jump). Suppose three observations 1.2, 2.4 and 1.3 are modeled as i.i.d. uniform on the interval from 0 to $\theta$. Find the likelihood function and the 1/8 likelihood interval.
Solution. The uniform density is $1/\theta$ for $0 \le x \le \theta$ and 0 otherwise. The joint density is $\theta^{-3}$ if all three observations lie in $[0, \theta]$, that is, if $\theta \ge 2.4$, and 0 if $\theta < 2.4$, because a value of $\theta$ below the largest observation makes that observation impossible:
$$L(\theta) = \begin{cases} 0, & \theta < 2.4, \\ \theta^{-3}, & \theta \ge 2.4. \end{cases}$$
So $L(2.0) = 0$, $L(2.4) = 0.0723$, $L(3.0) = 0.0370$ and $L(4.0) = 0.0156$. The likelihood jumps from zero to its largest value at $\theta = 2.4$ and decreases from there. The relative likelihood of $\theta \ge 2.4$ is $(2.4/\theta)^3$, for example $0.512$ at $\theta = 3$. It is at least 1/8 when $2.4/\theta \ge 1/2$, so the 1/8 interval is $[2.4,\ 4.8]$. The likelihood here is not differentiable at its peak, has no curvature there, and the quadratic approximation of the section on likelihood intervals does not apply. Maximum Likelihood Estimation returns to this model as its standard example of a maximum that calculus does not find.
Summary
| Quantity | Definition or result |
|---|---|
| Likelihood | $L(\theta) = f(\mathbf{x}; \theta)$ with the data fixed; $\prod f(x_i; \theta)$ for i.i.d. data |
| Log-likelihood | $\ell(\theta) = \ln L(\theta) = \sum \ln f(x_i; \theta)$ |
| Likelihood ratio | $L(\theta_1)/L(\theta_2)$; unchanged by constant factors, renaming the parameter, rescaling the data |
| Relative likelihood | $L(\theta)/L(\hat\theta)$, between 0 and 1 |
| Likelihood interval | $\{\theta : \ell(\hat\theta) - \ell(\theta) \le -\ln c\}$; $c = 0.1465$ for approximate 95% coverage |
| Binomial | $\ell(p) = x\ln p + (n - x)\ln(1 - p)$; area under $L$ is $1/(n + 1)$ |
| Poisson | $\ell(\lambda) = (\sum x_i)\ln\lambda - n\lambda$ |
| Normal | $\ell(\mu, \sigma^2) = -\frac{n}{2}\ln(2\pi\sigma^2) - \frac{1}{2\sigma^2}\sum(x_i - \mu)^2$ |
| Profile likelihood of a normal mean | relative value $(1 + t^2/(n - 1))^{-n/2}$ |
| Expected log-likelihood | $E_{\theta_0}[\ln f(X; \theta)] \le E_{\theta_0}[\ln f(X; \theta_0)]$, with equality only for the same distribution |
| Dependent data | $L = \prod_i f(x_i \mid x_1, \ldots, x_{i-1}; \theta)$ |
What we have learned
The likelihood function is the probability of the observed data as a function of the parameter. For 10 hypertensive adults in 40 it peaks at 0.25, gives the true value 0.2294 a relative likelihood of 0.9539, and gives 0.40 a relative likelihood of 0.1361. It is not a distribution for the parameter: its area is 0.0244, 0.1333 or 0.0460 depending on the scale. Its size depends on how the data are described. Its ratios depend on neither, and they are what the log-likelihood, likelihood intervals and the profile likelihood are built from. The same definition gives the product over cases that logistic regression maximizes and the chain of conditional probabilities that a language model is trained on.
The article has used the value at which the likelihood peaks without saying how to find it. Maximum Likelihood Estimation takes that value as an estimate of the parameter, derives it for the models of the course, and treats the cases where no formula exists.
In Python
import numpy as np
from scipy import stats
p = np.array([0.2294, 0.25, 0.40])
logL = stats.binom.logpmf(10, 40, p) # log-likelihood of 10 in 40 at each p
print(np.exp(logL - logL[1])) # relative likelihoods: about 0.9539, 1, 0.1361
logpmf and logpdf return logarithms directly and should be preferred to taking the logarithm of pmf, which underflows for large samples. Every number in this article is computed by tools/experiments/statistics/likelihood-function.py in the site's repository, with the data loaded by ledata.py beside it.
References
- Aldrich, J. (1997). R. A. Fisher and the making of maximum likelihood 1912–1922. Statistical Science, 12(3), 162–176. https://doi.org/10.1214/ss/1030037906 record · 3 pages
- Berger, J. O., and Wolpert, R. L. (1988). The Likelihood Principle. Institute of Mathematical Statistics Lecture Notes, Monograph Series. https://doi.org/10.1214/lnms/1215466210 record
- Fisher, R. A. (1922). On the mathematical foundations of theoretical statistics. Philosophical Transactions of the Royal Society of London, Series A, 222, 309–368. https://doi.org/10.1098/rsta.1922.0009 record · 4 pages
- National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018: DEMO_J, BPX_J and BPQ_J data files and documentation. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 67 pages
- National Center for Health Statistics (2021). Weekly Counts of Deaths by State and Select Causes, 2014–2019. Centers for Disease Control and Prevention, data.cdc.gov dataset 3yf8-kanr. https://data.cdc.gov/NCHS/Weekly-Counts-of-Deaths-by-State-and-Select-Causes/3yf8-kanr record · 8 pages
- Stigler, S. M. (2007). The epic story of maximum likelihood. Statistical Science, 22(4), 598–620. https://doi.org/10.1214/07-STS249 record · 4 pages