A random sample of 60 men aged 18 to 39 from the National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018 contains 15 with hypertension (National Center for Health Statistics, 2020). The national figure for adults of those ages is 22.4%. Tests and Sample Size for a Proportion tests $H_0\colon p = 0.224$ with the statistic $z = 0.483$. It then notes that some software divides by a different standard error and reports $0.465$. A third calculation, which compares the likelihood of the sample at $p = 0.224$ with its largest possible value, gives $0.477$ on the same scale. The three $p$-values are 0.629, 0.642 and 0.634. Why are there three answers to one question, and which is right?
All three are legitimate, and in large samples they agree. They are the score test, the Wald test and the likelihood ratio test: three ways of measuring, on the graph of the log-likelihood, how far the null value lies from the best-fitting value. This article defines them for any model with one parameter, shows what each measures, and proves that under the null hypothesis each statistic has, in large samples, the chi-squared distribution with one degree of freedom. The result for the likelihood ratio is Wilks's theorem.
First we read the three statistics off the log-likelihood and compute them for the 60 young men. Then we turn each test into a confidence interval: the score test gives Wilson's interval and the Wald test gives the textbook interval, which is how Confidence Intervals for a Proportion came to have two intervals for one proportion, and the likelihood ratio gives a third. Next we find why the Wald test alone depends on the scale on which the parameter is written. Then come the proofs of the large-sample distributions, the statement for several parameters, and the cases where the approximation fails.
The article assumes the log-likelihood, score and Fisher information as defined in The Likelihood Function and Fisher Information and the Cramér–Rao Bound. The proofs rest on the asymptotic normality of the maximum likelihood estimate, proved in Asymptotics of the Maximum Likelihood Estimator. The Neyman–Pearson Lemma treats the case this article leaves behind, two fully specified hypotheses, where one test is provably best.
Three ways to measure a distance on the log-likelihood
The data are a sample $\mathbf{x} = (x_1, \ldots, x_n)$ from a model with one real parameter $\theta$, and the hypothesis is $H_0\colon \theta = \theta_0$ against $H_1\colon \theta \ne \theta_0$. Four objects from the earlier articles of the unit are needed:
- the log-likelihood $\ell(\theta) = \ln L(\theta) = \sum_i \ln f(x_i; \theta)$;
- the score $U(\theta) = \ell'(\theta)$, its slope;
- the Fisher information of the sample, $I_n(\theta) = n I(\theta)$, where $I(\theta) = E_\theta[u(X; \theta)^2]$ is the information in one observation and $u = \partial \ln f/\partial\theta$; under the regularity conditions it is also the expected value of the curvature $-\ell''(\theta)$;
- the maximum likelihood estimate (MLE) $\hat\theta$, where $\ell$ is highest and, for a smooth log-likelihood with an interior maximum, $U(\hat\theta) = 0$.
If $H_0$ is true, $\hat\theta$ should be near $\theta_0$. There are three natural ways to say how near.
The likelihood ratio statistic measures the vertical distance: how far the log-likelihood falls between its peak and the null value,
$$W_{\mathrm{LR}} = 2\,[\ell(\hat\theta) - \ell(\theta_0)] = -2 \ln \Lambda, \qquad \Lambda = \frac{L(\theta_0)}{L(\hat\theta)}.$$
The ratio $\Lambda$ lies between 0 and 1, and small values count against $H_0$. It is the ratio of the Neyman–Pearson lemma with the unspecified alternative replaced by the value the data like best.
The Wald statistic measures the horizontal distance between the estimate and the null value, in units of the estimate's standard error:
$$W_{\mathrm{W}} = (\hat\theta - \theta_0)^2\, I_n(\hat\theta) = \left(\frac{\hat\theta - \theta_0}{\operatorname{SE}(\hat\theta)}\right)^2, \qquad \operatorname{SE}(\hat\theta) = \frac{1}{\sqrt{I_n(\hat\theta)}}.$$
The standard error is the large-sample one of Asymptotics of the Maximum Likelihood Estimator, evaluated at the estimate.
The score statistic measures the slope of the log-likelihood at the null value. At the peak the slope is zero, so a steep slope at $\theta_0$ means the peak is far away:
$$W_{\mathrm{S}} = \frac{U(\theta_0)^2}{I_n(\theta_0)}.$$
The divisor is the variance of the score under $H_0$ (Fisher Information and the Cramér–Rao Bound proves that the score has mean 0 and variance $I_n(\theta_0)$), so $W_{\mathrm{S}}$ is the squared score in units of its own standard deviation.
Each test rejects $H_0$ at level $\alpha$ when its statistic exceeds $\chi^2_{1, 1-\alpha}$, the point with probability $1 - \alpha$ below it under the chi-squared distribution with one degree of freedom: 3.8415 for $\alpha = 0.05$, which is $1.959964^2$, the square of the normal point $z_{0.975}$ that the course rounds to 1.960. The sections on large samples prove that this is the right reference distribution.
The three statistics need different amounts of computation. The Wald statistic uses only the estimate and never evaluates anything at $\theta_0$ except the difference. The score statistic uses only the null value and never needs the estimate. The likelihood ratio needs the log-likelihood at both.
One picture for all three
There is a way to see the three as the same measurement taken on three different curves. Each statistic is twice a drop in height.
- $W_{\mathrm{LR}}$ is twice the drop of the log-likelihood itself from its peak to $\theta_0$.
- Replace the log-likelihood by the parabola that has the same peak and the curvature $I_n(\hat\theta)$ there, $q_{\mathrm{W}}(\theta) = \ell(\hat\theta) - \tfrac12 I_n(\hat\theta)(\theta - \hat\theta)^2$. Its drop from the peak to $\theta_0$ is $\tfrac12 I_n(\hat\theta)(\hat\theta - \theta_0)^2$, so twice the drop is $W_{\mathrm{W}}$.
- Replace it instead by the parabola that passes through the point $(\theta_0, \ell(\theta_0))$ with the true slope $U(\theta_0)$ and the curvature $I_n(\theta_0)$, $q_{\mathrm{S}}(\theta) = \ell(\theta_0) + U(\theta_0)(\theta - \theta_0) - \tfrac12 I_n(\theta_0)(\theta - \theta_0)^2$. Setting its derivative to zero, it peaks at $\theta_0 + U(\theta_0)/I_n(\theta_0)$, where it is higher than at $\theta_0$ by $U(\theta_0)^2/[2 I_n(\theta_0)]$. Twice that rise is $W_{\mathrm{S}}$.
So the likelihood ratio uses the curve as it is, the Wald statistic uses a parabola fitted at the estimate, and the score statistic uses a parabola fitted at the null value. The three agree to the extent that the log-likelihood is a parabola between $\theta_0$ and $\hat\theta$. Buse (1982) gives an account of the three tests by diagrams of this kind.
The three statistics for a proportion
Back to the young men. The count $X$ of hypertensive men among $n$ is $\text{Bin}(n, p)$, and up to a constant that does not involve $p$,
$$\ell(p) = x \ln p + (n - x) \ln(1 - p).$$
Differentiating once and twice,
$$U(p) = \frac{x}{p} - \frac{n - x}{1 - p} = \frac{x - np}{p(1 - p)} = \frac{n(\hat{p} - p)}{p(1 - p)}, \qquad -\ell''(p) = \frac{x}{p^2} + \frac{n - x}{(1 - p)^2},$$
with $\hat{p} = x/n$. The score is zero at $\hat{p}$, and $\ell'' < 0$ everywhere on $(0, 1)$, so for $0 < x < n$ the log-likelihood is strictly concave with its maximum at $\hat{p}$: the MLE is the sample proportion, as Maximum Likelihood Estimation shows. Taking the expected value of the curvature with $E_p[X] = np$ gives the information, $I_n(p) = n/p + n/(1 - p) = n/[p(1 - p)]$. Substituting into the three definitions:
$$W_{\mathrm{S}} = \frac{n(\hat{p} - p_0)^2}{p_0(1 - p_0)}, \qquad W_{\mathrm{W}} = \frac{n(\hat{p} - p_0)^2}{\hat{p}(1 - \hat{p})}, \qquad W_{\mathrm{LR}} = 2\left[x \ln\frac{\hat{p}}{p_0} + (n - x)\ln\frac{1 - \hat{p}}{1 - p_0}\right].$$
For the score statistic, $U(p_0)^2/I_n(p_0) = [n(\hat{p} - p_0)/(p_0(1 - p_0))]^2 \cdot p_0(1 - p_0)/n$, which simplifies to the form shown. For a proportion $U(p_0)/I_n(p_0) = \hat{p} - p_0$, so the parabola fitted at the null value peaks exactly at $\hat{p}$, as the figure shows, though not at the same height as the log-likelihood.
The first two are familiar. $W_{\mathrm{S}}$ is the square of the course's $z$ statistic, $(\hat{p} - p_0)/\sqrt{p_0(1 - p_0)/n}$, with the standard error computed under the null hypothesis. $W_{\mathrm{W}}$ is the square of the same difference divided by the estimated standard error $\sqrt{\hat{p}(1 - \hat{p})/n}$. The rule of Tests and Sample Size for a Proportion, "for a test use $p_0$, for an interval use $\hat{p}$", is the statement that the course's test is a score test and its textbook interval a Wald interval. The third statistic has no counterpart on the course pages.
For $x = 15$, $n = 60$ and $p_0 = 0.224$: $\ell(\hat{p}) = -33.7401$ and $\ell(p_0) = -33.8538$; the score at the null value is $U(p_0) = 8.9746$; the information is $I_n(p_0) = 345.177$ and $I_n(\hat{p}) = 320.000$.
| Test | Statistic | Square root | $p$-value |
|---|---|---|---|
| Score | $8.9746^2/345.177 = 0.2333$ | 0.4831 | 0.6291 |
| Wald | $0.026^2 \times 320.000 = 0.2163$ | 0.4651 | 0.6419 |
| Likelihood ratio | $2 \times (33.8538 - 33.7401) = 0.2273$ | 0.4768 | 0.6335 |
(Each value is computed from unrounded quantities.) The $p$-values are $P(\chi^2_1 \ge W)$, which equals $P(\lvert Z \rvert \ge \sqrt{W})$ for a standard normal $Z$, because $\chi^2_1$ is by definition the distribution of $Z^2$ (The Chi-Squared and t Distributions). The square roots, given the sign of $\hat{p} - p_0$, are the three numbers of the opening paragraph, and they can be referred to the normal distribution for a one-sided test.
Here the three agree closely, because 0.224 is near 0.25 and the log-likelihood is nearly a parabola over so short a stretch. They separate when the null value is further away:
| $p_0$ | Score | Wald | Likelihood ratio |
|---|---|---|---|
| 0.10 | 15.000 | 7.200 | 11.080 |
| 0.15 | 4.706 | 3.200 | 4.060 |
| 0.224 | 0.233 | 0.216 | 0.227 |
| 0.40 | 5.625 | 7.200 | 5.983 |
| 0.45 | 9.697 | 12.800 | 10.280 |
Two things stand out. The Wald statistic is the same at 0.10 and at 0.40, the two values equally far from $\hat{p} = 0.25$: it is a function of the distance alone, because its standard error is fixed by the sample. The other two are not symmetric. In this table, against null values below $\hat{p}$ the score statistic is the largest of the three and the Wald statistic the smallest; above $\hat{p}$ the order reverses. The reason is the variance $p(1 - p)/n$ of a proportion, which grows as $p$ moves toward one half. A null value of 0.10 implies a small variance, under which an observed 0.25 is very surprising; the Wald statistic ignores this and uses the variance at 0.25 for every null value. In this table the likelihood ratio always lies between the other two.
Example (the NIST wafers). The one-sided manufacturing example that the course page takes from the NIST handbook, 26 defective wafers among 200 against $p_0 = 0.10$, has $z = 0.03/\sqrt{0.1 \times 0.9/200} = \sqrt{2} = 1.414$, so the score statistic is $z^2 = 2$ exactly. The Wald statistic is 1.5915 and the likelihood ratio statistic 1.8452. The score statistic is the largest because the null value lies below the estimate, 0.13.
When the log-likelihood is a parabola
The picture of the three parabolas says the statistics coincide when the log-likelihood is itself a parabola. That happens exactly in the model behind the $z$ test of a mean. Let $X_1, \ldots, X_n$ be independent and normal with mean $\mu$ and known variance $\sigma^2$. Then
$$\ell(\mu) = -\frac{1}{2\sigma^2}\sum_{i=1}^n (x_i - \mu)^2 + \text{constant} = -\frac{n}{2\sigma^2}(\mu - \bar{x})^2 - \frac{1}{2\sigma^2}\sum_{i=1}^n (x_i - \bar{x})^2 + \text{constant},$$
using $\sum(x_i - \mu)^2 = \sum(x_i - \bar{x})^2 + n(\bar{x} - \mu)^2$. The log-likelihood is a parabola in $\mu$ with peak at $\hat\mu = \bar{x}$, score $U(\mu) = n(\bar{x} - \mu)/\sigma^2$ and constant curvature $-\ell''(\mu) = n/\sigma^2 = I_n(\mu)$. Therefore
$$W_{\mathrm{LR}} = 2 \cdot \frac{n}{2\sigma^2}(\bar{x} - \mu_0)^2, \qquad W_{\mathrm{W}} = (\bar{x} - \mu_0)^2 \cdot \frac{n}{\sigma^2}, \qquad W_{\mathrm{S}} = \frac{n^2(\bar{x} - \mu_0)^2/\sigma^4}{n/\sigma^2},$$
and all three equal
$$\frac{n(\bar{x} - \mu_0)^2}{\sigma^2} = z^2, \qquad z = \frac{\bar{x} - \mu_0}{\sigma/\sqrt{n}}.$$
In this model the three tests are one test, the two-sided $z$ test of Hypothesis Tests for a Mean, and the statistic has exactly the $\chi^2_1$ distribution under $H_0$ at every sample size, since $z$ is exactly standard normal. The script checks the identity on the first 25 adults aged 60 and over in the NHANES file, with the null mean $\mu_0 = 126.36$ (the mean of all adults) and $\sigma = 21.02$ (the standard deviation of the adults aged 60 and over), as in The Neyman–Pearson Lemma: their mean is 135.507, $z = 2.1753$, and the three statistics all equal $4.7321$. Buse (1982) makes the same observation in general: when the log-likelihood is quadratic the three statistics are identical.
For every other model the log-likelihood is only approximately a parabola near its peak, and the three tests differ in small samples and agree in large ones. The proofs below make "approximately" precise.
Inverting the tests: three intervals for a proportion
A test of each possible null value yields a confidence interval: collect the values the test does not reject. If the test of every $\theta_0$ has Type I error rate $\alpha$, the collection covers the true value with probability $1 - \alpha$, because the true value is left out exactly when the test of it rejects. In symbols, for the set $C(\mathbf{X}) = \{\theta_0 : \text{the test of } \theta_0 \text{ does not reject}\}$,
$$P_\theta\big(\theta \in C(\mathbf{X})\big) = 1 - P_\theta(\text{the test of } \theta \text{ rejects}),$$
which holds for any family of tests and any sample size. The coverage of the interval at $\theta$ and the actual Type I error rate of the test of $\theta$ always add to 1.
Apply this to the three tests of a proportion, with $z = z_{1-\alpha/2}$ and $z^2 = \chi^2_{1, 1-\alpha}$.
The Wald test gives the Wald interval. The values not rejected satisfy $n(\hat{p} - p)^2/[\hat{p}(1 - \hat{p})] \le z^2$, that is, $\lvert \hat{p} - p \rvert \le z\sqrt{\hat{p}(1 - \hat{p})/n}$. This is the textbook interval $\hat{p} \pm z\sqrt{\hat{p}(1 - \hat{p})/n}$.
The score test gives Wilson's interval. The values not rejected satisfy $n(\hat{p} - p)^2/[p(1 - p)] \le z^2$, that is, $\lvert \hat{p} - p \rvert \le z\sqrt{p(1 - p)/n}$. This is the inequality that Confidence Intervals for a Proportion solves as a quadratic in $p$ to obtain Wilson's (1927) formula. That page calls it the score interval; this is the reason for the name.
The likelihood ratio test gives a third interval. The values not rejected satisfy $2[\ell(\hat{p}) - \ell(p)] \le z^2$: every $p$ whose log-likelihood is within $z^2/2 = 1.9207$ of the maximum. Because $\ell$ is strictly concave, that set is an interval containing $\hat{p}$. It has no closed form, and its limits are found numerically as the two solutions of $\ell(p) = \ell(\hat{p}) - 1.9207$. The Likelihood Function met this interval as the set of values whose likelihood is at least $e^{-1.9207} = 0.1465$ of the maximum, and deferred the reason for that cut-off to this page.
For the sample of that page, 10 hypertensive among 40 young adults, $\ell(\hat{p}) = -22.4934$, and the likelihood ratio limits are the two values of $p$ at which $\ell(p) = -24.4141$.
| 95% interval | Lower | Upper | Width |
|---|---|---|---|
| Wald | 0.1158 | 0.3842 | 0.2684 |
| Wilson (score) | 0.1419 | 0.4019 | 0.2601 |
| Likelihood ratio | 0.1340 | 0.3970 | 0.2630 |
The Wald interval is symmetric about $\hat{p}$, and the other two are shifted toward one half. The figure shows why. At the Wald interval's lower limit, 0.1158, the other two tests reject clearly: the likelihood ratio statistic there is 5.514 and the score statistic 7.034. A proportion of 0.1158 has a small variance, and under it an observed 0.25 would be a larger surprise than the Wald calculation allows. At the upper limit, 0.3842, the other two statistics are 3.235 and 3.044, below 3.8415, so both intervals extend further up.
How do the three compare in coverage? For a proportion this can be computed exactly, as on the course page, by adding the binomial probabilities of the counts whose interval contains $p$. The true proportion among the 1,696 young adults is $389/1{,}696 = 0.2294$.
| 95% interval | Coverage at $p = 0.2294$, $n = 40$ | Lowest, $10 \le n \le 200$ | Mean over those $n$ | Share of $n$ below 0.93 |
|---|---|---|---|---|
| Wald | 0.9109 | 0.8202 (at $n = 13$) | 0.9374 | 18.3% |
| Wilson (score) | 0.9628 | 0.9107 (at $n = 13$) | 0.9505 | 1.0% |
| Likelihood ratio | 0.9414 | 0.9059 (at $n = 18$) | 0.9479 | 2.6% |
The Wald and Wilson rows reproduce the course page's figures. Across 2,000 values of $p$ spread over $(0, 1)$ at $n = 40$, the grid of that page, the mean coverage is 0.8907 for Wald, 0.9520 for Wilson and 0.9476 for the likelihood ratio interval, with mean expected lengths 0.2341, 0.2367 and 0.2350. For $0.1 \le p \le 0.9$ the lowest coverage is 0.8599, 0.9278 and 0.9128. The likelihood ratio interval behaves much more like Wilson's than like the Wald interval: its coverage oscillates around a level just under 0.95, where the Wald interval's sits well below.
The same comparison can be made for the tests. By the identity above, the Type I error rate of each test of $p_0 = 0.224$ is one minus the coverage of its interval at 0.224.
| Test of $p_0 = 0.224$ at 0.05 | Rejects at $n = 60$ when | Actual size, $n = 60$ | Range over $10 \le n \le 200$ | Mean | Share of $n$ above 0.05 |
|---|---|---|---|---|---|
| Score | $x \le 7$ or $x \ge 20$ | 0.0612 | 0.0203 to 0.0705 | 0.0492 | 45.5% |
| Wald | $x \le 8$ or $x \ge 21$ | 0.0749 | 0.0439 to 0.1892 | 0.0634 | 85.9% |
| Likelihood ratio | $x \le 7$ or $x \ge 21$ | 0.0446 | 0.0313 to 0.0958 | 0.0519 | 53.9% |
The score and Wald rows are those of Tests and Sample Size for a Proportion. At $n = 60$ the likelihood ratio test happens to reject at exactly the counts where that page's exact binomial test rejects. The score and likelihood ratio tests hold their level on average and miss it in either direction at particular sample sizes, because a whole-number count cannot give a size of exactly 0.05. The Wald test rejects too often at nearly every sample size.
Why the Wald test depends on how the parameter is written
A proportion can be described by $p$ or equally well by its log-odds $\eta = \ln[p/(1 - p)]$, the scale of logistic regression. The hypothesis $p = 0.224$ is the same hypothesis as $\eta = \ln(0.224/0.776) = -1.2425$. A test of a hypothesis should not depend on the name given to the parameter. Two of the three tests have that property and the Wald test does not.
Theorem (invariance under a change of parameter). Let $\theta = g(\eta)$, where $g$ is differentiable and strictly monotone with $g'(\eta) \ne 0$, and let $\theta_0 = g(\eta_0)$. Write $\ell^*$, $U^*$, $I^*_n$ for the log-likelihood, score and information as functions of $\eta$. Then the likelihood ratio statistic and the score statistic for $H_0\colon \eta = \eta_0$ equal those for $H_0\colon \theta = \theta_0$. The Wald statistic for $\eta$ is
$$W^*_{\mathrm{W}} = (\hat\eta - \eta_0)^2\, [g'(\hat\eta)]^2\, I_n(\hat\theta),$$
which differs from $(\hat\theta - \theta_0)^2 I_n(\hat\theta)$ unless $g(\hat\eta) - g(\eta_0) = g'(\hat\eta)(\hat\eta - \eta_0)$.
Proof. The model under $\eta$ is the model under $\theta = g(\eta)$, so $\ell^*(\eta) = \ell(g(\eta))$. As $\eta$ runs over its range $\theta$ runs over $\Theta$, so the two functions have the same maximum, attained at $\hat\theta = g(\hat\eta)$ (the invariance of the MLE, Maximum Likelihood Estimation). Also $\ell^*(\eta_0) = \ell(\theta_0)$. Hence $2[\ell^*(\hat\eta) - \ell^*(\eta_0)] = 2[\ell(\hat\theta) - \ell(\theta_0)]$.
For the score, the chain rule gives $U^*(\eta) = \ell'(g(\eta))\, g'(\eta) = U(\theta)\, g'(\eta)$. The information transforms as $I^*_n(\eta) = I_n(\theta)\,[g'(\eta)]^2$ (the change-of-parameter theorem of Fisher Information and the Cramér–Rao Bound). So
$$\frac{U^*(\eta_0)^2}{I^*_n(\eta_0)} = \frac{U(\theta_0)^2\,[g'(\eta_0)]^2}{I_n(\theta_0)\,[g'(\eta_0)]^2} = \frac{U(\theta_0)^2}{I_n(\theta_0)}.$$
For the Wald statistic, $W^*_{\mathrm{W}} = (\hat\eta - \eta_0)^2 I^*_n(\hat\eta) = (\hat\eta - \eta_0)^2 [g'(\hat\eta)]^2 I_n(\hat\theta)$. This equals $(\hat\theta - \theta_0)^2 I_n(\hat\theta)$ exactly when $[g'(\hat\eta)(\hat\eta - \eta_0)]^2 = [g(\hat\eta) - g(\eta_0)]^2$. Since $g$ is monotone, the two quantities in brackets have the same sign, so the condition is $g(\hat\eta) - g(\eta_0) = g'(\hat\eta)(\hat\eta - \eta_0)$: the tangent line to $g$ at $\hat\eta$ must pass through the point $(\eta_0, g(\eta_0))$. $\square$
The invariance of the score statistic holds for the statistic as defined here, with the expected information $I_n(\theta_0)$ in the divisor. Some software divides by the observed information $J(\theta_0) = -\ell''(\theta_0)$ instead. That version is not invariant: differentiating $U^*(\eta) = U(g(\eta))\,g'(\eta)$ once more gives $J^*(\eta_0) = J(\theta_0)\,[g'(\eta_0)]^2 - U(\theta_0)\,g''(\eta_0)$, for a twice-differentiable $g$, and the second term does not vanish unless $g''(\eta_0) = 0$ or the score is zero. Fisher Information and the Cramér–Rao Bound gives the same rule in its section "Information depends on the parameter chosen".
The condition for the Wald statistic holds for every sample when $g$ is linear, and fails in general otherwise. In terms of the picture, the Wald statistic measures a horizontal distance, and stretching the horizontal axis unevenly changes it. The likelihood ratio measures a height, which no relabeling of the axis changes. The score statistic measures a slope in units that change with the axis in the same proportion.
For the log-odds, $p = g(\eta) = 1/(1 + e^{-\eta})$ and $g'(\eta) = p(1 - p)$, so $I^*_n(\eta) = [n/(p(1 - p))]\,[p(1 - p)]^2 = np(1 - p)$. For the 60 young men, $\hat\eta = \ln(0.25/0.75) = -1.0986$:
| Statistic for 15 of 60, $p_0 = 0.224$ | On $p$ | On the log-odds |
|---|---|---|
| Score | 0.2333 | 0.2333 |
| Wald | 0.2163 | 0.2329 |
| Likelihood ratio | 0.2273 | 0.2273 |
The same holds for intervals. For 10 of 40 the Wald interval for the log-odds is $-1.0986 \pm 1.960 \times 0.3651$, where $0.3651 = 1/\sqrt{40 \times 0.25 \times 0.75}$, which is $(-1.8143,\ -0.3829)$ when computed from unrounded values. Transformed back to proportions it is $(0.1401,\ 0.4054)$, close to Wilson's $(0.1419,\ 0.4019)$ and far from the Wald interval for $p$ itself, $(0.1158,\ 0.3842)$. At $p = 0.2294$ and $n = 40$ its exact coverage is 0.9628, and over $10 \le n \le 200$ its lowest coverage is 0.9387. (When $x$ is 0 or $n$ the log-odds estimate is infinite and this interval is the whole of $[0, 1]$.) Both are Wald intervals for the same quantity. The data did not change; the scale did.
So "the Wald test" is not one test but a family, one for each way of writing the parameter, and its accuracy depends on choosing a scale on which the log-likelihood is close to a parabola. For a proportion the log-odds scale is better than the probability scale. No scale is good everywhere:
Example (a Wald statistic that shrinks as the evidence grows). Keep $n = 60$ and $p_0 = 0.224$, and let the count rise far above what $H_0$ predicts.
| $x$ of 60 | Wald on the log-odds | Score | Likelihood ratio |
|---|---|---|---|
| 30 | 23.157 | 26.29 | 21.81 |
| 45 | 61.659 | 95.50 | 74.78 |
| 55 | 60.741 | 165.61 | 132.69 |
| 58 | 41.084 | 190.38 | 157.03 |
| 59 | 27.831 | 199.02 | 166.88 |
The score and likelihood ratio statistics grow with every additional hypertensive man. The Wald statistic on the log-odds scale rises to its largest value, 67.803, at 51 of 60 and then falls: 59 of 60 is weaker evidence against $p = 0.224$, by its measure, than 45 of 60. As $\hat{p}$ approaches 1 the estimated log-odds grows without bound, but its standard error $1/\sqrt{n\hat{p}(1 - \hat{p})}$ grows faster. Hauck and Donner (1977) showed that the Wald test behaves this way for a single parameter of a binomial logit model: the statistic decreases to zero as the estimate moves away from the null value. The disagreement recorded in Logistic Regression for the Challenger data, where the Wald statistic for temperature is $(-2.145)^2 = 4.60$ and the likelihood ratio statistic 7.952, is in the same direction, though with 23 flights neither chi-squared approximation is exact.
Why the reference distribution is chi-squared
So far the cut-off 3.8415 has been taken on trust. This section proves that, when $H_0$ is true and the sample is large, each of the three statistics has approximately the $\chi^2_1$ distribution, and that the three differ from one another by an amount that tends to zero.
The setting is that of Asymptotics of the Maximum Likelihood Estimator. The observations $X_1, \ldots, X_n$ are independent with mass function or density $f(x; \theta_0)$, and the model satisfies that article's regularity conditions C1 to C5:
- C1. The parameter space $\Theta$ is an open interval containing the true value $\theta_0$.
- C2. The set of $x$ with $f(x; \theta) > 0$ is the same for every $\theta$.
- C3. $\ln f(x; \theta)$ is three times continuously differentiable in $\theta$.
- C4. $E_{\theta_0}[u(X; \theta_0)] = 0$ and $E_{\theta_0}[u(X; \theta_0)^2] = -E_{\theta_0}[\partial u(X; \theta_0)/\partial\theta] = I(\theta_0)$, with $0 < I(\theta_0) < \infty$.
- C5. There is a number $c > 0$, with the interval $(\theta_0 - c, \theta_0 + c)$ inside $\Theta$, such that for every $\theta$ in that interval the third derivative of $\ln f(x; \theta)$ in $\theta$ is bounded in absolute value by a function $M(x)$ with $E_{\theta_0}[M(X)] < \infty$.
Three results of that article are used, all under C1 to C5 and with probabilities computed under $\theta_0$.
- (A consistent root; Theorem 1 and Corollary 1 there.) With probability tending to 1 the likelihood equation $U(\theta) = 0$ has a root $\hat\theta_n$ that is a local maximum of $\ell$, and $\hat\theta_n \xrightarrow{p} \theta_0$. When $\ell$ is strictly concave, as for the binomial, Poisson and normal-mean models, the root is unique and is the MLE.
- (Asymptotic normality; Theorem 2 there.) $\sqrt{n}\,(\hat\theta_n - \theta_0) \xrightarrow{d}$ the normal distribution with mean 0 and variance $1/I(\theta_0)$. Equivalently, $\sqrt{n I(\theta_0)}\,(\hat\theta_n - \theta_0) \xrightarrow{d} Z$, a standard normal variable.
- (Curvature near the true value; the lemma of that name there.) If $\theta^*_n$ are random points with $\theta^*_n \xrightarrow{p} \theta_0$, then $-\ell''(\theta^*_n)/n \xrightarrow{p} I(\theta_0)$.
Three tools come from Convergence in Distribution and the Proof of the Central Limit Theorem.
- The continuous mapping theorem. If $Y_n \xrightarrow{d} Y$ and $g$ is continuous, then $g(Y_n) \xrightarrow{d} g(Y)$. The same holds for convergence in probability to a constant.
- Slutsky's theorem. If $A_n \xrightarrow{p} a$ and $Y_n \xrightarrow{d} Y$, then $A_n Y_n \xrightarrow{d} aY$.
- Limits that are constants. Convergence in distribution to a constant is convergence in probability.
One more remark is needed. Two statistics that are equal with probability tending to 1 have the same limit in distribution, because their distribution functions differ at any point by at most the probability that the statistics differ. That remark lets us carry out the algebra on the event where the root $\hat\theta_n$ exists.
Throughout, write
$$Q_n = n I(\theta_0)\,(\hat\theta_n - \theta_0)^2 = \left[\sqrt{n I(\theta_0)}\,(\hat\theta_n - \theta_0)\right]^2.$$
By result 2 and the continuous mapping theorem with $g(y) = y^2$, $Q_n \xrightarrow{d} Z^2$, which has the $\chi^2_1$ distribution. The plan is to show that each of the three statistics is $Q_n$ multiplied by a factor that tends to 1 in probability.
The score statistic
The score statistic is the easiest, and it needs neither the estimate nor conditions C3 and C5.
Theorem (score test). Under C1, C2 and C4, if $H_0\colon \theta = \theta_0$ is true then $W_{\mathrm{S}} \xrightarrow{d} \chi^2_1$.
Proof. The score of the sample is a sum, $U(\theta_0) = \sum_{i=1}^n u(X_i; \theta_0)$, of independent and identically distributed terms. By C4 each term has mean 0 and variance $I(\theta_0)$, finite and positive. By the central limit theorem,
$$\frac{U(\theta_0)}{\sqrt{n I(\theta_0)}} \xrightarrow{d} Z.$$
$W_{\mathrm{S}}$ is the square of the left side, so by the continuous mapping theorem $W_{\mathrm{S}} \xrightarrow{d} Z^2$. $\square$
For a proportion this is the statement that $z^2$ is approximately $\chi^2_1$, that is, that the count is approximately normal: the theorem of The Normal Approximation to the Binomial.
Wilks's theorem
Here is the idea of the proof in plain words. Near its peak the log-likelihood is close to a parabola whose curvature is close to $n I(\theta_0)$. The drop of a parabola from its peak is half the curvature times the squared horizontal distance. So twice the drop from $\hat\theta_n$ to $\theta_0$ is close to $n I(\theta_0)(\hat\theta_n - \theta_0)^2 = Q_n$, and $Q_n$ is close to the square of a standard normal variable. Taylor's theorem turns "close to a parabola" into an exact equation with the curvature evaluated at an intermediate point, and the curvature result controls that point.
Theorem (Wilks, one parameter). Under C1 to C5, if $H_0\colon \theta = \theta_0$ is true, then
$$W_{\mathrm{LR}} = 2\,[\ell(\hat\theta_n) - \ell(\theta_0)] \xrightarrow{d} \chi^2_1,$$
where $\hat\theta_n$ is the consistent root of result 1.
Proof. Work on the event that $\hat\theta_n$ exists and $U(\hat\theta_n) = 0$, whose probability tends to 1. By C3 the function $\ell$ is twice continuously differentiable, so Taylor's theorem with the remainder in Lagrange's form, expanding about $\hat\theta_n$, gives a point $\theta^*_n$ between $\theta_0$ and $\hat\theta_n$ with
$$\ell(\theta_0) = \ell(\hat\theta_n) + U(\hat\theta_n)(\theta_0 - \hat\theta_n) + \tfrac12\,\ell''(\theta^*_n)(\theta_0 - \hat\theta_n)^2.$$
The middle term is zero because $U(\hat\theta_n) = 0$. Rearranging,
$$W_{\mathrm{LR}} = 2\,[\ell(\hat\theta_n) - \ell(\theta_0)] = -\ell''(\theta^*_n)\,(\hat\theta_n - \theta_0)^2 = A_n\, Q_n, \qquad A_n = \frac{-\ell''(\theta^*_n)/n}{I(\theta_0)}.$$
Since $\theta^*_n$ lies between $\theta_0$ and $\hat\theta_n$, $\lvert \theta^*_n - \theta_0 \rvert \le \lvert \hat\theta_n - \theta_0 \rvert$, and $\hat\theta_n \xrightarrow{p} \theta_0$ by result 1, so $\theta^*_n \xrightarrow{p} \theta_0$. By result 3, $-\ell''(\theta^*_n)/n \xrightarrow{p} I(\theta_0)$, so $A_n \xrightarrow{p} 1$. Since $Q_n \xrightarrow{d} Z^2$, Slutsky's theorem gives $A_n Q_n \xrightarrow{d} Z^2$. $\square$
The theorem is named for Wilks, whose 1938 paper gave the large-sample distribution of the likelihood ratio for composite hypotheses, the case described under "More than one parameter" below.
The Wald statistic
Theorem (Wald test). Under C1 to C5, if $I(\theta)$ is continuous at $\theta_0$ and $H_0\colon \theta = \theta_0$ is true, then $W_{\mathrm{W}} = (\hat\theta_n - \theta_0)^2\, n I(\hat\theta_n) \xrightarrow{d} \chi^2_1$. The same holds with the observed information $J(\hat\theta_n) = -\ell''(\hat\theta_n)$ in place of $n I(\hat\theta_n)$, without the continuity assumption.
Proof. Write $W_{\mathrm{W}} = B_n Q_n$ with $B_n = I(\hat\theta_n)/I(\theta_0)$. Since $\hat\theta_n \xrightarrow{p} \theta_0$ and $I$ is continuous at $\theta_0$, the continuous mapping theorem gives $I(\hat\theta_n) \xrightarrow{p} I(\theta_0)$, so $B_n \xrightarrow{p} 1$, and Slutsky's theorem gives $B_n Q_n \xrightarrow{d} Z^2$. With the observed information the factor is $[-\ell''(\hat\theta_n)/n]/I(\theta_0)$, which tends to 1 in probability by result 3 applied to $\theta^*_n = \hat\theta_n$. $\square$
The three are equivalent under the null hypothesis
Theorem (asymptotic equivalence). Under C1 to C5 with $I$ continuous at $\theta_0$, if $H_0$ is true then $W_{\mathrm{S}} - W_{\mathrm{LR}} \xrightarrow{p} 0$ and $W_{\mathrm{W}} - W_{\mathrm{LR}} \xrightarrow{p} 0$.
Proof. Work again on the event that the root exists. We have $W_{\mathrm{LR}} = A_n Q_n$ and $W_{\mathrm{W}} = B_n Q_n$ with $A_n, B_n \xrightarrow{p} 1$. For the score statistic, apply the mean value theorem to $U = \ell'$ between $\theta_0$ and $\hat\theta_n$: there is a point $\theta^{**}_n$ between them with $U(\hat\theta_n) - U(\theta_0) = \ell''(\theta^{**}_n)(\hat\theta_n - \theta_0)$. Since $U(\hat\theta_n) = 0$, this says $U(\theta_0) = -\ell''(\theta^{**}_n)(\hat\theta_n - \theta_0)$, and so
$$W_{\mathrm{S}} = \frac{U(\theta_0)^2}{n I(\theta_0)} = \frac{[\ell''(\theta^{**}_n)]^2 (\hat\theta_n - \theta_0)^2}{n I(\theta_0)} = C_n\, Q_n, \qquad C_n = \left[\frac{-\ell''(\theta^{**}_n)/n}{I(\theta_0)}\right]^2.$$
As for $\theta^*_n$, $\theta^{**}_n \xrightarrow{p} \theta_0$, so by result 3 and the continuous mapping theorem $C_n \xrightarrow{p} 1$. Therefore $W_{\mathrm{S}} - W_{\mathrm{LR}} = (C_n - A_n) Q_n$ with $C_n - A_n \xrightarrow{p} 0$, and $W_{\mathrm{W}} - W_{\mathrm{LR}} = (B_n - A_n) Q_n$ with $B_n - A_n \xrightarrow{p} 0$. By Slutsky's theorem each product converges in distribution to $0 \cdot Z^2 = 0$, and convergence in distribution to a constant is convergence in probability. $\square$
The equivalence is a statement about samples drawn under $H_0$. It says nothing about samples drawn under an alternative. The tables above, at a fixed sample size of 60, show the three statistics far apart when the data lie far from the null value.
The theorems at work on a proportion
A single observation with $f(x; p) = p^x (1 - p)^{1 - x}$ for $x$ in $\{0, 1\}$ and $0 < p < 1$ satisfies the conditions.
- Its support, $\{0, 1\}$, does not depend on $p$, and $(0, 1)$ is an open interval (C1, C2).
- $\ln f = x \ln p + (1 - x)\ln(1 - p)$ has derivatives of every order on $(0, 1)$ (C3).
- The score $u = (x - p)/[p(1 - p)]$ has mean 0 and variance $1/[p(1 - p)] = I(p)$. That is also $-E[\partial u/\partial p]$, by the calculation of the curvature above with $n = 1$ (C4).
- The third derivative, $2x/p^3 - 2(1 - x)/(1 - p)^3$, is bounded on any closed interval inside $(0, 1)$ (C5).
$I(p)$ is continuous. So all four theorems apply to the tests of a proportion.
Because the count is binomial, the distribution of each statistic under $H_0$ can be computed exactly at any $n$ and set beside the limit. For $p_0 = 0.224$:
| $n$ | $P(W_{\mathrm{LR}} \ge 2.7055)$ | $P(W_{\mathrm{LR}} \ge 3.8415)$ | $P(W_{\mathrm{LR}} \ge 6.6349)$ | $P(\lvert W_{\mathrm{LR}} - W_{\mathrm{S}} \rvert > 0.1)$ | $P(\lvert W_{\mathrm{LR}} - W_{\mathrm{W}} \rvert > 0.1)$ |
|---|---|---|---|---|---|
| 20 | 0.1014 | 0.0633 | 0.0125 | 0.2818 | 0.4537 |
| 60 | 0.1198 | 0.0446 | 0.0077 | 0.2146 | 0.3532 |
| 200 | 0.1068 | 0.0506 | 0.0108 | 0.1493 | 0.2395 |
| 1,000 | 0.1028 | 0.0486 | 0.0099 | 0.0580 | 0.1210 |
| 5,000 | 0.0999 | 0.0491 | 0.0099 | 0.0115 | 0.0456 |
| 20,000 | 0.0999 | 0.0501 | 0.0099 | 0.0015 | 0.0118 |
| $\chi^2_1$ limit | 0.1000 | 0.0500 | 0.0100 | 0 | 0 |
The three cut-offs are the 0.90, 0.95 and 0.99 points of $\chi^2_1$. The tail probabilities approach their limits, though not steadily, since the count is a whole number and the set of rejected counts changes in jumps. The last two columns show the equivalence theorem: the probability that two of the statistics differ by more than 0.1 falls toward zero, more slowly for the Wald statistic than for the score statistic.
More than one parameter
Most models have several parameters, and most hypotheses fix some of them or a relation among them and leave the rest free. The one-parameter results extend, and this section states the extension without proof.
Let the parameter be a vector $\boldsymbol\theta = (\theta_1, \ldots, \theta_k)$, with score vector $\mathbf{U}(\boldsymbol\theta)$ of partial derivatives of $\ell$ and information matrix $\mathbf{I}_n(\boldsymbol\theta)$, as in the section on several parameters of Fisher Information and the Cramér–Rao Bound. Let $H_0$ impose $r$ independent restrictions on $\boldsymbol\theta$, so that $k - r$ parameters remain free under it. Write $\hat{\boldsymbol\theta}$ for the unrestricted MLE and $\tilde{\boldsymbol\theta}$ for the MLE under $H_0$. The three statistics are
$$W_{\mathrm{LR}} = 2\,[\ell(\hat{\boldsymbol\theta}) - \ell(\tilde{\boldsymbol\theta})], \qquad W_{\mathrm{S}} = \mathbf{U}(\tilde{\boldsymbol\theta})^{\mathsf T}\, \mathbf{I}_n(\tilde{\boldsymbol\theta})^{-1}\, \mathbf{U}(\tilde{\boldsymbol\theta}),$$
and, when $H_0$ fixes the whole vector at $\boldsymbol\theta_0$ (so $r = k$), $W_{\mathrm{W}} = (\hat{\boldsymbol\theta} - \boldsymbol\theta_0)^{\mathsf T}\, \mathbf{I}_n(\hat{\boldsymbol\theta})\, (\hat{\boldsymbol\theta} - \boldsymbol\theta_0)$. For a single restriction $h(\boldsymbol\theta) = 0$ the Wald statistic is $h(\hat{\boldsymbol\theta})^2$ divided by the estimated large-sample variance of $h(\hat{\boldsymbol\theta})$, which the delta method supplies.
Theorem (Wilks, several parameters; cited). Under regularity conditions of the same kind as C1 to C5, if $H_0$ is true then $W_{\mathrm{LR}}$ converges in distribution to the chi-squared distribution with $r$ degrees of freedom. The number $r$ of restrictions is the number of free parameters in the full model minus the number free under $H_0$.
This is the result of Wilks (1938), whose title names the large-sample distribution of the likelihood ratio for composite hypotheses. The NIST/SEMATECH handbook (§8.2.3.3) states it in the form used in practice: $-2\ln\lambda$ has approximately a chi-squared distribution with as many degrees of freedom as the null model has fewer parameters. van der Vaart (1998) devotes Chapter 16, "Likelihood Ratio Tests", to the theory. Wald (1943) studied tests of hypotheses about several parameters in large samples; the last two sections of his paper before the summary are titled "Optimum properties of the likelihood ratio test" and "Large sample distribution of the likelihood ratio". Rao (1948) proposed the score test for hypotheses about several parameters. Following Fisher, he calls the derivatives $\phi_i = \partial \log L/\partial\theta_i$ efficient scores, writes $(\alpha_{ij})$ for their covariance matrix, which is the information matrix, and $(\alpha^{ij})$ for its inverse, and states that $\sum\sum\alpha^{ij}\phi_i\phi_j$ at the true values is distributed in large samples as $\chi^2$ with $k$ degrees of freedom (Section 2, p. 51). For one parameter he takes the statistic $\phi_1^2(\theta^0)/\alpha_{11}(\theta^0)$ as $\chi^2$ with 1 degree of freedom, citing Fisher for the name "efficient score" and earlier papers of Wald and of Rao and Poti for the optimum properties of this test. For the simple hypothesis $\theta = \theta^0$ on $k$ parameters he proposes $\sum\sum\alpha^{ij}(\theta^0)\phi_i(\theta^0)\phi_j(\theta^0)$, his equation (3·4), as $\chi^2$ with $k$ degrees of freedom, contrasts it with the statistic of Wald (1943), $\sum\sum\alpha_{ij}(\hat\theta)(\hat\theta_i - \theta^0_i)(\hat\theta_j - \theta^0_j)$, and remarks: "The test associated with (3·4) besides being simpler than Wald's has some theoretical advantages as shown in § 5" (Section 3, p. 53). For a composite hypothesis he evaluates the same quadratic form at the maximum likelihood estimates under the $s$ restrictions, found with Lagrangian multipliers, and uses it as $\chi^2$ with $s$ degrees of freedom (pp. 51–54). He demonstrates this limit for two parameters and one restriction, under assumptions he states, and describes the general case. These are the page's $W_{\mathrm{S}}$ and, for $r = k$, its $W_{\mathrm{W}}$. The test is known as Rao's score test and, in the econometrics literature, as the Lagrange multiplier test. Bera and Bilias (2024, p. 10), who review the test's history, trace the second name to a paper of Silvey of 1959, and Buse (1982) uses it in his title. Their longer history is Bera and Bilias (2001), which we cite by its title. Bera and Bilias (2024, equations 15 and 16) give the same forms for a general set of restrictions and state the $\chi^2_r$ limit of the score statistic. These limits are given here as cited results and not from a general proof we have read, and no result on this page depends on them. The third worked problem uses the forms for two proportions, and there the reference distribution of each statistic is justified separately.
Three uses of the several-parameter result appear elsewhere on the site.
- Two proportions. The hypothesis $p_1 = p_2$ is one restriction on two parameters. The third worked problem shows that the score statistic is Pearson's chi-square statistic for the $2 \times 2$ table, which Comparing Two Proportions proves equal to the square of the pooled $z$ statistic. The Wald statistic is the square of the unpooled $z$. The likelihood ratio statistic is $2\sum O\ln(O/E)$ over the four cells, with $O$ the observed and $E$ the expected counts.
- Deviance. Logistic Regression defines the deviance of a fitted model as $-2\ell$ at the fitted coefficients. The difference between the deviances of two nested models is $2[\ell(\hat{\boldsymbol\theta}) - \ell(\tilde{\boldsymbol\theta})]$, the likelihood ratio statistic for the restrictions that turn the larger model into the smaller. The drop of 7.952 from adding temperature is referred to $\chi^2_1$ because one coefficient is set to zero.
- Penalized comparisons. Model selection by AIC compares $-2\ell$ plus a penalty per parameter; the first term is the same quantity.
When the chi-squared approximation fails
The theorems have hypotheses, and each can fail in practice.
Small samples and discrete data. The limits are limits. At $n = 60$ the three tests of $p_0 = 0.224$ had actual sizes 0.0612, 0.0749 and 0.0446, not 0.05, and the table of exact tail probabilities shows the approach is slow. For a count the distribution of each statistic is discrete at every $n$.
A true value on the boundary of the parameter space. C1 requires $\theta_0$ to be an interior point. Suppose the true value sits on the edge of the parameter space, such as a variance or a proportion that is zero. Then the MLE cannot fall on both sides of it, and the argument through a symmetric normal limit fails. Self and Liang (1987) derive the limits in such cases and give examples in which the likelihood ratio statistic has a mixture of chi-squared distributions. The second worked problem shows a related small-sample effect for a proportion that is near zero but not on it.
A case of a true boundary arises in Maximum Likelihood Estimation, which fits a negative binomial model to Iowa's weekly deaths and finds its maximized log-likelihood 231.43 above that of the Poisson model. The Poisson model is the limit of the negative binomial as the shape $k$ grows without bound. Written with $1/k$ as the parameter, the Poisson model is the value $1/k = 0$, the edge of the parameter space. So C1 fails under the null hypothesis, and the theorems of this page justify no reference distribution for the statistic, $2 \times 231.43 = 462.86$. We therefore report the statistic without a calibrated $p$-value. For a sense of scale only: the 0.999 point of $\chi^2_1$ is 10.83, and 462.86 is more than forty times that. That the extra parameter is needed does not rest on this statistic. Overdispersion in Count Data applies Fisher's index of dispersion, which has a calibrated chi-squared reference, to the same 313 weeks of all-cause deaths. It finds 1,201 on 312 degrees of freedom and concludes that the Poisson model is ruled out. Judging a small gain in such a problem would need the boundary theory of Self and Liang (1987).
A support that moves with the parameter. C2 fails for the uniform distribution on $(0, \theta)$. There the likelihood ratio statistic has exactly the $\chi^2_2$ distribution under $H_0$ at every sample size, not $\chi^2_1$, and a test that uses the cut-off 3.8415 has size 0.1465. The fourth worked problem proves this.
An unsuitable scale, for the Wald test. The example of the shrinking Wald statistic shows that a Wald statistic can shrink exactly when the evidence is strongest. The likelihood ratio and score tests do not depend on the scale.
Example (a common misreading). "The three tests are equivalent, so it does not matter which is used." The equivalence is a limit under $H_0$. For 15 of 60 against 0.224 the three statistics are 0.2333, 0.2163 and 0.2273 and nothing depends on the choice. For corps XI in the first worked problem they are 13.43, 6.55 and 10.27, and the largest $p$-value is about forty times the smallest. And the three intervals for samples of 40 young adults cover the true proportion 96.3%, 91.1% and 94.1% of the time. Where the three disagree noticeably, the log-likelihood is far from a parabola between the estimate and the null value, and the disagreement is itself a warning that the large-sample approximation is under strain.
What is proved here, what is cited, and what is computed
Proved on this page:
- the description of each statistic as twice a drop on a curve;
- the three statistics for a binomial proportion, and their equality with $z^2$ for a normal mean with known variance;
- the duality between tests and intervals, with the Wald and Wilson intervals as inverted Wald and score tests;
- the invariance of the likelihood ratio and score statistics under a change of parameter, and the formula for the Wald statistic;
- the $\chi^2_1$ limit of each statistic under $H_0$ in a one-parameter model, and their asymptotic equivalence;
- the exact $\chi^2_2$ distribution in the uniform model;
- the $\chi^2_1$ limit of the pooled and unpooled $z^2$ for two groups of equal size.
Taken from other pages:
- the existence of a consistent root, its asymptotic normality and the curvature lemma (Asymptotics of the Maximum Likelihood Estimator);
- the properties of the score and the change-of-parameter rule for the information (Fisher Information and the Cramér–Rao Bound);
- the central limit theorem, the continuous mapping theorem and Slutsky's theorem (Convergence in Distribution and the Proof of the Central Limit Theorem);
- the weak law of large numbers, used in the third worked problem (The Laws of Large Numbers);
- the algebra of Wilson's interval (Confidence Intervals for a Proportion).
Cited, not proved:
- Wilks's theorem and the limits of the Wald and score statistics for several parameters, from the sources named and not from proofs we have read;
- the behavior of the Wald test in logit models in general (Hauck and Donner, 1977);
- the limits when the true value is on a boundary (Self and Liang, 1987).
Computed exactly by the script: every statistic, interval, coverage and error rate on the page, from binomial and Poisson probabilities. There is no simulation in this article, and the exact figures describe the sample sizes and proportions shown, not all others.
Worked problems
The four problems are posed on data used elsewhere in the unit and the course (the horse-kick deaths of The Poisson Distribution, the hypertension samples of Inference on Proportions) and on one small exact model. They exercise the three statistics in a new family, the intervals at the edge of the parameter space, the several-parameter forms, and the failure of a regularity condition. The article's script recomputes every number.
Problem 1 (three tests of a Poisson rate). In Bortkiewicz's data the ten army corps of standard composition had 122 deaths from horse kicks in 200 corps-years, a rate of 0.61 per corps per year; treat that rate as known. The Guard corps had 16 deaths in 20 years and corps XI had 25. Model a corps' yearly counts as independent Poisson variables with mean $\lambda$. (a) Derive the score, Wald and likelihood ratio statistics for $H_0\colon \lambda = \lambda_0$. (b) Compute them for each corps against $\lambda_0 = 0.61$. (c) Find the three 95% intervals for the Guard corps' rate.
Solution. (a) With $n$ yearly counts and total $t = \sum x_i$, the Poisson mass function $e^{-\lambda}\lambda^x/x!$ gives, up to a constant,
$$\ell(\lambda) = t \ln\lambda - n\lambda, \qquad U(\lambda) = \frac{t}{\lambda} - n, \qquad -\ell''(\lambda) = \frac{t}{\lambda^2}.$$
The score is zero at $\hat\lambda = t/n$ and $\ell'' < 0$, so $\hat\lambda$ is the MLE. Since $E_\lambda[T] = n\lambda$, the information is $I_n(\lambda) = n\lambda/\lambda^2 = n/\lambda$. Then
$$W_{\mathrm{S}} = \frac{(t/\lambda_0 - n)^2}{n/\lambda_0} = \frac{(t - n\lambda_0)^2}{n\lambda_0}, \qquad W_{\mathrm{W}} = \frac{n(\hat\lambda - \lambda_0)^2}{\hat\lambda}, \qquad W_{\mathrm{LR}} = 2\left[t \ln\frac{\hat\lambda}{\lambda_0} - n(\hat\lambda - \lambda_0)\right].$$
The score statistic is the squared difference between the observed total and its expected value $n\lambda_0$, divided by that expected value, which is the Poisson variance under $H_0$. The Wald statistic divides the same squared difference, $n^2(\hat\lambda - \lambda_0)^2$, by the estimated variance $n\hat\lambda$.
(b) Here $n = 20$ and $n\lambda_0 = 12.2$.
| Corps | $t$ | $\hat\lambda$ | Score | Wald | Likelihood ratio |
|---|---|---|---|---|---|
| Guard | 16 | 0.80 | $3.8^2/12.2 = 1.1836$ | $20 \times 0.19^2/0.80 = 0.9025$ | $2\,(16 \times 0.2712 - 3.8) = 1.0769$ |
| XI | 25 | 1.25 | $12.8^2/12.2 = 13.4295$ | $20 \times 0.64^2/1.25 = 6.5536$ | $2\,(25 \times 0.7174 - 12.8) = 10.2720$ |
The logarithms are $\ln(0.80/0.61) = 0.2712$ and $\ln(1.25/0.61) = 0.7174$, and the statistics are computed from their unrounded values. For the Guard corps the $p$-values are 0.2766, 0.3421 and 0.2994: no evidence of a rate different from 0.61, by any of the three. For corps XI they are 0.0002, 0.0105 and 0.0014. All three reject at 0.05, but the largest $p$-value is about forty times the smallest, and at the 0.01 level the Wald test would not reject. The null value 0.61 is far below the estimate 1.25, and the Wald statistic measures the difference against the larger variance at 1.25. As in the proportion table, the likelihood ratio lies between the other two. Treating 0.61 as known ignores its own sampling error, so this is an illustration of the three statistics and not a full analysis.
(c) For the Guard corps, $\hat\lambda = 0.80$. The Wald interval is $\hat\lambda \pm 1.960\sqrt{\hat\lambda/n} = 0.80 \pm 1.960 \times 0.2 = (0.4080,\ 1.1920)$. The score interval collects the $\lambda$ with $n(\hat\lambda - \lambda)^2 \le z^2\lambda$, that is, $\lambda^2 - (2\hat\lambda + z^2/n)\lambda + \hat\lambda^2 \le 0$. With $z^2/n = 3.8415/20 = 0.1921$ the roots of the quadratic are
$$\lambda = \frac{1.7921 \pm \sqrt{1.7921^2 - 4 \times 0.64}}{2} = 0.4925 \ \text{and} \ 1.2996.$$
(The roots are computed from the unrounded coefficient; with 1.7921 as shown the first comes out as 0.4924.) The likelihood ratio interval, found numerically from $2[16\ln(0.80/\lambda) - 20(0.80 - \lambda)] \le 3.8415$, is $(0.4692,\ 1.2585)$. The pattern of the proportion is repeated: the Wald interval is symmetric about the estimate, the other two reach further up than down, and the likelihood ratio interval lies between them. All three contain 0.61, in agreement with part (b).
Problem 2 (no events among forty). A sample of 40 has no events, $x = 0$. Find the 95% likelihood ratio interval for $p$, compare it with the Wilson and exact intervals of Confidence Intervals for a Proportion, and explain the difference.
Solution. With $x = 0$ the log-likelihood is $\ell(p) = n\ln(1 - p)$, which is decreasing in $p$, so its largest value over $0 \le p \le 1$ is $\ell(0) = 0$ at $\hat{p} = 0$. The likelihood ratio statistic is $W_{\mathrm{LR}} = 2[0 - n\ln(1 - p)] = -2n\ln(1 - p)$, and the interval contains the $p$ with
$$-2n\ln(1 - p) \le z^2 \quad \Longleftrightarrow \quad p \le 1 - e^{-z^2/(2n)} = 1 - e^{-3.8415/80} = 0.0469.$$
The 95% likelihood ratio interval is $(0,\ 0.0469)$. Wilson's interval for $x = 0$ is $(0,\ z^2/(n + z^2)) = (0,\ 0.0876)$, the Clopper–Pearson upper limit is $1 - 0.025^{1/40} = 0.0881$, and the exact one-sided 95% bound of the course page is $1 - 0.05^{1/40} = 0.0722$.
The likelihood ratio limit is about half the others, and it is too short. At $p = 0.0469$ the probability of observing no events among 40 is $(1 - p)^{40} = e^{-z^2/2} = 0.1465$. So the interval excludes values of $p$ under which the observed outcome has probability nearly 15%, whereas a 95% interval should exclude only values under which the data are in an extreme 5%. The cause is a small sample close to the boundary, not a failure of the regularity conditions. Take a true proportion such as $p = 0.04725$. It is an interior point, so C1 holds and Wilks's theorem applies to it in the limit of large $n$. But at $n = 40$ the expected number of events is only $np = 1.89$, and a sample with no events, whose estimate $\hat{p} = 0$ sits on the edge of the parameter space, has probability $(1 - 0.04725)^{40} = 0.1443$. The $\chi^2_1$ approximation describes estimates that fall on either side of the true value, and it is poor when the estimate lands on the boundary one time in seven.
The exact computation of the main text shows the cost. Across the 2,000 values of $p$ at $n = 40$, the lowest coverage of the likelihood ratio interval is 0.8450, at that same $p = 0.04725$, just above the limit 0.0469. For any $p$ just above the limit the sample with no events gives an interval that misses, and its probability is just under 0.1465. With a larger sample the same $p$ fares better. At $n = 400$ the probability of no events is $(1 - 0.04725)^{400}$, less than one in a hundred million, and the exact coverage of the likelihood ratio interval at $p = 0.04725$ is 0.9404, against 0.8450 at $n = 40$. When no events are seen in a small sample, the exact bound is the one to report.
Problem 3 (two proportions, three ways). In the sample of Comparing Two Proportions, 26 of 80 young men and 9 of 80 young women have hypertension. For $H_0\colon p_1 = p_2$, compute the score, Wald and likelihood ratio statistics, and identify each with a statistic of that page.
Solution. There are two parameters, $(p_1, p_2)$, and one restriction, and the reference distribution is $\chi^2_1$, for reasons given at the end of the solution. The two samples are independent, so the log-likelihood is the sum
$$\ell(p_1, p_2) = \sum_{i=1}^{2}\big[x_i \ln p_i + (n_i - x_i)\ln(1 - p_i)\big].$$
Estimates. Without the restriction each term is maximized separately: $\hat{p}_1 = 26/80 = 0.3250$ and $\hat{p}_2 = 9/80 = 0.1125$. Under $H_0$ the common value $p$ has log-likelihood $(x_1 + x_2)\ln p + (N - x_1 - x_2)\ln(1 - p)$ with $N = n_1 + n_2$, a single binomial log-likelihood, maximized at the pooled proportion $\tilde{p} = 35/160 = 0.21875$.
Score. The components of the score vector are $U_i = n_i(\hat{p}_i - p_i)/[p_i(1 - p_i)]$, and the information matrix is diagonal with entries $n_i/[p_i(1 - p_i)]$, because the cross derivative of $\ell$ is zero. At $(\tilde{p}, \tilde{p})$, writing $\tilde{q} = 1 - \tilde{p}$,
$$W_{\mathrm{S}} = \sum_{i=1}^{2} \left[\frac{n_i(\hat{p}_i - \tilde{p})}{\tilde{p}\tilde{q}}\right]^2 \frac{\tilde{p}\tilde{q}}{n_i} = \sum_{i=1}^{2} \frac{n_i(\hat{p}_i - \tilde{p})^2}{\tilde{p}\tilde{q}}.$$
Since $\tilde{p} = (n_1\hat{p}_1 + n_2\hat{p}_2)/N$, we have $\hat{p}_1 - \tilde{p} = n_2(\hat{p}_1 - \hat{p}_2)/N$ and $\hat{p}_2 - \tilde{p} = -n_1(\hat{p}_1 - \hat{p}_2)/N$. Substituting,
$$W_{\mathrm{S}} = \frac{(n_1 n_2^2 + n_2 n_1^2)(\hat{p}_1 - \hat{p}_2)^2}{N^2\,\tilde{p}\tilde{q}} = \frac{(\hat{p}_1 - \hat{p}_2)^2}{\tilde{p}\tilde{q}\,(1/n_1 + 1/n_2)},$$
using $n_1 n_2(n_1 + n_2)/N^2 = n_1 n_2/N = 1/(1/n_1 + 1/n_2)$. That is the square of the pooled $z$ statistic, and so, by the proof on that page, Pearson's chi-square statistic for the $2 \times 2$ table. Numerically, $W_{\mathrm{S}} = 0.2125^2/(0.21875 \times 0.78125 \times 0.025) = 10.5691$, the page's 10.569, the square of its $z = 3.2510$ before rounding.
Wald. The restriction is $h(p_1, p_2) = p_1 - p_2 = 0$. The estimate $\hat{p}_1 - \hat{p}_2$ has estimated variance $\hat{p}_1(1 - \hat{p}_1)/n_1 + \hat{p}_2(1 - \hat{p}_2)/n_2$, the sum of the two estimated variances, since the samples are independent. So
$$W_{\mathrm{W}} = \frac{(\hat{p}_1 - \hat{p}_2)^2}{\hat{p}_1(1 - \hat{p}_1)/n_1 + \hat{p}_2(1 - \hat{p}_2)/n_2} = 11.3167,$$
the square of the $z$ statistic with the unpooled standard error (3.3640 after rounding), the one that page uses for its confidence interval.
Likelihood ratio. The unrestricted maximum is $\ell(\hat{p}_1, \hat{p}_2) = -78.5833$ and the restricted maximum is $\ell(\tilde{p}, \tilde{p}) = -84.0514$, so $W_{\mathrm{LR}} = 2 \times 5.4681 = 10.9362$. Collecting the four cells of the table, with observed counts $O$ (26, 54, 9, 71) and the expected counts under $H_0$, $E$ (17.5, 62.5, 17.5, 62.5), the same number is $2\sum O\ln(O/E)$: each term $x_i\ln(\hat{p}_i/\tilde{p})$ is $O\ln(O/E)$ for a cell of hypertensive adults, since $O = n_i\hat{p}_i$ and $E = n_i\tilde{p}$, and each term $(n_i - x_i)\ln[(1 - \hat{p}_i)/\tilde{q}]$ is the same for a cell of the others.
| Statistic | Value | $p$-value | Name on the course page |
|---|---|---|---|
| Score | 10.5691 | 0.00115 | pooled $z^2$, Pearson's $X^2$ |
| Wald | 11.3167 | 0.00077 | unpooled $z^2$ |
| Likelihood ratio | 10.9362 | 0.00094 | not on that page |
The three agree that the difference is real. The course's rule "pooled for the test, unpooled for the interval" is again the choice of a score test and a Wald interval.
Why $\chi^2_1$ is the reference. For the score and Wald statistics this follows from results proved on the site, for two groups of the same size $n$, as here. Under $H_0$ both groups have one proportion $p$, with $q = 1 - p$. Pair the observations: the $j$th difference $Y_{1j} - Y_{2j}$ of the 0/1 outcomes has mean 0 and variance $2pq$, and the differences are independent and identically distributed. Their mean is $\hat{p}_1 - \hat{p}_2$, so by the central limit theorem $\sqrt{n}(\hat{p}_1 - \hat{p}_2)/\sqrt{2pq} \xrightarrow{d} Z$, a standard normal variable. By the weak law of large numbers (The Laws of Large Numbers) $\hat{p}_1$, $\hat{p}_2$ and $\tilde{p}$ all converge in probability to $p$. So by the continuous mapping theorem both variance estimates, $2\tilde{p}\tilde{q}$ and $\hat{p}_1(1 - \hat{p}_1) + \hat{p}_2(1 - \hat{p}_2)$, converge in probability to $2pq$. Slutsky's theorem (Convergence in Distribution and the Proof of the Central Limit Theorem) then gives that the pooled $z$ and the unpooled $z$ each converge in distribution to $Z$, and their squares to $\chi^2_1$. One case needs a word. A statistic is undefined when its variance estimate is zero: for the pooled $z$ when $\tilde{p}$ is 0 or 1, and for the unpooled $z$ when each group is all events or all non-events. Define the statistic as 0 on that event. The event requires at least one group to be all of one kind, which has probability at most $2[p^n + q^n]$, and that tends to 0 because $0 < p < 1$. By the remark in the section on the reference distribution, statistics that are equal with probability tending to 1 have the same limit, so the definition on that event does not affect the result. For the likelihood ratio statistic the $\chi^2_1$ reference rests on the cited several-parameter form of Wilks's theorem, which this page does not prove.
At one design the three can be compared exactly. When both groups of 80 are drawn with the young adults' common proportion 0.2294, the probability that each statistic exceeds 3.8415, summed over all pairs of counts, is 0.0513 for the score statistic, 0.0517 for the Wald statistic and 0.0517 for the likelihood ratio. All three are close to 0.05 at this design; one design says nothing about others.
Problem 4 (a model in which Wilks's theorem fails). Let $X_1, \ldots, X_n$ be independent and uniform on $(0, \theta)$. Find the exact distribution of the likelihood ratio statistic for $H_0\colon \theta = \theta_0$ when $H_0$ is true, and the actual size of the test that rejects when the statistic exceeds 3.8415.
Solution. The density of one observation is $1/\theta$ for $0 < x < \theta$, so the likelihood is $L(\theta) = \theta^{-n}$ when $\theta$ exceeds the largest observation $m = \max x_i$, and 0 otherwise. It is decreasing in $\theta$ where it is positive, so the MLE is $\hat\theta = m$, as Maximum Likelihood Estimation shows. Under $H_0$ every observation is below $\theta_0$, so $m < \theta_0$ and $L(\theta_0) = \theta_0^{-n} > 0$. Therefore
$$\Lambda = \frac{L(\theta_0)}{L(\hat\theta)} = \left(\frac{m}{\theta_0}\right)^n, \qquad W_{\mathrm{LR}} = -2\ln\Lambda = 2n\ln\frac{\theta_0}{m}.$$
Under $H_0$, for $0 < v < 1$, the event $\Lambda \le v$ is the event $m \le \theta_0 v^{1/n}$, that every observation is at most $\theta_0 v^{1/n}$. Each observation satisfies this with probability $v^{1/n}$, and they are independent, so $P(\Lambda \le v) = (v^{1/n})^n = v$: the ratio $\Lambda$ is uniform on $(0, 1)$. Then for $w > 0$,
$$P(W_{\mathrm{LR}} > w) = P(\Lambda < e^{-w/2}) = e^{-w/2}.$$
This is the upper tail of the chi-squared distribution with 2 degrees of freedom, whose density $\tfrac12 e^{-x/2}$ (the formula of The Chi-Squared and t Distributions with $\nu = 2$) integrates from $w$ to infinity to $e^{-w/2}$. So $W_{\mathrm{LR}}$ has exactly the $\chi^2_2$ distribution for every $n \ge 1$, not $\chi^2_1$, and the test with cut-off 3.8415 has size $e^{-3.8415/2} = 0.1465$ at every sample size. The correct cut-off for a 5% test is $-2\ln 0.05 = 5.9915$.
Wilks's theorem does not apply because the support $(0, \theta)$ depends on the parameter, so C2 fails; the log-likelihood has no stationary point at the MLE, and the estimate approaches $\theta_0$ from one side only. The proof broke at its first step, the Taylor expansion about a point where the score is zero. The number 0.1465 also appeared in Problem 2. Both are $e^{-3.8415/2}$, the value of the ratio $\Lambda$ at which a test with the $\chi^2_1$ cut-off begins to reject. In Problem 2 the ratio was itself the probability of the observed sample. Here the ratio is uniformly distributed under $H_0$, so the probability that it falls below that value is the value itself.
Summary
| Statistic | One parameter | What it measures | Proportion | Inverted, it gives |
|---|---|---|---|---|
| Likelihood ratio | $2[\ell(\hat\theta) - \ell(\theta_0)]$ | drop of the log-likelihood | $2[x\ln(\hat{p}/p_0) + (n - x)\ln((1 - \hat{p})/(1 - p_0))]$ | likelihood ratio interval |
| Wald | $(\hat\theta - \theta_0)^2 I_n(\hat\theta)$ | distance in standard errors at $\hat\theta$ | $n(\hat{p} - p_0)^2/[\hat{p}(1 - \hat{p})]$ | Wald interval |
| Score | $U(\theta_0)^2/I_n(\theta_0)$ | slope at $\theta_0$ in its standard deviations | $n(\hat{p} - p_0)^2/[p_0(1 - p_0)]$ | Wilson's interval |
| Result | Statement |
|---|---|
| Quadratic log-likelihood | the three statistics are equal (normal mean, known variance: all equal $z^2$) |
| Large samples, $H_0$ true, C1 to C5 | each statistic $\xrightarrow{d} \chi^2_1$; differences $\xrightarrow{p} 0$ |
| Several parameters, $r$ restrictions | $W_{\mathrm{LR}} \xrightarrow{d} \chi^2_r$ (cited) |
| Change of parameter | likelihood ratio and score unchanged; Wald changes |
| Test and interval | coverage at $\theta$ is one minus the size of the test of $\theta$ |
What we have learned
The three numbers for the 60 young men, 0.483, 0.465 and 0.477, are the square roots of the score, Wald and likelihood ratio statistics: the slope of the log-likelihood at the null value, the distance from the estimate in estimated standard errors, and the drop in log-likelihood. When the log-likelihood is a parabola they are one number, and in large samples under the null hypothesis each has the $\chi^2_1$ distribution, by the central limit theorem for the score and a Taylor expansion for the other two. Inverted, the score test is Wilson's interval and the Wald test is the textbook interval, which is why the course found the first reliable and the second not: the Wald statistic uses the variance at the estimate for every null value and changes with the scale of the parameter, and the other two do neither.
The likelihood ratio statistic compares a model with a restricted version of itself by the height of the log-likelihood alone. The last article of the unit, The EM Algorithm, is about reaching that height when the likelihood cannot be maximized directly.
In Python
import numpy as np
from scipy import stats
x, n, p0 = 15, 60, 0.224
ph = x / n
loglik = lambda p: x * np.log(p) + (n - x) * np.log(1 - p)
score = n * (ph - p0) ** 2 / (p0 * (1 - p0)) # about 0.2333
wald = n * (ph - p0) ** 2 / (ph * (1 - ph)) # about 0.2163
lr = 2 * (loglik(ph) - loglik(p0)) # about 0.2273
print(stats.chi2.sf([score, wald, lr], df=1)) # about 0.6291, 0.6419, 0.6335
Every number in this article is computed by tools/experiments/statistics/likelihood-ratio-wald-and-score-tests.py in the site's repository, from the NHANES files loaded by proportions.py and statdata.py and the horse-kick table in distdata.py; the figures are drawn by tools/figures/likelihood-ratio-wald-and-score-tests.py.
References
- Bera, A. K., and Bilias, Y. (2001). Rao's score, Neyman's C(α) and Silvey's LM tests: an essay on historical developments and some new results. Journal of Statistical Planning and Inference, 97(1), 9–44. record
- Bera, A. K., and Bilias, Y. (2024). Three score and 15 years (1948–2023) of Rao's score test: a brief history. arXiv:2406.19956 (version 3, 6 October 2024). https://arxiv.org/abs/2406.19956 record
- Buse, A. (1982). The likelihood ratio, Wald, and Lagrange multiplier tests: an expository note. The American Statistician, 36(3), 153–157. record
- Hauck, W. W., and Donner, A. (1977). Wald's test as applied to hypotheses in logit analysis. Journal of the American Statistical Association, 72(360), 851–853. record
- National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018: DEMO_J, BPX_J and BPQ_J data files and documentation. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 67 pages
- NIST/SEMATECH. e-Handbook of Statistical Methods, §8.2.3.3 (Likelihood ratio tests). National Institute of Standards and Technology. https://www.itl.nist.gov/div898/handbook/ record · 34 pages
- Rao, C. R. (1948). Large sample tests of statistical hypotheses concerning several parameters with applications to problems of estimation. Mathematical Proceedings of the Cambridge Philosophical Society, 44(1), 50–57. record
- Self, S. G., and Liang, K.-Y. (1987). Asymptotic properties of maximum likelihood estimators and likelihood ratio tests under nonstandard conditions. Journal of the American Statistical Association, 82(398), 605–610. record
- van der Vaart, A. W. (1998). Asymptotic Statistics. Cambridge University Press. Chapter 16, "Likelihood Ratio Tests". record · 8 pages
- Wald, A. (1943). Tests of statistical hypotheses concerning several parameters when the number of observations is large. Transactions of the American Mathematical Society, 54(3), 426–482. record
- Wilks, S. S. (1938). The large-sample distribution of the likelihood ratio for testing composite hypotheses. The Annals of Mathematical Statistics, 9(1), 60–62. record
- Wilson, E. B. (1927). Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158), 209–212. record · 4 pages