The Neyman–Pearson Lemma

Which test of a given size has the most power. The lemma proved in full for discrete data and for densities, with the randomized rule that discrete data need and the sense in which the best test is unique; the one-sided z test and a binomial test of hypertension among young men as examples; uniformly most powerful one-sided tests under a monotone likelihood ratio, proved; and a proof that no uniformly most powerful two-sided test of a normal mean exists. Worked problems on ten young men, Bortkiewicz's horse-kick deaths, the price of a two-sided test, and a normal null hypothesis against a Cauchy alternative.

Adults aged 60 and over in the National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018 have a mean systolic blood pressure of 136.85 mm Hg (National Center for Health Statistics, 2020). That is 10.49 above the 126.36 of all adults. Type I and Type II Errors tests the false null hypothesis that their mean is 126.36 with samples of 25, treating their standard deviation, 21.02 mm Hg, as known. That 21.02 is the standard deviation of the 1,923 older adults themselves; all 5,247 adults have 20.12. At a Type I error rate of 0.05, three tests of that hypothesis detect the difference with different probabilities under the normal model. A test that rejects when the sample mean is high has power 0.8022. The two-sided $z$ test has 0.7035. A test that counts how many of the 25 readings lie above 126.36 has 0.6204. Is there a fourth test, of the same size, that does better than 0.8022?

There is not, and the reason is a theorem. A test of one fully specified hypothesis against another is most powerful at level $\alpha$ if no other test with Type I error rate at most $\alpha$ has a smaller Type II error rate. The Neyman–Pearson lemma says what such a test looks like: it rejects when the likelihood of the data under the alternative, divided by the likelihood under the null hypothesis, is large. For the blood pressures that ratio is large exactly when the sample mean is large, so the first test is the most powerful one.

First we set out what a test is, including tests that decide some samples by chance. Then we find the likelihood ratio by asking where a fixed budget of Type I error is best spent, and prove the lemma. Next come the two examples, the $z$ test and a binomial test, and a measurement on the NHANES populations. Finally we ask how far the result reaches beyond two fully specified hypotheses: to one-sided hypotheses it extends whenever the model has a monotone likelihood ratio, and to two-sided hypotheses it does not extend at all. Both statements are proved.

The article assumes the two error rates of Type I and Type II Errors and the likelihood of The Likelihood Function, and uses the notation of Likelihood and Estimation. The course page on the two errors states the lemma without proof and points here.

Tests, size and power

The data are a sample $\mathbf{X} = (X_1, \ldots, X_n)$ with observed value $\mathbf{x}$. Its joint probability mass function or joint density is $f(\mathbf{x}; \theta)$, where the parameter $\theta$ is unknown. A hypothesis is simple if it names one value of $\theta$, so that it fixes the distribution of the data completely, and composite if it allows several. The lemma concerns two simple hypotheses,

$$H_0\colon \theta = \theta_0 \qquad \text{against} \qquad H_1\colon \theta = \theta_1,$$

and we write $f_0(\mathbf{x}) = f(\mathbf{x}; \theta_0)$ and $f_1(\mathbf{x}) = f(\mathbf{x}; \theta_1)$ for the two joint distributions. Nothing in the lemma needs them to belong to one parametric family: $f_0$ and $f_1$ may be any two distributions on the same sample space, and the last worked problem takes a normal and a Cauchy distribution.

A test is a rule that says, for each possible sample, whether to reject $H_0$. We describe it by a test function $\psi$, where $\psi(\mathbf{x})$ is the probability of rejecting $H_0$ when $\mathbf{x}$ is observed, so $0 \le \psi(\mathbf{x}) \le 1$. An ordinary test has $\psi(\mathbf{x}) = 1$ on its rejection region and 0 elsewhere. A randomized test also takes values strictly between 0 and 1: on such samples the decision is made by an independent chance device, such as a draw from a table of random numbers, that rejects with probability $\psi(\mathbf{x})$.

The probability that a test rejects, as a function of the parameter, is its power function,

$$\pi_\psi(\theta) = E_\theta[\psi(\mathbf{X})] = \sum_{\mathbf{x}} \psi(\mathbf{x})\, f(\mathbf{x}; \theta) \quad \text{or} \quad \int \psi(\mathbf{x})\, f(\mathbf{x}; \theta)\, d\mathbf{x},$$

a sum for discrete data and an integral over the sample space for data with a density. For the two simple hypotheses, $\pi_\psi(\theta_0) = E_0[\psi(\mathbf{X})]$ is the Type I error rate, called the size of the test, and $\pi_\psi(\theta_1) = E_1[\psi(\mathbf{X})]$ is its power. The course pages write $\beta$ for the Type II error rate, so the power is $1 - \beta$. A test has level $\alpha$ if its size is at most $\alpha$. Throughout, $0 < \alpha < 1$.

Example (a test that needs a chance device). Let $K$ be the number of the 25 blood pressure readings above 126.36. Under the normal model with mean 126.36, each reading is above its mean with probability one half, so $K \sim \text{Bin}(25, 0.5)$ if $H_0$ is true. Rejecting when $K \ge 18$ has size $P_0(K \ge 18) = 0.0216$, and rejecting when $K \ge 17$ has size 0.0539. No cut-off gives 0.05. The randomized test that rejects when $K \ge 18$, and with probability $\gamma$ when $K = 17$, has size $0.0216 + \gamma \times 0.0322$, which is 0.05 for $\gamma = 0.8797$ (computed from the unrounded probabilities). That is the count test of the opening paragraph, and randomizing is what lets it be compared with the other two at exactly the same size.

Proofs on this page are written for discrete data, where expectations are sums, and for data with a joint density, where they are integrals over the sample space. Random Variables and Distribution Functions says what the measure-theoretic treatment would add. One fact is taken from Probability Inequalities: a nonnegative random variable with expected value zero is zero with probability 1.

Where to spend the Type I error

Here is the idea in plain words. Building a test means choosing sample points at which to reject. Putting a point $\mathbf{x}$ into the rejection region costs $f_0(\mathbf{x})$ of size and buys $f_1(\mathbf{x})$ of power. With a budget of $\alpha$ to spend, the points to buy first are those that give the most power per unit of size, the points where the ratio

$$r(\mathbf{x}) = \frac{f_1(\mathbf{x})}{f_0(\mathbf{x})} = \frac{L(\theta_1; \mathbf{x})}{L(\theta_0; \mathbf{x})}$$

is largest. This is the likelihood ratio of the alternative to the null hypothesis: the two joint distributions evaluated at the observed sample are the two likelihoods. The unit reserves the symbol $\Lambda$ for a different ratio, $L(\theta_0)/L(\hat\theta)$, the subject of Likelihood Ratio, Wald and Score Tests. Small values of $\Lambda$ and large values of $r$ both count against $H_0$. Buying points in decreasing order of $r$ until the budget is spent gives a rejection region of the form $r(\mathbf{x}) > k$. If the last point bought would overspend the budget, a fraction of it is bought: that is the randomized rule.

A small case shows the idea at work. Among the 822 men aged 18 to 39 in NHANES, 261 have hypertension, a proportion of 0.3175, while the national estimate for all adults of those ages is 0.224 (Tests and Sample Size for a Proportion). Take 10 young men, let $X$ be the number with hypertension, and test $H_0\colon p = 0.224$ against $H_1\colon p = 0.3175$. The sample space has eleven points:

$x$ $f_0(x)$ $f_1(x)$ $r(x)$
0 0.07918 0.02192 0.277
1 0.22856 0.10200 0.446
2 0.29689 0.21354 0.719
3 0.22854 0.26493 1.159
4 0.11545 0.21570 1.868
5 0.03999 0.12042 3.011
6 0.00962 0.04669 4.853
7 0.00159 0.01241 7.822
8 0.00017 0.00217 12.607
9 0.00001 0.00022 20.320
10 0.00000 0.00001 32.750

The ratio rises with $x$, so the points are bought from the top down: 10, 9, 8, 7, 6 cost 0.0114 of size together. The next point, 5, costs 0.0400, and the budget of 0.05 has only $0.05 - 0.0114 = 0.0386$ left. Buying the fraction $0.0386/0.0400 = 0.9655$ of it (computed from unrounded values) spends the budget exactly. The first worked problem completes this calculation and compares the result with every other test on these eleven points.

The lemma

Theorem (Neyman–Pearson lemma). Let $f_0$ and $f_1$ be the joint mass functions or densities of the data under $H_0$ and $H_1$, and let $0 < \alpha < 1$.

  1. (Existence.) There are constants $k \ge 0$ and $0 \le \gamma \le 1$ such that the test

$$\psi^*(\mathbf{x}) = \begin{cases} 1 & \text{if } f_1(\mathbf{x}) > k\, f_0(\mathbf{x}), \\ \gamma & \text{if } f_1(\mathbf{x}) = k\, f_0(\mathbf{x}), \\ 0 & \text{if } f_1(\mathbf{x}) < k\, f_0(\mathbf{x}) \end{cases}$$

has size exactly $\alpha$.

  1. (Sufficiency.) Let $\psi^*$ be any test for which there is a constant $k \ge 0$ with $\psi^*(\mathbf{x}) = 1$ wherever $f_1(\mathbf{x}) > k f_0(\mathbf{x})$ and $\psi^*(\mathbf{x}) = 0$ wherever $f_1(\mathbf{x}) < k f_0(\mathbf{x})$, whatever its values where $f_1(\mathbf{x}) = k f_0(\mathbf{x})$. Then every test $\psi$ whose size is at most the size of $\psi^*$ has power at most the power of $\psi^*$:

$$E_0[\psi(\mathbf{X})] \le E_0[\psi^*(\mathbf{X})] \quad \Longrightarrow \quad E_1[\psi(\mathbf{X})] \le E_1[\psi^*(\mathbf{X})].$$

  1. (Necessity.) Let $\psi^*$ be a test of the form in part 1 with size $\alpha$ and constant $k$, and let $\psi$ be any most powerful test of level $\alpha$. Then, with probability 1 under $H_0$ and under $H_1$, $\psi(\mathbf{X}) = \psi^*(\mathbf{X})$ on the event $f_1(\mathbf{X}) \ne k f_0(\mathbf{X})$. For discrete data, $\psi(\mathbf{x}) = \psi^*(\mathbf{x})$ at every sample point with $f_1(\mathbf{x}) \ne k f_0(\mathbf{x})$. If $k > 0$, the size of $\psi$ is exactly $\alpha$.

The inequalities are written with products, $f_1 > k f_0$, and not with the ratio, so that points where $f_0(\mathbf{x}) = 0$ need no separate treatment. Where $f_0(\mathbf{x}) > 0$, the condition $f_1(\mathbf{x}) > k f_0(\mathbf{x})$ says $r(\mathbf{x}) > k$. A point with $f_0(\mathbf{x}) = 0 < f_1(\mathbf{x})$ is always rejected: it costs nothing.

Part 2 is the heart of the lemma, and it is three lines long. Here is its idea. At every sample point, the difference $\psi^* - \psi$ and the quantity $f_1 - k f_0$ have the same sign, because $\psi^*$ rejects for certain where $f_1 - k f_0$ is positive and never where it is negative. So their product is nonnegative everywhere, and adding it up over the sample space turns that pointwise fact into a comparison of power with size.

Proof of part 2. Fix a sample point $\mathbf{x}$. If $f_1(\mathbf{x}) > k f_0(\mathbf{x})$, then $\psi^*(\mathbf{x}) = 1 \ge \psi(\mathbf{x})$, so both factors of

$$\big(\psi^*(\mathbf{x}) - \psi(\mathbf{x})\big)\big(f_1(\mathbf{x}) - k f_0(\mathbf{x})\big)$$

are nonnegative. If $f_1(\mathbf{x}) < k f_0(\mathbf{x})$, then $\psi^*(\mathbf{x}) = 0 \le \psi(\mathbf{x})$, so both factors are nonpositive. If $f_1(\mathbf{x}) = k f_0(\mathbf{x})$, the second factor is zero. In every case the product is at least zero. Summing over all sample points (or integrating over the sample space),

$$0 \le \sum_{\mathbf{x}} \big(\psi^* - \psi\big)\big(f_1 - k f_0\big) = \big(E_1[\psi^*] - E_1[\psi]\big) - k\,\big(E_0[\psi^*] - E_0[\psi]\big).$$

Hence

$$E_1[\psi^*(\mathbf{X})] - E_1[\psi(\mathbf{X})] \ \ge\ k\,\big(E_0[\psi^*(\mathbf{X})] - E_0[\psi(\mathbf{X})]\big) \ \ge\ 0,$$

because $k \ge 0$ and the size of $\psi$ is at most that of $\psi^*$. $\square$

Proof of part 1. The task is to choose $k$ and $\gamma$ so that the size comes out as $\alpha$. Under $H_0$ the sample falls where $f_0(\mathbf{x}) > 0$ with probability 1, so the ratio $R = r(\mathbf{X})$ is a well-defined nonnegative random variable under $H_0$, and the size of $\psi^*$ is

$$E_0[\psi^*(\mathbf{X})] = P_0(R > k) + \gamma\, P_0(R = k).$$

Let $G(c) = P_0(R > c)$, which is one minus the distribution function of $R$ under $H_0$. By the properties of distribution functions proved in Random Variables and Distribution Functions, $G$ is nonincreasing and continuous from the right, $G(c) \to 0$ as $c \to \infty$, and its limit from the left at $c$ is $P_0(R \ge c)$. Define

$$k = \inf\{c \ge 0 : G(c) \le \alpha\}.$$

The set is not empty, because $G(c) \to 0 < \alpha$. By continuity from the right, $G(k) \le \alpha$. If $k > 0$, then $G(c) > \alpha$ for every $c < k$, so the limit from the left gives $P_0(R \ge k) \ge \alpha$. If $k = 0$, then $P_0(R \ge 0) = 1 \ge \alpha$. In both cases

$$P_0(R > k) \ \le\ \alpha\ \le\ P_0(R > k) + P_0(R = k).$$

If $P_0(R = k) = 0$, the two ends are equal, the size is $\alpha$ for any $\gamma$, and we take $\gamma = 0$. Otherwise set

$$\gamma = \frac{\alpha - P_0(R > k)}{P_0(R = k)},$$

which lies between 0 and 1 by the displayed inequalities and makes the size $P_0(R > k) + \gamma P_0(R = k) = \alpha$. $\square$

When the sample space is finite, as for a binomial count, the proof needs no limits. The ratio takes finitely many values, and $k$ is the value at which the upper tail probability $P_0(R > k)$ first drops to $\alpha$ or below. Then $\gamma$ is the fraction of the last point's probability that the remaining budget buys, as in the table above.

Proof of part 3. Let $\psi$ be most powerful at level $\alpha$. Then its power is at least that of $\psi^*$, which has level $\alpha$, and by part 2 it is at most that of $\psi^*$, so $E_1[\psi] = E_1[\psi^*]$. Put this into the identity of part 2:

$$\sum_{\mathbf{x}} \big(\psi^* - \psi\big)\big(f_1 - k f_0\big) = 0 - k\,\big(\alpha - E_0[\psi]\big) \ \le\ 0,$$

since $E_0[\psi] \le \alpha$. The left side is a sum of nonnegative terms, so it is also at least zero. Hence it is zero, and two things follow. First, $k(\alpha - E_0[\psi]) = 0$, so if $k > 0$ the size of $\psi$ is exactly $\alpha$. Second, for discrete data every term of the sum is zero, and at a point where $f_1(\mathbf{x}) \ne k f_0(\mathbf{x})$ that forces $\psi(\mathbf{x}) = \psi^*(\mathbf{x})$.

For data with a density, write $h(\mathbf{x}) = (\psi^* - \psi)(f_1 - k f_0) \ge 0$; its integral is zero. Under $H_j$, for $j = 0$ or 1, consider the nonnegative random variable $Y = h(\mathbf{X})/f_j(\mathbf{X})$, defined where $f_j > 0$. Its expected value is the integral of $h$ over the region where $f_j > 0$, which is at most the integral of $h$ over the whole space, namely zero. A nonnegative variable with expected value zero is zero with probability 1 (Probability Inequalities, the lemma on a nonnegative variable with mean zero). So $h(\mathbf{X}) = 0$ with probability 1 under $H_j$, and on the event $f_1(\mathbf{X}) \ne k f_0(\mathbf{X})$ that means $\psi(\mathbf{X}) = \psi^*(\mathbf{X})$. $\square$

Two consequences follow at once.

Corollary (power exceeds size). A most powerful level-$\alpha$ test has power at least $\alpha$, and its power equals $\alpha$ only if $f_0$ and $f_1$ are the same distribution.

Proof. The test that ignores the data and rejects with probability $\alpha$, $\psi(\mathbf{x}) = \alpha$ for every $\mathbf{x}$, has size $\alpha$ and power $\alpha$. A most powerful test has at least that power. Suppose its power is exactly $\alpha$. Then the constant test is itself most powerful, and by part 3 it agrees with $\psi^*$, with probability 1, wherever $f_1 \ne k f_0$. But $\psi^*$ is 0 or 1 there, and the constant test is $\alpha$, strictly between them. So the event $f_1(\mathbf{X}) \ne k f_0(\mathbf{X})$ has probability zero under both hypotheses. Under $H_1$, then, the sample falls with probability 1 where $f_1 = k f_0$, so $1 = P_1(f_1 = k f_0) = k\, P_0(f_1 = k f_0) = k$, the middle step by summing (or integrating) $f_1 = k f_0$ over that set. Hence $k = 1$, and $f_1 = f_0$ except on a set that has probability zero under both hypotheses. $\square$

Corollary (the test depends on a sufficient statistic). If $T(\mathbf{X})$ is a sufficient statistic, the most powerful test can be taken to depend on the data only through $T$. By the factorization theorem of Sufficient Statistics (proved there for discrete data), $f(\mathbf{x}; \theta) = g(T(\mathbf{x}); \theta)\, h(\mathbf{x})$, so wherever $h(\mathbf{x}) > 0$ the comparison of $f_1$ with $k f_0$ is the comparison of $g(T(\mathbf{x}); \theta_1)$ with $k\, g(T(\mathbf{x}); \theta_0)$. That is why the examples below reduce 25 readings to their mean, and 60 yes-or-no outcomes to their count.

The lemma is the main result of Neyman and Pearson's 1933 paper, "On the problem of the most efficient tests of statistical hypotheses". The paper calls a rejection region a critical region, and the region $w_0$ that maximizes the probability $P_1(w_0)$ under the alternative among all regions with the same probability $\varepsilon$ under $H_0$ the "Best Critical Region for $H_0$ with regard to $H_1$" (p. 297). Its Section III shows that such a region is defined by the inequality $p_0 \le k\,p_1$ between the two probability laws: the condition is derived as a necessary one on pp. 298–300 by the calculus of variations, and shown to be sufficient on pp. 300–301 by comparing the region with any other region of the same size $\varepsilon$, the argument of part 2 above. The paper treats a continuous sample space and calls the discrete case "quite analogous as far as the general argument is concerned" (p. 293). Its first example (pp. 302–304) is the $z$ test of the next section: for a normal mean with known standard deviation the best critical region is the same for every alternative above the null value, at $\varepsilon = 0.05$ the region where the sample mean is at least $a_0 + 1.6449\,\sigma_0/\sqrt{n}$, and when the alternatives lie on both sides there is no single best critical region. Lehmann (1992), introducing a reprint of the paper, recounts that Neyman told Pearson in March 1930 that he had solved the problem of maximizing power at a fixed level when both hypotheses are simple, and that the solution became known as the Neyman–Pearson fundamental lemma. The view of testing that the paper argues for, a rule of behavior judged by its two error rates, is described with quotations in Type I and Type II Errors.

The z test is most powerful

Back to the older adults. The model is $X_1, \ldots, X_n$ independent and normal with mean $\mu$ and known variance $\sigma^2$, and the hypotheses are $H_0\colon \mu = \mu_0$ and $H_1\colon \mu = \mu_1$ with $\mu_1 > \mu_0$. The joint density is

$$f(\mathbf{x}; \mu) = (2\pi\sigma^2)^{-n/2} \exp\left\{-\frac{1}{2\sigma^2}\sum_{i=1}^n (x_i - \mu)^2\right\}.$$

In the ratio $f_1/f_0$ the constants cancel, and the exponent is $-\frac{1}{2\sigma^2}\sum[(x_i - \mu_1)^2 - (x_i - \mu_0)^2]$. Each term of the sum is $-2x_i(\mu_1 - \mu_0) + \mu_1^2 - \mu_0^2$, so with $\sum x_i = n\bar{x}$,

$$r(\mathbf{x}) = \exp\left\{\frac{n(\mu_1 - \mu_0)\,\bar{x}}{\sigma^2} - \frac{n(\mu_1^2 - \mu_0^2)}{2\sigma^2}\right\}.$$

It helps to write this with the $z$ statistic $z = (\bar{x} - \mu_0)/(\sigma/\sqrt{n})$ and the standardized difference $\lambda = (\mu_1 - \mu_0)/(\sigma/\sqrt{n})$, the notation of Power and Sample Size for a Test of a Mean. Substituting $\bar{x} = \mu_0 + z\sigma/\sqrt{n}$ and $\mu_1 = \mu_0 + \lambda\sigma/\sqrt{n}$ gives

$$r(\mathbf{x}) = \exp\left\{\lambda z - \tfrac{1}{2}\lambda^2\right\}.$$

Since $\lambda > 0$, the ratio is a strictly increasing function of $z$. So $r(\mathbf{x}) > k$ exactly when $z > c$, where $c = (\ln k + \lambda^2/2)/\lambda$. Under $H_0$, $z$ is standard normal, the event $r = k$ has probability zero, and no randomization is needed. The size is $P_0(z > c) = 1 - \Phi(c)$, which is $\alpha$ for $c = z_{1-\alpha}$. By the lemma, the test that rejects when

$$z \ge z_{1-\alpha}, \qquad \text{that is,} \qquad \bar{x} \ge \mu_0 + z_{1-\alpha}\,\frac{\sigma}{\sqrt{n}},$$

is most powerful at level $\alpha$. Its power is $\Phi(\lambda - z_{1-\alpha})$, the formula derived in Power and Sample Size for a Test of a Mean. The lemma adds that no test of level $\alpha$ exceeds it.

For the older adults the three numbers come from two groups, and it matters which is which.

  • $\mu_0 = 126.36$ is the mean of all 5,247 adults, the value the null hypothesis claims for the older adults.
  • $\mu_1 = 136.85$ is the mean of the 1,923 adults aged 60 and over.
  • $\sigma = 21.02$ is the standard deviation of those 1,923 older adults (computed with divisor $N$). It is not the standard deviation of all adults, which is 20.12. The model assumes the older adults' readings have this spread under either hypothesis, since the hypotheses differ only in the mean.

The sample size is $n = 25$. The standard error is 4.2048 and $\lambda = 2.4944$. The log of the ratio is $0.5932\,(\bar{x} - 126.36) - 3.1109$. At $\alpha = 0.05$ the test rejects when $\bar{x} \ge 126.36 + 1.6449 \times 4.2048 = 133.28$, the point where the ratio reaches $k = 2.6965$, whose logarithm is 0.9919. A sample is rejected when it is at least 2.6965 times as probable under a mean of 136.85 as under a mean of 126.36. The power is $\Phi(2.4944 - 1.6449) = 0.8022$.

The lemma now explains the opening paragraph. Each of these tests has size 0.05 under the normal model:

Test Power at $\mu_1 = 136.85$
Reject when $z \ge 1.645$ (most powerful) 0.8022
Two-sided: reject when $\lvert z \rvert \ge 1.960$ 0.7035
$z$ test on the first 16 readings only 0.6371
Count of readings above 126.36: reject when $K \ge 18$, with probability 0.8797 when $K = 17$ 0.6204

The two-sided test spends half its budget on samples with low means, where the ratio is smallest. The third test throws away nine readings. The count test keeps only the sign of each reading's deviation and so discards how far each reading lies from 126.36. Under the alternative a reading exceeds 126.36 with probability $\Phi(10.49/21.02) = 0.6911$, and the power is the corresponding tail of $\text{Bin}(25, 0.6911)$. None of the three rejection regions has the form $r(\mathbf{x}) > k$, and part 3 of the lemma says that every such test must fall short.

Example (the cut-off does not depend on the alternative). The rejection region $z \ge 1.645$ was derived for $\mu_1 = 136.85$, but $\mu_1$ does not appear in it. It entered only through $\lambda > 0$, which made the ratio increasing in $z$. The same region is therefore most powerful against a mean of 130, or 150, or any other value above 126.36. A test that is most powerful against every alternative at once is the subject of the section on one-sided hypotheses below.

A binomial test and the need to randomize

The count of hypertensive men among 60 young men is the example of Tests and Sample Size for a Proportion: $X \sim \text{Bin}(60, p)$, with the national figure $p_0 = 0.224$ as the null value and the young men's true proportion $p_1 = 0.3175$ as the alternative. The likelihood ratio is

$$r(x) = \frac{p_1^x (1 - p_1)^{60 - x}}{p_0^x (1 - p_0)^{60 - x}} = \left(\frac{1 - p_1}{1 - p_0}\right)^{60} \left[\frac{p_1 (1 - p_0)}{p_0 (1 - p_1)}\right]^{x}.$$

The binomial coefficients cancel. The bracket is the odds ratio of the two proportions, 1.6117. It is greater than 1 because $p_1 > p_0$, so each additional hypertensive man multiplies the ratio by 1.6117, and $r(x)$ is strictly increasing in $x$. The most powerful test rejects for large counts.

The count takes whole values, so the available sizes are the upper tail probabilities of $\text{Bin}(60, 0.224)$:

Rejection region Size, $P_0$ Power, $P_1$
$X \ge 21$ 0.0177 0.3389
$X \ge 20$ 0.0344 0.4439
$X \ge 19$ 0.0626 0.5541

No region has size 0.05. The construction in the proof of part 1 gives $k = r(19) = 3.9111$, since $P_0(r > 3.9111) = P_0(X \ge 20) = 0.0344 \le 0.05 \le 0.0626$, and

$$\gamma = \frac{0.05 - 0.0344}{P_0(X = 19)} = \frac{0.0156}{0.0282} = 0.5535.$$

The most powerful test of size exactly 0.05 rejects when $X \ge 20$ and, when $X = 19$, rejects with probability 0.5535. Its power is $0.4439 + 0.5535 \times 0.1102 = 0.5049$, where $0.1102 = P_1(X = 19)$. (The quotient is computed from unrounded values; the rounded ones shown give 0.553.)

By part 2, each of the three nonrandomized regions in the table is also most powerful, at its own size. The region $X \ge 20$ is the best test of size 0.0344. What the lemma shows is that the best test of size 0.05 is not any of them.

For comparison, a test that guards against both directions, rejecting in each tail with probability 0.025 under $H_0$ (randomizing at the counts 7 and 20 to make each tail exact), also has size 0.05. Its power at $p_1 = 0.3175$ is 0.3848. As with the two-sided $z$ test, the budget spent on low counts buys nothing against an alternative above $p_0$.

Size and power of all 2,048 nonrandomized tests of p = 0.224 against p = 0.3175 based on the number of hypertensive men among 10 (dots), with the upper boundary traced by the randomized Neyman-Pearson tests (solid line with diamonds); the dotted diagonal marks power equal to size, and the right panel magnifies the corner near a size of 0.05 (dashed). Notice that no dot lies above the line, that the corners of the line are the twelve tests that reject when the count is at least some value, and that at a size of 0.05 the line passes between two corners: the most powerful test of that size, with power 0.1778, has to randomize, and the best nonrandomized test, the region 5, 6, 8, 9, 10, sits just below the line.
Size and power of all 2,048 nonrandomized tests of p = 0.224 against p = 0.3175 based on the number of hypertensive men among 10 (dots), with the upper boundary traced by the randomized Neyman-Pearson tests (solid line with diamonds); the dotted diagonal marks power equal to size, and the right panel magnifies the corner near a size of 0.05 (dashed). Notice that no dot lies above the line, that the corners of the line are the twelve tests that reject when the count is at least some value, and that at a size of 0.05 the line passes between two corners: the most powerful test of that size, with power 0.1778, has to randomize, and the best nonrandomized test, the region 5, 6, 8, 9, 10, sits just below the line.

The figure shows the whole picture for the smaller sample of 10 men, where every test can be listed. There are $2^{11} = 2{,}048$ ways to choose a rejection region among the eleven counts. Each is a dot at its size and power. The randomized Neyman–Pearson tests trace the upper boundary. The script confirms that none of the 2,048 tests lies above it, and that exactly twelve lie on it: the regions $X \ge c$ for $c = 0, 1, \ldots, 11$, from "always reject" to "never reject". Between two neighboring corners the boundary is a straight segment, because the randomized test with probability $\gamma$ at the cut-off is a mixture of the two corner tests, and both size and power are linear in $\gamma$.

Example (what randomization is for). Nobody reports that a hypothesis was rejected by a random number, and the lemma does not ask for it. In practice the conservative region is used, here $X \ge 20$ with size 0.0344, or a $p$-value is reported and the reader is left to judge. The one-sided exact binomial test of Tests and Sample Size for a Proportion, which rejects when the tail probability $P_0(X \ge x)$ is at most 0.05, is the region $X \ge 20$. Randomized tests matter for the theory. They make every size attainable, so that tests can be compared at equal size, and they make the set of attainable pairs of size and power a region with a straight-edged upper boundary, as in the figure. Without them the best test of level 0.05 can be a strange object: the first worked problem finds that for 10 men it is the region $\{5, 6, 8, 9, 10\}$, which skips the count 7.

One-sided hypotheses: uniformly most powerful tests

Two simple hypotheses are rare in practice. The question about young men is whether their proportion is higher than 0.224, not whether it is 0.3175. Does the lemma say anything about $H_0\colon p \le 0.224$ against $H_1\colon p > 0.224$?

It does, for a reason already visible in both examples: the most powerful test did not depend on which alternative was chosen. For hypotheses $H_0\colon \theta \in \Theta_0$ and $H_1\colon \theta \in \Theta_1$, the size of a test is the largest Type I error rate over the null hypothesis, $\sup_{\theta \in \Theta_0} \pi_\psi(\theta)$, and the test has level $\alpha$ if its size is at most $\alpha$. A level-$\alpha$ test $\psi$ is uniformly most powerful (UMP) if, for every $\theta_1 \in \Theta_1$ and every level-$\alpha$ test $\psi'$, $\pi_\psi(\theta_1) \ge \pi_{\psi'}(\theta_1)$. A UMP test is the most powerful test against each alternative separately, all at once. Nothing guarantees that such a test exists. The property that produces one is the following.

Let the parameter $\theta$ range over an interval $\Theta$ of real numbers, and suppose the set $S$ of sample points with $f(\mathbf{x}; \theta) > 0$ is the same for every $\theta$. The family has a monotone likelihood ratio in the statistic $T(\mathbf{x})$ if, for every pair $\theta' < \theta''$ in $\Theta$, there is a nondecreasing function $g_{\theta', \theta''}$ with

$$\frac{f(\mathbf{x}; \theta'')}{f(\mathbf{x}; \theta')} = g_{\theta', \theta''}\big(T(\mathbf{x})\big) \qquad \text{for every } \mathbf{x} \in S.$$

In words: larger values of $T$ always favor larger values of $\theta$, whichever two values are compared. Three families used throughout the course have the property.

  • Binomial. For $X \sim \text{Bin}(n, p)$ and $p' < p''$, the calculation above gives the ratio $[(1 - p'')/(1 - p')]^n\, \omega^x$ with $\omega = p''(1 - p')/[p'(1 - p'')] > 1$, which increases with $T = x$.
  • Poisson. For a sample from the Poisson distribution with mean $\lambda$, $f(\mathbf{x}; \lambda) = e^{-n\lambda} \lambda^{\sum x_i} / \prod x_i!$, so for $\lambda' < \lambda''$ the ratio is $e^{-n(\lambda'' - \lambda')} (\lambda''/\lambda')^{\sum x_i}$, which increases with $T = \sum x_i$.
  • Normal mean, known variance. For $\mu' < \mu''$ the ratio is $\exp\{n(\mu'' - \mu')\bar{x}/\sigma^2 - n(\mu''^2 - \mu'^2)/(2\sigma^2)\}$, by the calculation of the previous section with $\mu'$ and $\mu''$ in place of $\mu_0$ and $\mu_1$. It increases with $T = \bar{x}$.

Example (a family without the property). The Cauchy distribution with center $\theta$ has density $1/\{\pi[1 + (x - \theta)^2]\}$. For one observation and $\theta' < \theta''$ the ratio is $[1 + (x - \theta')^2]/[1 + (x - \theta'')^2]$. At $x = \theta''$ it equals $1 + (\theta'' - \theta')^2 > 1$, and as $x \to \infty$ it tends to 1. So it is not nondecreasing in $x$, and the Cauchy family has no monotone likelihood ratio in $x$: a very large observation is weaker evidence for the larger center than a moderate one.

Theorem (uniformly most powerful one-sided tests). Suppose the family has a monotone likelihood ratio in $T$. Let $c$ be a value that $T$ takes on $S$, let $0 \le \gamma \le 1$, and let

$$\psi(\mathbf{x}) = \begin{cases} 1 & \text{if } T(\mathbf{x}) > c, \\ \gamma & \text{if } T(\mathbf{x}) = c, \\ 0 & \text{if } T(\mathbf{x}) < c. \end{cases}$$

Then:

  1. the power function $\pi_\psi(\theta)$ is nondecreasing in $\theta$;
  2. for any $\theta_0 \in \Theta$, with $\alpha = \pi_\psi(\theta_0)$, the test $\psi$ is uniformly most powerful of level $\alpha$ for $H_0\colon \theta \le \theta_0$ against $H_1\colon \theta > \theta_0$.

Here is the plan of the proof. For any two parameter values, the test $\psi$ has the Neyman–Pearson form, so it is most powerful for that pair. Comparing it with the test that ignores the data shows its power can only rise with $\theta$. That settles the size over the whole null hypothesis, and the lemma applied at $\theta_0$ against each alternative gives the rest.

Proof. Step 1: $\psi$ has the Neyman–Pearson form for every pair. Fix $\theta' < \theta''$, write $g$ for $g_{\theta', \theta''}$, and let $k = g(c)$, a positive number because both densities are positive on $S$. Take $\mathbf{x} \in S$. If $f(\mathbf{x}; \theta'') > k f(\mathbf{x}; \theta')$, then $g(T(\mathbf{x})) > g(c)$. Since $g$ is nondecreasing, $T(\mathbf{x}) \le c$ would give $g(T(\mathbf{x})) \le g(c)$, so $T(\mathbf{x}) > c$ and $\psi(\mathbf{x}) = 1$. If $f(\mathbf{x}; \theta'') < k f(\mathbf{x}; \theta')$, then $g(T(\mathbf{x})) < g(c)$. Since $T(\mathbf{x}) \ge c$ would give $g(T(\mathbf{x})) \ge g(c)$, we have $T(\mathbf{x}) < c$ and $\psi(\mathbf{x}) = 0$. A point outside $S$ has both densities equal to zero and satisfies neither strict inequality. So $\psi$ meets the condition of part 2 of the lemma for testing $\theta'$ against $\theta''$, with this $k$. Every test whose size at $\theta'$ is at most $\pi_\psi(\theta')$ therefore has power at $\theta''$ at most $\pi_\psi(\theta'')$.

Step 2: the power function is nondecreasing. The constant test that rejects with probability $\pi_\psi(\theta')$ whatever the data has size $\pi_\psi(\theta')$ at $\theta'$ and power $\pi_\psi(\theta')$ at $\theta''$. By step 1, $\pi_\psi(\theta') \le \pi_\psi(\theta'')$. This proves statement 1.

Step 3: $\psi$ has level $\alpha$ for the composite null hypothesis. By step 2, $\pi_\psi(\theta) \le \pi_\psi(\theta_0) = \alpha$ for every $\theta \le \theta_0$, so $\sup_{\theta \le \theta_0} \pi_\psi(\theta) = \alpha$.

Step 4: no level-$\alpha$ test has more power at any alternative. Let $\psi'$ be any test of level $\alpha$ for $H_0\colon \theta \le \theta_0$, and let $\theta_1 > \theta_0$. In particular $\pi_{\psi'}(\theta_0) \le \alpha = \pi_\psi(\theta_0)$. By step 1 with $\theta' = \theta_0$ and $\theta'' = \theta_1$, $\pi_{\psi'}(\theta_1) \le \pi_\psi(\theta_1)$. Since $\theta_1 > \theta_0$ was arbitrary, $\psi$ is uniformly most powerful. $\square$

The condition that $c$ be a value of $T$ loses nothing in the examples. A binomial or Poisson count takes whole values, and a test with a cut-off between two of them is the test with $c$ at the lower one and $\gamma = 0$. For a normal mean every real $c$ is a value of $\bar{x}$. A cut-off and a $\gamma$ giving any chosen size $\alpha$ with $0 < \alpha < 1$ at $\theta_0$ exist in these examples by the argument of part 1 of the lemma, with one change: $T$ may be negative, so the infimum is taken over all real numbers, $c = \inf\{t : P_{\theta_0}(T > t) \le \alpha\}$. The set is not empty and is bounded below, because $P_{\theta_0}(T > t)$ tends to 0 as $t \to \infty$ and to 1 as $t \to -\infty$. Continuity from the right gives $P_{\theta_0}(T > c) \le \alpha$. Every $t < c$ has $P_{\theta_0}(T > t) > \alpha$, so the limit from the left gives $P_{\theta_0}(T \ge c) \ge \alpha$. If $P_{\theta_0}(T = c) > 0$, take $\gamma = [\alpha - P_{\theta_0}(T > c)]/P_{\theta_0}(T = c)$, which lies between 0 and 1; otherwise the size is already $\alpha$ and $\gamma = 0$. For a discrete $T$ such as a count, $P_{\theta_0}(T > t)$ is constant between the values that $T$ takes with positive probability and changes only at those values. So $c$ is one of them, and for a count it is a whole number.

The mirror image. For $H_0\colon \theta \ge \theta_0$ against $H_1\colon \theta < \theta_0$, apply the theorem to the same family indexed by $\eta = -\theta$ with the statistic $-T$. For $\eta' < \eta''$, that is, $-\eta'' < -\eta'$, the ratio of the density at $\eta''$ to the density at $\eta'$ is $1/g_{-\eta'', -\eta'}(T(\mathbf{x}))$. This is a nondecreasing function of $-T(\mathbf{x})$, because $g$ is positive and nondecreasing in $T$. So the family has a monotone likelihood ratio in $-T$, and the theorem gives: the test that rejects when $T < c$, and with probability $\gamma$ when $T = c$, has a nonincreasing power function in $\theta$ and is UMP for $H_0\colon \theta \ge \theta_0$ against $H_1\colon \theta < \theta_0$ at level $\pi_\psi(\theta_0)$.

Back to the 60 young men. The test that rejects when $X \ge 20$, and with probability 0.5535 when $X = 19$, was most powerful against $p_1 = 0.3175$. By the theorem it is uniformly most powerful of level 0.05 for $H_0\colon p \le 0.224$ against $H_1\colon p > 0.224$. Its power function, computed exactly, rises through 0.05 at the boundary of the null hypothesis, as statement 1 requires:

$p$ 0.15 0.20 0.224 0.25 0.30 0.3175 0.35 0.40
$\pi_\psi(p)$ 0.0006 0.0170 0.0500 0.1235 0.3895 0.5049 0.7058 0.9080

The script checks that the function is increasing at 999 values of $p$ between 0.001 and 0.999. By the same theorem, applied to the normal family with $T = \bar{x}$, the one-sided $z$ test is UMP for $H_0\colon \mu \le \mu_0$ against $H_1\colon \mu > \mu_0$ when $\sigma$ is known, the statement made without proof in Type I and Type II Errors.

The decision theory of families with a monotone likelihood ratio was developed by Karlin and Rubin (1956). Their paper defines the property (p. 273), takes the normal, binomial and Poisson families as examples, and treats the one-sided testing problem as one of a wider class of decision problems; for that problem in exponential families it cites earlier results of Allen and of Sobel and Chernoff, which it "unifies, extends, and strengthens" (p. 272). Lehmann and Romano (2005, Section 3.4, "Distributions with Monotone Likelihood Ratio") give the textbook treatment; we cite the section by its title in the book's table of contents and did not have its text open. The proof above assumes a common support $S$. Families whose support moves with the parameter, such as the uniform distribution on $(0, \theta)$, are not covered by it.

Why no uniformly most powerful two-sided test exists

Example (a common overstatement). "The likelihood ratio test is the most powerful test." The lemma says this for two simple hypotheses, and the theorem above extends it to one-sided hypotheses in families with a monotone likelihood ratio. For the most common hypothesis of all, $H_0\colon \mu = \mu_0$ against $H_1\colon \mu \ne \mu_0$, no test is most powerful against every alternative. The two-sided $z$ test had power 0.7035 against a mean of 136.85, where the upper-tailed test has 0.8022.

The reason is simple to state. Against an alternative above $\mu_0$ the best test puts its whole rejection region in the upper tail, and against one below $\mu_0$ in the lower tail. A UMP test would have to do both. The proof makes this precise, and the only work in it is to show that a test matching the upper-tailed test at one alternative above $\mu_0$ must behave like the upper-tailed test everywhere, in particular below $\mu_0$.

Write $\psi^+$ for the upper-tailed test, which rejects when $z \ge z_{1-\alpha}$, and for any mean $\mu$ let $\lambda_\mu = (\mu - \mu_0)/(\sigma/\sqrt{n})$. Under mean $\mu$ the statistic $z$ is normal with mean $\lambda_\mu$ and variance 1, so $\pi_{\psi^+}(\mu) = \Phi(\lambda_\mu - z_{1-\alpha})$.

Proposition. In the normal model with known variance, let $\psi$ be a test of level $\alpha$ for $H_0\colon \mu = \mu_0$ whose power at some $\mu_1 > \mu_0$ equals that of $\psi^+$. Then $\pi_\psi(\mu) = \pi_{\psi^+}(\mu) = \Phi(\lambda_\mu - z_{1-\alpha})$ for every $\mu$.

Here is the idea. The proof of part 3 of the lemma shows that $\psi$ and $\psi^+$ can differ only on a set that contributes nothing to a certain integral. The work is to show that such a set contributes nothing to the power at any other mean either. That is done by bounding the integrand separately away from the cut-off, near the cut-off and far out in the tails.

Proof. Let $k = \exp\{\lambda_{\mu_1} z_{1-\alpha} - \lambda_{\mu_1}^2/2\}$, the constant for which $\psi^+$ is the Neyman–Pearson test of $\mu_0$ against $\mu_1$, and let $h = (\psi^+ - \psi)(f_1 - k f_0)$ with $f_j = f(\,\cdot\,; \mu_j)$. By the proof of part 2 of the lemma, the two factors of $h$ never have opposite signs, so $h = \lvert \psi^+ - \psi \rvert\,\lvert f_1 - k f_0 \rvert \ge 0$ at every sample point. Its integral over the sample space is $(E_1[\psi^+] - E_1[\psi]) - k(\alpha - E_0[\psi]) = -k(\alpha - E_0[\psi]) \le 0$, because the two powers are equal and $\psi$ has level $\alpha$. The integral of a nonnegative function cannot be negative, so it is zero.

Fix any $\mu$. Then

$$\lvert \pi_{\psi^+}(\mu) - \pi_\psi(\mu) \rvert \ \le\ \int \lvert \psi^+ - \psi \rvert\, f(\mathbf{x}; \mu)\, d\mathbf{x}.$$

By the formula for the ratio, $f(\mathbf{x}; \mu) = f_0(\mathbf{x}) \exp\{\lambda_\mu z - \lambda_\mu^2/2\}$ and $f_1(\mathbf{x}) - k f_0(\mathbf{x}) = f_0(\mathbf{x})\,[\exp\{\lambda_{\mu_1} z - \lambda_{\mu_1}^2/2\} - k]$, where $z$ is the $z$ statistic of the sample $\mathbf{x}$. The bracket is zero only at $z = z_{1-\alpha}$. So at every sample with $z \ne z_{1-\alpha}$,

$$\lvert \psi^+ - \psi \rvert\, f(\mathbf{x}; \mu) = h(\mathbf{x})\, w(z), \qquad w(z) = \frac{\exp\{\lambda_\mu z - \lambda_\mu^2/2\}}{\big\lvert \exp\{\lambda_{\mu_1} z - \lambda_{\mu_1}^2/2\} - k \big\rvert},$$

and $w$ is a continuous function of $z$ away from $z_{1-\alpha}$. Take any $\varepsilon > 0$ and any $B > 0$, and split the sample space into the part $A$ where $\lvert z - z_{1-\alpha} \rvert \ge \varepsilon$ and $\lvert z \rvert \le B$, and the rest. On $A$ the values of $z$ lie in a closed bounded set on which $w$ is continuous, so $w \le M$ there for some number $M$, and

$$\int_A \lvert \psi^+ - \psi \rvert\, f(\mathbf{x}; \mu)\, d\mathbf{x} = \int_A h\, w \ \le\ M \int_A h \ \le\ M \int h = 0.$$

Off $A$, $\lvert \psi^+ - \psi \rvert \le 1$, so the integral there is at most the probability, under mean $\mu$, that $\lvert z - z_{1-\alpha} \rvert < \varepsilon$ or $\lvert z \rvert > B$. The first event has probability at most $2\varepsilon/\sqrt{2\pi}$, because the density of $z$ never exceeds $1/\sqrt{2\pi}$. The second has probability at most $P(\lvert Z \rvert > B - \lvert \lambda_\mu \rvert)$ for a standard normal $Z$. Altogether

$$\lvert \pi_{\psi^+}(\mu) - \pi_\psi(\mu) \rvert \ \le\ \frac{2\varepsilon}{\sqrt{2\pi}} + P(\lvert Z \rvert > B - \lvert \lambda_\mu \rvert)$$

for every $\varepsilon > 0$ and $B > 0$. The right side can be made as small as we like by taking $\varepsilon$ small and $B$ large, so the left side is zero. $\square$

Theorem (no UMP two-sided test). In the normal model with known variance there is no uniformly most powerful test of level $\alpha$ for $H_0\colon \mu = \mu_0$ against $H_1\colon \mu \ne \mu_0$.

Proof. Suppose $\psi$ were one. Pick any $\mu_1 > \mu_0$. The test $\psi^+$ has level $\alpha$ and is most powerful against $\mu_1$, so a UMP test must have the same power there, and the proposition gives $\pi_\psi(\mu) = \Phi(\lambda_\mu - z_{1-\alpha})$ for every $\mu$. Now pick any $\mu_2 < \mu_0$. Then $\lambda_{\mu_2} < 0$ and $\pi_\psi(\mu_2) = \Phi(\lambda_{\mu_2} - z_{1-\alpha}) < \Phi(-z_{1-\alpha}) = \alpha$. But the constant test that rejects with probability $\alpha$ has level $\alpha$ and power $\alpha$ at $\mu_2$, which is more. So $\psi$ is not most powerful against $\mu_2$, a contradiction. $\square$

For the older adults the three natural tests at $\alpha = 0.05$ compare as follows.

True mean Upper-tailed, $z \ge 1.645$ Lower-tailed, $z \le -1.645$ Two-sided, $\lvert z \rvert \ge 1.960$
115.87 ($\mu_0 - 10.49$) 0.00002 0.8022 0.7035
126.36 ($\mu_0$) 0.0500 0.0500 0.0500
136.85 ($\mu_0 + 10.49$) 0.8022 0.00002 0.7035
Power functions of three z tests of a mean of 126.36 mm Hg at level 0.05, for samples of 25 with the older adults' standard deviation, 21.02. Notice that each one-sided test is the highest curve on its own side of the null value and falls below 0.05 on the other side, and that the two-sided test (solid) is never the highest curve but never falls below 0.05. No single curve is on top everywhere, which is what a uniformly most powerful test would need.
Power functions of three z tests of a mean of 126.36 mm Hg at level 0.05, for samples of 25 with the older adults' standard deviation, 21.02. Notice that each one-sided test is the highest curve on its own side of the null value and falls below 0.05 on the other side, and that the two-sided test (solid) is never the highest curve but never falls below 0.05. No single curve is on top everywhere, which is what a uniformly most powerful test would need.

Each one-sided test is best on its side and worse than guessing on the other. The two-sided test is second on both sides. It has a different virtue. A test is unbiased if its power is at least $\alpha$ at every alternative, so that it is never more likely to reject a true null hypothesis than a false one. The one-sided tests are biased as tests of $\mu \ne \mu_0$. The two-sided $z$ test is unbiased: its power is $\Phi(\lambda - z_{1-\alpha/2}) + \Phi(-\lambda - z_{1-\alpha/2})$ with $\lambda = \lambda_\mu$, and the derivative of this with respect to $\lambda$ is $\phi(\lambda - z_{1-\alpha/2}) - \phi(\lambda + z_{1-\alpha/2})$, where $\phi$ is the standard normal density. For $\lambda > 0$ the point $\lambda - z_{1-\alpha/2}$ is closer to zero than $\lambda + z_{1-\alpha/2}$, so the derivative is positive and the power increases. Replacing $\lambda$ by $-\lambda$ exchanges the two terms of the power and leaves their sum unchanged, so the power at $-\lambda$ equals the power at $\lambda$. The power is therefore smallest at $\lambda = 0$, where it is $\alpha$.

Restricting attention to unbiased tests removes the one-sided tests from the competition, and within that smaller class a best test can exist. The stronger statement, that the two-sided $z$ test is uniformly most powerful among unbiased tests, belongs to the theory of unbiased tests (Lehmann and Romano, 2005, Chapter 4, "Unbiasedness: Theory and First Applications", whose Section 4.2 is titled "One-Parameter Exponential Families") and is not proved here. For the $t$ test the corresponding statement is cited in Type I and Type II Errors.

For discrete data the conclusion is the same, and part 3 of the lemma gives it directly.

Theorem (no UMP two-sided test of a proportion). For $X \sim \text{Bin}(n, p)$ with $n \ge 1$ and $0 < p_0 < 1$, there is no uniformly most powerful test of level $\alpha$ for $H_0\colon p = p_0$ against $H_1\colon p \ne p_0$.

Proof. Suppose $\psi$ were one. Take any $p_1 > p_0$. The ratio $r(x)$ of $p_1$ to $p_0$ is positive and strictly increasing in $x$. So the Neyman–Pearson test of size $\alpha$ rejects when $X > c^+$ and with some probability when $X = c^+$, for a count $c^+$ between 0 and $n$, and its constant is $k = r(c^+) > 0$. The only count at which $f_1 = k f_0$ is $c^+$. Since $\psi$ is most powerful against $p_1$, part 3 of the lemma gives

$$\psi(x) = 1 \ \text{for } x > c^+, \qquad \psi(x) = 0 \ \text{for } x < c^+, \qquad E_0[\psi] = \alpha.$$

Now take any $p_2 < p_0$. The ratio of $p_2$ to $p_0$ is strictly decreasing in $x$, so the Neyman–Pearson test rejects when $X < c^-$ and with some probability when $X = c^-$, for a count $c^-$ between 0 and $n$. Part 3 gives

$$\psi(x) = 1 \ \text{for } x < c^-, \qquad \psi(x) = 0 \ \text{for } x > c^-.$$

If some count $x$ is larger than both $c^+$ and $c^-$, the first display says $\psi(x) = 1$ and the second says $\psi(x) = 0$, a contradiction. So the larger of $c^+$ and $c^-$ is $n$. If some count is smaller than both, the first display says $\psi(x) = 0$ and the second says $\psi(x) = 1$, again a contradiction. So the smaller of the two is 0. Two cases remain. If $c^+ = n$ and $c^- = 0$, then $\psi(x) = 0$ for every $x < n$ by the first display and for every $x > 0$ by the second. Since $n \ge 1$, that covers every count, so $\psi$ never rejects and its size is 0. If $c^+ = 0$ and $c^- = n$, then $\psi(x) = 1$ for every $x > 0$ and for every $x < n$, so $\psi$ always rejects and its size is 1. Both cases contradict $E_0[\psi] = \alpha$ with $0 < \alpha < 1$. $\square$

For models with more than one parameter, or without a monotone likelihood ratio, even one-sided UMP tests are the exception. That is why practice relies on a general recipe built from the same ratio, with the unknown parameters replaced by their estimates. It is the subject of Likelihood Ratio, Wald and Score Tests.

What the lemma promises on real data

The lemma is a statement about a model. The powers above assume that blood pressures are normal, and the NHANES readings are skewed to the right. How do the tests behave when samples are drawn from the actual population?

The script drew samples of 25, with replacement, from two populations: the 1,923 adults aged 60 and over, whose mean is 136.85, and the same 1,923 values shifted down by 10.49 so that their mean is exactly 126.36. The second is a population in which the null hypothesis is true and the shape and standard deviation are those of the first. For each of 30 seeds it drew 10,000 samples from each population and recorded how often the $z$ test ($\bar{x} \ge 133.28$) and the count test rejected. The table gives the median over the seeds, with the interquartile range.

Test Null population Older adults Normal model
$z$ test 0.0550 (0.0529 to 0.0560) 0.8008 (0.7982 to 0.8033) 0.05 and 0.8022
Count test 0.0131 (0.0123 to 0.0139) 0.5000 (0.4976 to 0.5020) 0.05 and 0.6204

In this experiment the $z$ test kept nearly the size and power the normal model gives it. The sample mean of 25 readings is close to normal even though single readings are not, as the central limit theorem leads one to expect. The theorem is a statement about a limit and does not say that 25 readings are enough; the observed size, 0.0550, is the evidence that they nearly are here, with the skewness leaving the size a little above 0.05. The count test moved much more. Its size and power depend on the share of the population above 126.36, which the normal model puts at 0.5 under the null hypothesis and 0.6911 under the alternative. In the skewed populations the shares are 0.4415 and 0.6625, because the mean of a right-skewed distribution lies above its median. With those shares the binomial distribution gives the count test a size of 0.0128 and a power of 0.4999, in agreement with the simulation. The count test is in effect a test about the median, and the null population's median is below 126.36.

The ranking of the two tests is the same in the model and in the populations, but the lemma did not guarantee that. Its guarantee is exact for the distributions $f_0$ and $f_1$ that were written down, and for data from other distributions it is only as good as the model.

What is proved here, what is cited, and what the experiment shows

Proved on this page:

  • the three parts of the Neyman–Pearson lemma: existence with randomization, sufficiency, and necessity, for discrete data and for densities;
  • the two corollaries;
  • that the one-sided $z$ test and the upper-tailed binomial test are most powerful;
  • the theorem on uniformly most powerful one-sided tests for families with a common support and a monotone likelihood ratio, and its mirror image;
  • the proposition and the two theorems showing that no UMP two-sided test exists for a normal mean or for a binomial proportion;
  • the unbiasedness of the two-sided $z$ test.

Taken from other pages:

Cited, not proved: that the two-sided $z$ test is uniformly most powerful among unbiased tests.

Not covered: families whose support depends on the parameter.

Computed exactly by the script: every binomial, Poisson and normal probability on the page; the size and power of all 2,048 nonrandomized tests for 10 men.

Observed in one experiment: the rejection rates of the two tests in samples from the NHANES populations. They describe those two populations at $n = 25$ and nothing more general.

Worked problems

The four problems are posed on this article's own examples and on the horse-kick data of The Poisson Distribution, to exercise the lemma with randomization, the one-sided theorem, the two-sided result and the lemma outside a parametric family once each. The article's script recomputes every number.

Problem 1 (ten young men). Let $X$ be the number with hypertension among 10 young men, $X \sim \text{Bin}(10, p)$. (a) Find the most powerful test of size 0.05 of $H_0\colon p = 0.224$ against $H_1\colon p = 0.3175$, and its power. (b) Find the most powerful test of level 0.05 that does not randomize and rejects for large counts. (c) Among all nonrandomized tests of level 0.05, which has the most power?

Solution. (a) The ratio $r(x)$ increases with $x$ (the table in the section "Where to spend the Type I error"), so the test rejects for large $x$. From that table,

$$P_0(X \ge 6) = 0.00962 + 0.00159 + 0.00017 + 0.00001 + 0.00000 = 0.0114, \qquad P_0(X \ge 5) = 0.0114 + 0.0400 = 0.0514.$$

Since $0.0114 \le 0.05 \le 0.0514$, the constant is $k = r(5) = 3.011$ and

$$\gamma = \frac{0.05 - P_0(X \ge 6)}{P_0(X = 5)} = 0.9655$$

(from unrounded values). The test rejects when $X \ge 6$ and, when $X = 5$, with probability 0.9655. Its power is

$$P_1(X \ge 6) + \gamma\, P_1(X = 5) = 0.0615 + 0.9655 \times 0.1204 = 0.1778$$

(again from unrounded values; the rounded ones shown give 0.1777).

Ten men are far too few: even the best test detects the difference less than one time in five.

(b) Without randomization the cut-off must be a whole number, and $X \ge 5$ has size 0.0514, just above 0.05. So the region is $X \ge 6$, with size 0.0114 and power 0.0615. By part 2 of the lemma it is the most powerful test of size 0.0114, but it uses less than a quarter of the allowed 0.05.

(c) The best nonrandomized test of level 0.05 rejects when $X$ is 5, 6, 8, 9 or 10: it has size 0.0498 and power 0.1695. The script found it by listing all 2,048 regions. The reason is visible in the table. The region $X \ge 5$ overspends by $0.0514 - 0.05 = 0.0014$, and removing the count 7, which costs 0.00159 of size, brings the size down to 0.0498 while giving up only 0.0124 of power. Removing 8, 9 and 10 instead would save too little size (0.00018), and removing 6 would give up more power (0.0467). This region is not of the Neyman–Pearson form, since it rejects at 6 and 8 but not at 7, and it is not most powerful at its own size either: the randomized test of size 0.0498 has power 0.1771. It is only the best that can be done at level 0.05 without a chance device, and it shows why the theory admits randomized tests: with them the best test always has the simple form "reject for large values of the ratio".

Problem 2 (horse kicks in one corps). Bortkiewicz's data, described in The Poisson Distribution, give the number of soldiers killed by horse kicks in each of 14 Prussian army corps in each of the 20 years 1875 to 1894. In the ten corps of standard composition there were 122 deaths in 200 corps-years, a rate of 0.61 per corps per year. Treat that rate as known. For one of the other corps, model the 20 yearly counts as independent Poisson variables with mean $\lambda$. (a) Find the uniformly most powerful test of level 0.05 of $H_0\colon \lambda \le 0.61$ against $H_1\colon \lambda > 0.61$. (b) Apply it to the Guard corps (16 deaths) and to corps XI (25 deaths). (c) Find its power at rates of 0.8, 1.0 and 1.2.

Solution. (a) The Poisson family has a monotone likelihood ratio in $T = \sum x_i$, the total number of deaths in the 20 years, so by the theorem the UMP test rejects for large $T$. Sums of independent Poisson counts are Poisson (The Poisson Distribution, "Sums of independent Poisson counts"), so $T$ is Poisson with mean $20\lambda$, which is 12.2 at the boundary $\lambda = 0.61$. The upper tail probabilities there are

$$P(T \ge 18) = 0.0710, \qquad P(T \ge 19) = 0.0428, \qquad P(T = 18) = 0.0282.$$

So the test rejects when $T \ge 19$ and, when $T = 18$, with probability $\gamma = (0.05 - 0.0428)/0.0282 = 0.2561$ (from unrounded values). By the theorem its size over all $\lambda \le 0.61$ is 0.05, attained at 0.61.

(b) The Guard corps had $T = 16 < 18$: not rejected. The tail probability $P(T \ge 16) = 0.1704$ shows how ordinary 16 deaths would be at the standard rate. Corps XI had $T = 25 \ge 19$: rejected, with $P(T \ge 25) = 0.0009$. Treating 0.61 as known ignores the sampling error in a rate estimated from 200 corps-years, so these are illustrations of the test and not a full analysis of the corps.

(c) At a rate $\lambda$ the power is $P(T \ge 19) + 0.2561\, P(T = 18)$ for $T$ Poisson with mean $20\lambda$:

$\lambda$ 0.8 1.0 1.2
Power 0.2789 0.6402 0.8823

The Guard corps' own rate was $16/20 = 0.8$. Twenty years of one corps would detect a rate of 0.8 against 0.61 only 28% of the time, so failing to reject says little. The power rises with $\lambda$, as statement 1 of the theorem requires.

Problem 3 (the price of a two-sided test). For the older adults (null mean $\mu_0 = 126.36$, the mean of all adults; their own standard deviation $\sigma = 21.02$; true mean 136.85; $\alpha = 0.05$), the upper-tailed $z$ test with 25 observations has power 0.8022. (a) What does each of the three $z$ tests achieve if the true mean is instead 10.49 below $\mu_0$? (b) How many observations does the two-sided test need to reach a power of 0.8022 against 136.85?

Solution. (a) At a mean of 115.87, $\lambda = -2.4944$. The upper-tailed test has power $\Phi(-2.4944 - 1.6449) = \Phi(-4.139) = 0.00002$. The lower-tailed test has $\Phi(2.4944 - 1.6449) = 0.8022$. The two-sided test has $\Phi(-2.4944 - 1.9600) + \Phi(2.4944 - 1.9600) = \Phi(-4.454) + \Phi(0.534) = 0.7035$, of which the first term is less than 0.00001. These are the first row of the table in the section on two-sided tests. The upper-tailed test, which is most powerful above $\mu_0$, is almost certain to miss a difference of the same size below it.

(b) The sample size formula of Power and Sample Size for a Test of a Mean for a two-sided test is $n = [(z_{1-\alpha/2} + z_{1-\beta})\sigma/(\mu_1 - \mu_0)]^2$. The power 0.8022 corresponds to $z_{1-\beta} = 0.8495$, since $\Phi(0.8495) = 0.8022$; indeed $0.8495 = 2.4944 - 1.6449$. So, with the standard deviation and the difference to one more digit than elsewhere on the page (21.024 and 10.4883),

$$n = \left[\frac{(1.9600 + 0.8495) \times 21.024}{10.4883}\right]^2 = 31.72,$$

which rounds up to 32. Exactly, the two-sided test has power 0.7932 with 31 observations and 0.8057 with 32. Protection against a difference in the unexpected direction costs seven more observations, 28% more. That is the trade the theorem on two-sided tests says cannot be avoided: no test has the upper-tailed test's power above $\mu_0$ and any useful power below it.

Problem 4 (a normal null hypothesis against a Cauchy alternative). One observation $X$ is either standard normal ($H_0$) or standard Cauchy, with density $1/[\pi(1 + x^2)]$ ($H_1$). Find the most powerful test of size 0.05 and its power. Compare it with the one-sided region $x \ge 1.645$.

Solution. The two hypotheses are simple, so the lemma applies although the distributions belong to different families. The ratio is

$$r(x) = \frac{1/[\pi(1 + x^2)]}{e^{-x^2/2}/\sqrt{2\pi}} = \sqrt{\frac{2}{\pi}}\; \frac{e^{x^2/2}}{1 + x^2}.$$

It depends on $x$ only through $u = x^2$. The derivative of $e^{u/2}/(1 + u)$ with respect to $u$ is $e^{u/2}(u - 1)/[2(1 + u)^2]$, negative for $u < 1$ and positive for $u > 1$. So $r$ falls from $r(0) = 0.7979$ to its minimum $r(\pm 1) = 0.6577$ and then rises without bound as $\lvert x \rvert$ grows. For a constant $k > 0.7979$ the region $r(x) > k$ is therefore $\lvert x \rvert > b$ for the number $b$ with $r(b) = k$. Its size under $H_0$ is $2[1 - \Phi(b)]$, which is 0.05 for $b = 1.960$, and $k = r(1.960) = 1.1249 > 0.7979$ as required. No randomization is needed, because $X$ is continuous.

The most powerful test of size 0.05 rejects when $\lvert x \rvert \ge 1.960$. It is the two-sided region, although neither hypothesis says anything about a direction: the Cauchy alternative differs from the normal by its heavy tails on both sides. The Cauchy distribution function is $\tfrac12 + \arctan(x)/\pi$, so the power is

$$P_1(\lvert X \rvert \ge 1.960) = 1 - \frac{2}{\pi}\arctan(1.960) = 0.3003.$$

The one-sided region $x \ge 1.645$ also has size 0.05, and its power is $\tfrac12 - \arctan(1.645)/\pi = 0.1739$. The lemma finds the shape of the best region from the two distributions, and the shape need not be a single tail. For smaller constants the region changes shape again: with $k = 0.75$, between the minimum 0.6577 and $r(0) = 0.7979$, the set $r(x) > k$ is $\lvert x \rvert < 0.3772$ together with $\lvert x \rvert > 1.4855$, a test of size 0.4314 and power 0.6068 that also rejects for observations very near zero, where the ratio has a local maximum.

Summary

Result Statement
Likelihood ratio $r(\mathbf{x}) = f_1(\mathbf{x})/f_0(\mathbf{x})$: power bought per unit of size at $\mathbf{x}$
Neyman–Pearson test reject if $f_1 > k f_0$; reject with probability $\gamma$ if $f_1 = k f_0$; $k$ and $\gamma$ set the size to $\alpha$
Sufficiency any test of that form is most powerful at its own size
Necessity a most powerful test has that form wherever $f_1 \ne k f_0$ (with probability 1)
Power and size power $\ge \alpha$, with equality only if $f_0 = f_1$
Normal mean, $\sigma$ known reject when $z \ge z_{1-\alpha}$; power $\Phi(\lambda - z_{1-\alpha})$
Monotone likelihood ratio in $T$ reject for large $T$: UMP for $H_0\colon \theta \le \theta_0$ against $H_1\colon \theta > \theta_0$, power nondecreasing in $\theta$
Two-sided hypothesis, normal mean or binomial proportion no UMP test; the two-sided $z$ test is unbiased

What we have learned

For two fully specified hypotheses, the most powerful test of a given size rejects when the likelihood ratio is large, and for discrete data it may have to settle one borderline outcome by chance to use its size exactly. For the older adults that test is the one-sided $z$ test, with power 0.8022 where the two-sided test has 0.7035. For 60 young men it rejects at 20 or more hypertensive men, and with probability 0.5535 at 19. When larger values of one statistic always favor larger values of the parameter, the same test is best against every alternative on one side. Against alternatives on both sides no test is best, and the proof shows why: a test that matches the upper-tailed test above the null value has its power function below it too.

Most hypotheses in practice are composite, with parameters that are not specified at all. Likelihood Ratio, Wald and Score Tests takes the ratio of this article, replaces the unknown parameter by its maximum likelihood estimate, and finds the large-sample distribution that makes the result a usable test.

In Python

from scipy import stats

n, p0, p1 = 60, 0.224, 261 / 822
null, alt = stats.binom(n, p0), stats.binom(n, p1)
c = 19                                              # reject above c, randomize at c
gamma = (0.05 - null.sf(c)) / null.pmf(c)           # about 0.5535
print(null.sf(c), null.sf(c - 1))                   # about 0.0344 and 0.0626
print(alt.sf(c) + gamma * alt.pmf(c))               # power: about 0.5049

sf(c) is $P(X > c)$. Every number in this article is computed by tools/experiments/statistics/neyman-pearson-lemma.py in the site's repository, from the NHANES files loaded by statdata.py and proportions.py and the horse-kick table in distdata.py; the figures are drawn by tools/figures/neyman-pearson-lemma.py.

References

  • Karlin, S., and Rubin, H. (1956). The theory of decision procedures for distributions with monotone likelihood ratio. The Annals of Mathematical Statistics, 27(2), 272–299. https://doi.org/10.1214/aoms/1177728259 record
  • Lehmann, E. L. (1992). Introduction to Neyman and Pearson (1933), On the problem of the most efficient tests of statistical hypotheses. In S. Kotz and N. L. Johnson (eds.), Breakthroughs in Statistics, Volume I: Foundations and Basic Theory, 67–72. Springer. https://doi.org/10.1007/978-1-4612-0919-5_5 record
  • Lehmann, E. L., and Romano, J. P. (2005). Testing Statistical Hypotheses (3rd ed.). Springer. Chapter 3, "Uniformly Most Powerful Tests" (Section 3.4, "Distributions with Monotone Likelihood Ratio"), and Chapter 4, "Unbiasedness: Theory and First Applications" (Section 4.2, "One-Parameter Exponential Families"). record · 2 pages
  • National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018: DEMO_J, BPX_J and BPQ_J data files and documentation. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 67 pages
  • Neyman, J., and Pearson, E. S. (1933). On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London, Series A, 231, 289–337. https://doi.org/10.1098/rsta.1933.0009 record · 3 pages