Hypothesis Tests for a Mean ran two tests on samples of 25 from the NHANES blood pressure population, where the truth is known. The first tested whether all adults have mean systolic pressure 126.36 mm Hg, which they do. It found $p = 0.53$: no rejection, the right decision. The second tested whether adults aged 60 and over have the same mean, which they do not; theirs is 10.49 mm Hg higher. It found $p = 0.081$: no rejection, the wrong decision. Nothing in either calculation was faulty. A test decides on the basis of a random sample, and some samples mislead. How often do they mislead, and in which direction?
A test can go wrong in two ways. A Type I error rejects a null hypothesis that is true; a Type II error fails to reject one that is false. The rates of the two errors, not the $p$-value of any one study, are what a test's design controls. This article is the course's home for those two rates; the power calculations of the next article build on them.
Here is the route. First we define the two errors and trace the idea to Neyman and Pearson. Then we measure the Type I rate by simulation on normal and skewed populations, and see why $p$-values are uniform when the null hypothesis is true. Next we measure the Type II rate and its trade-off with $\alpha$. Finally we look at practices that quietly raise the Type I rate, and at what an error rate does not say about a single result. The article assumes the tests and $p$-values of Hypothesis Tests for a Mean.
Two ways to be wrong
Every test ends in one of two decisions, and the null hypothesis is either true or false. So there are four possible outcomes.
| Fail to reject $H_0$ | Reject $H_0$ | |
|---|---|---|
| $H_0$ true | Correct decision, probability $1 - \alpha$ | Type I error, probability $\alpha$ |
| $H_0$ false | Type II error, probability $\beta$ | Correct decision, probability $1 - \beta$ |
A Type I error is rejecting a null hypothesis that is true. Examples are concluding that the older adults' mean differs when it does not, or that a drug works when it does not. Its probability is the significance level,
$$\alpha = P(\text{reject } H_0 \mid H_0 \text{ true}).$$
A Type II error is failing to reject a null hypothesis that is false: missing a real difference, as the second NHANES test did. Its probability is written $\beta$,
$$\beta = P(\text{fail to reject } H_0 \mid H_0 \text{ false}).$$
Its complement $1 - \beta$, the probability of rejecting a false null hypothesis, is the test's power.
The two rates are not symmetric. The analyst chooses $\alpha$ by setting the critical value. The NIST/SEMATECH e-Handbook of Statistical Methods (§7.1.3) notes that $\beta$, by contrast, "is not chosen by the user but is determined ... by the magnitude of the real discrepancy." Large departures from $H_0$ are easy to detect and small ones hard, and $\beta$ also grows as $\alpha$ is made smaller. "$H_0$ false" covers every value of $\mu$ other than $\mu_0$. So $\beta$ is not one number but a function of the true mean, and a statement such as "$\beta = 0.2$" is always about one particular alternative value.
The course used screening language earlier for diagnostic tests, and it fits here. A Type I error is a false positive and a Type II error a false negative. $1 - \alpha$ plays the part of specificity, and the power the part of sensitivity (Sensitivity and Specificity). The analogy is close, and it is used again at the end of this article.
Where did the idea of two error rates come from, and why should a test be judged by them at all?
Tests as rules of behavior
The two kinds of error, and the idea of a test as a procedure judged by its error rates, are due to Jerzy Neyman and Egon Pearson. Their 1933 paper begins from a skeptical position. As far as a particular hypothesis is concerned, they wrote, no test based on probability "can by itself provide any valuable evidence of the truth or falsehood of that hypothesis." What a test can do is serve as a rule of behavior, "in following which we insure that, in the long run of experience, we shall not be too often wrong" (Neyman and Pearson, 1933, p. 291). Their example of such a rule is the one used throughout this topic. Compute a statistic from the data, reject the hypothesis if it exceeds a fixed value, and accept it otherwise.
They then named the two risks. "If we reject $H_0$, we may reject it when it is true; if we accept $H_0$, we may be accepting it when it is false" (p. 296). These two sources of error can rarely both be eliminated, and which matters more depends on the case. Their illustration is Laplace's old question of how many judges' votes should be needed to convict. Is it worse to convict an innocent person or to acquit a guilty one? That depends on the consequences, they wrote, and "the use of these statistical tools in any given case, in determining just how the balance should be struck, must be left to the investigator." Their paper speaks of errors of the first kind and of the second type; "Type I" and "Type II" are the modern names.
Fisher's significance tests and Neyman and Pearson's decision rules are different ideas that are now taught together. Hypothesis Tests for a Mean traced Fisher's tests to his 1925 handbook.
- The $p$-value is Fisher's graded measure of how unusual the data are under $H_0$.
- The fixed $\alpha$, the alternative hypothesis and $\beta$ are Neyman and Pearson's.
Rejecting when $p \le \alpha$ joins them, and the rest of this article uses that joined form.
The figure shows the two error rates for the $z$ version of the older-adult test, with $n = 10$ and $\sigma = 21.02$ mm Hg treated as known. If $H_0$ is true, the statistic $z = (\bar{x} - 126.36)/(\sigma/\sqrt{n})$ follows the standard normal curve on the left, and the orange tails beyond $\pm 1.96$ are $\alpha$. If the true mean is 136.85, $z$ follows the same curve shifted right by $10.49/(21.02/\sqrt{10}) = 1.578$. The hatched part of that curve between the critical values is $\beta$, here 0.65.
The picture shows the rates in theory, for a normal statistic. Do real tests on real data keep them?
Measuring the Type I error rate
When the assumptions of a test hold, its Type I error rate equals $\alpha$ by construction: the critical value was chosen so that the rejection region has probability $\alpha$ under $H_0$. Whether the rate holds in a given situation is a question for simulation.
The script type-i-and-type-ii-errors.py draws samples with $H_0$ true and runs the $t$ test at $\alpha = 0.05$ on each. It records the share rejected: 10,000 samples for each of 30 seeds, reporting the median of the 30 shares. It does this for two populations, testing each population's own mean:
- a normal population;
- the NHANES blood pressure population, whose distribution is skewed to the right (skewness 1.04).
Check your understanding. Before reading the table: for the right-skewed blood pressure population and a small sample, which one-sided test would reject a true null too often, the upper-tailed or the lower-tailed?
The lower-tailed one, $H_A: \mu < \mu_0$. In a right-skewed population, samples that miss the long upper tail have a low mean and a small standard deviation together. That combination produces large negative $t$ statistics far more often than the $t$ distribution predicts. The table measures how much.
| Population | $n$ | Two-sided | Upper-tailed ($H_A: \mu > \mu_0$) | Lower-tailed ($H_A: \mu < \mu_0$) |
|---|---|---|---|---|
| Normal | 5 | 0.0507 | 0.0499 | 0.0495 |
| Normal | 10 | 0.0502 | 0.0503 | 0.0497 |
| Normal | 100 | 0.0496 | 0.0503 | 0.0497 |
| Blood pressure | 5 | 0.0660 | 0.0224 | 0.0963 |
| Blood pressure | 10 | 0.0640 | 0.0258 | 0.0890 |
| Blood pressure | 25 | 0.0587 | 0.0324 | 0.0760 |
| Blood pressure | 100 | 0.0529 | 0.0399 | 0.0631 |
For the normal population, every version of the test holds its 5% to within the simulation's noise, about two tenths of a percentage point. For blood pressure, the two-sided test rejects a true null somewhat too often: 6.6% at $n = 5$, approaching 5% as $n$ grows.
The one-sided tests are lopsided. A test of $H_A: \mu < \mu_0$ rejects a true null nearly twice as often as promised at $n = 5$ and 10. The opposite test rejects it only about half as often. The cause is the one found for intervals in Confidence Intervals for a Mean. A one-sided test in the direction opposite to the skew is the one at risk. With skewed data and a small sample, the two-sided test is the safer choice. The nonparametric tests of a later unit, which ask about the median rather than the mean, are an alternative.
The p-value is uniform when the null hypothesis is true
There is a neat way to see what controlling the Type I error rate means. When $H_0$ is true and the test statistic has a continuous distribution, the $p$-value is equally likely to fall anywhere between 0 and 1.
The proof is two lines. For a two-sided $t$ test, $p \le a$ exactly when $|t| \ge t_{n-1,\,1-a/2}$, because the $p$-value falls as $|t|$ rises and equals $a$ at that critical value. Under $H_0$ that event has probability $a$. So
$$P(p \le a \mid H_0) = a \quad \text{for every } a \text{ between 0 and 1},$$
which is the definition of a uniform distribution on $[0, 1]$. Rejecting when $p \le 0.05$ therefore rejects a true null 5% of the time, and rejecting when $p \le 0.01$ does so 1% of the time.
Check your understanding. With a true null hypothesis, how often does the $p$-value land between 0.60 and 0.65? And is $p = 0.6$ more surprising under $H_0$ than $p = 0.3$?
The interval has width 0.05, so about 5% of the time, the same as the interval from 0 to 0.05. And neither value is more surprising than the other: under $H_0$ every value between 0 and 1 is equally likely. The left panel of the figure shows it.
When $H_0$ is false the $p$-values pile up near zero, and the share below $\alpha$ is the power. For the older adults with $n = 10$, 24.9% of samples gave $p < 0.05$ and 15.6% gave $p > 0.5$. A real difference of more than 10 mm Hg produced a $p$-value above one half in one study out of six. This is the ASA's sixth principle (Wasserstein and Lazar, 2016) in a single picture. A large $p$-value is common when the null hypothesis is false and the study is small, so it is not evidence that the null hypothesis is true.
That middle panel already measures the other error. How large is it, and what controls it?
Measuring the Type II error rate
The Type II error rate depends on four things:
- how far the truth is from $H_0$;
- the sample size;
- the variability of the data;
- $\alpha$.
Back to the older adults. Their true difference is 10.49 mm Hg and their standard deviation 21.02 mm Hg, so the difference is 0.499 standard deviations. The script measured $\beta$ for the two-sided $t$ test at $\alpha = 0.05$ by sampling from the 1,923 adults aged 60 and over. It compared the result with the value that theory gives for a normal population with the same mean and standard deviation. That theory uses the noncentral $t$ distribution, derived in Power and Sample Size for a Test of a Mean.
| $n$ | $\beta$, simulated from NHANES (IQR) | $\beta$, normal population theory | Power, simulated |
|---|---|---|---|
| 5 | 0.906 (0.904 to 0.908) | 0.860 | 0.094 |
| 10 | 0.752 (0.750 to 0.753) | 0.708 | 0.249 |
| 25 | 0.310 (0.307 to 0.311) | 0.332 | 0.690 |
| 50 | 0.044 (0.044 to 0.046) | 0.067 | 0.956 |
With 10 older adults, a study misses this real and substantial difference three times in four. With 25 it misses it about three times in ten. That puts the sample in Hypothesis Tests for a Mean that gave $p = 0.081$ among the ordinary outcomes of a study that size, not the unlucky ones. With 50 the miss rate falls below 5%.
The real population departs from normal theory in both directions: the test is weaker than the theory predicts at small $n$ and stronger at large $n$. The older adults' pressures are skewed to the right (skewness 0.75). With a skewed population the sample mean and standard deviation rise and fall together. So the samples whose means are high enough to reject tend also to have large standard errors, and that costs power when samples are small. For skewed data, the normal-theory figure is a guide, not a guarantee. Simulation from a realistic population is the way to check a particular design.
The trade-off with alpha
For a fixed sample, making $\alpha$ smaller moves the critical values outward. That shrinks the region where a false null is rejected as well as the region where a true one is.
Check your understanding. With 25 older adults, $\beta$ is 0.31 at $\alpha = 0.05$. Suppose we demand much stronger evidence, $\alpha = 0.001$. Roughly what happens to $\beta$?
It rises to about 0.90. The table shows the whole range.
| $\alpha$ | Critical value $t_{24,\,1-\alpha/2}$ | $\beta$, simulated | $\beta$, normal theory |
|---|---|---|---|
| 0.10 | 1.711 | 0.190 | 0.218 |
| 0.05 | 2.064 | 0.311 | 0.332 |
| 0.01 | 2.797 | 0.624 | 0.600 |
| 0.001 | 3.745 | 0.903 | 0.857 |
Cutting the Type I error rate from 5% to 0.1% raises the Type II error rate from 31% to 90%. At this sample size, a test that almost never raises a false alarm almost never raises a true one either. The NIST handbook (§7.1.3) states the same trade: the risk $\beta$ increases as the risk $\alpha$ decreases. For a fixed design, the only way to lower both is more information, usually a larger sample. Power and Sample Size for a Test of a Mean plans for exactly that.
Which error to guard against more is a question about consequences, as Neyman and Pearson said.
- If $H_0$ is "the new treatment is no better than the old," a Type I error would put an ineffective treatment into use, and a small $\alpha$ is appropriate.
- If $H_0$ is "this exposure is harmless," a Type II error would leave a hazard in place. The cost of missing harm may justify a larger $\alpha$ or, better, a larger study.
The 0.05 convention is a default, not a judgment about the costs in any particular study.
Practices that raise the Type I error rate
The guarantee $P(\text{reject} \mid H_0) = \alpha$ holds for a test specified in advance and run once. The ASA's fourth principle, "proper inference requires full reporting and transparency" (Wasserstein and Lazar, 2016), concerns the ways that guarantee is lost without any error in arithmetic. Here are three.
Stopping when the result looks good
A common temptation is to test as data arrive and stop as soon as $p < 0.05$. The script simulated it. It drew samples from a normal population with $H_0$ true and ran a two-sided $t$ test after every 10 observations. It stopped at the first $p < 0.05$ or at a maximum sample size.
Check your understanding. With two looks, at 10 and 20 observations, is the chance of a false rejection 10% (two chances of 5%), $1 - 0.95^2 = 9.75\%$ (two independent chances), or less than both?
Less than both: 8.6%. The second test reuses the first 10 observations, so the two tests are positively correlated. A sample that was unremarkable at 10 tends to stay unremarkable at 20, and the second look adds less than an independent test would. Each further look still adds something.
| Largest $n$ allowed | 10 | 20 | 30 | 50 | 100 | 100, testing after every observation from 10 on |
|---|---|---|---|---|---|---|
| Chance of a false rejection | 0.050 | 0.086 | 0.111 | 0.146 | 0.198 | 0.286 |
Each look is another chance for the $p$-value, which wanders when $H_0$ is true, to dip below 0.05. The chances accumulate. With ten looks the real error rate is four times the stated one, and with a test after every observation it passes 28%.
Some designs must monitor accumulating data, such as clinical trials with planned interim analyses. They plan the looks in advance and use stopping boundaries that demand more evidence than a single test would. The boundaries are chosen so that the overall Type I error rate is controlled (ICH E9, 1998, §3.4 and §4.5). The problem is not looking but looking without accounting for it.
Choosing the side after looking
One-Sided Confidence Bounds and Sample Size showed by simulation that choosing a one-sided bound in whichever direction the sample mean fell doubles the miss rate from 5% to 10%. The same is true of a one-sided test chosen after seeing the data, since the test and the bound are the same calculation. Such a "5%" test is a two-sided test at 10%.
Many tests
Suppose $m$ independent tests are run at $\alpha = 0.05$ and every null hypothesis is true. The chance that at least one rejects is $1 - 0.95^m$, which is 0.40 for ten tests. The information retrieval wiki's Significance Testing works this arithmetic and the Bonferroni correction on comparisons of search engines. The unit on analysis of variance treats it for comparisons among group means in Multiple Comparisons and the Bonferroni Correction.
The ASA statement adds that formal multiple testing is not needed for the problem to arise. Whenever a researcher chooses what to report based on the results, the reported $p$-values lose their meaning, unless the reader is told how the choice was made.
What an error rate does not say
$\alpha$ and $\beta$ are long-run frequencies over many uses of a test. They do not transfer to a single result. Greenland and colleagues (2016) list two misreadings that mirror each other.
- "If you reject the test hypothesis because $P \le 0.05$, the chance you are in error is 5%." (Misinterpretation 10.) If $H_0$ is in fact true, a rejection is an error with certainty; if it is false, a rejection is not an error at all. The 5% is the frequency of rejection over many studies in which $H_0$ and every other assumption are true.
- "If you accept the null hypothesis because the null P value exceeds 0.05 and the power of your test is 90%, the chance you are in error is 10%." (Misinterpretation 24.) The same reasoning applies. The 10% refers to many studies in which the particular alternative used to compute power is true.
A reader usually wants something else: "given that this result was significant, how likely is it that the effect is real?" That question has an answer only with one more ingredient, the share of tested hypotheses in which there is a real effect to find. The arithmetic is Bayes' theorem. It is the same arithmetic that turns a positive screening test into a probability of disease in Predictive Values and Bayes' Theorem:
- $\alpha$ plays the role of the false positive rate;
- power plays the role of sensitivity;
- the share of real effects plays the role of prevalence.
Suppose a field tests many hypotheses at $\alpha = 0.05$. A share $\pi$ of them are true effects, which the studies detect with power $1 - \beta$. Among all tests, a share $\pi(1 - \beta)$ are true positives and $(1 - \pi)\alpha$ false positives. So the share of significant results that are false is
$$\frac{(1 - \pi)\,\alpha}{(1 - \pi)\,\alpha + \pi\,(1 - \beta)}.$$
Check your understanding. In a field where one tested hypothesis in ten is a real effect, and studies have 80% power, what share of significant results are false?
Of every 1,000 tests, 100 are real effects and 80 of them reach significance. Of the 900 true nulls, 5%, or 45, also reach significance. So 45 of the 125 significant results are false: 0.360, more than a third. The table gives this case and three others.
| Share of tested hypotheses with a real effect $\pi$ | Power | Share of significant results that are false |
|---|---|---|
| 0.5 | 0.80 | 0.059 |
| 0.5 | 0.25 | 0.167 |
| 0.1 | 0.80 | 0.360 |
| 0.1 | 0.25 | 0.643 |
When real effects are common and studies well powered, about 6% of significant findings are false, close to what a naive reading of $\alpha$ suggests. When real effects are rare and studies small, nearly two thirds are. The $\alpha$ of each test is 0.05 in every row. This is one reason the ASA statement insists that a $p$-value be read with the design of the study and the external evidence for the effect. It is also one reason that low power matters even to someone who cares only about false positives.
What we have learned
A test can be wrong in two ways. Its Type I error rate is fixed by the critical value, and it held at 5% for normal data; for skewed blood pressures it held roughly for two-sided tests but tilted badly for one-sided ones. Under a true null the $p$-value is uniform. The Type II rate is not chosen: for the older adults it was 75% with 10, 31% with 25, 4% with 50. Lowering $\alpha$ raises $\beta$. Optional stopping, choosing the side and selective reporting break the Type I guarantee without any arithmetic error. And an error rate describes a procedure, not the chance that one result is wrong.
The table of $\beta$ against $n$ points to the remedy for the older-adult study: a larger sample. But how large, for a given difference and a given chance of finding it? The next article, Power and Sample Size for a Test of a Mean, derives the power of the test and turns it into a sample size.
In Python
The error rates of a test can be checked for any population by simulation in a few lines.
import numpy as np
from scipy import stats
rng = np.random.default_rng(0)
x = rng.standard_normal((10_000, 10)) # H0 true: mean 0
p = stats.ttest_1samp(x, 0, axis=1).pvalue
print((p < 0.05).mean()) # about 0.05: Type I error rate
y = rng.normal(0.5, 1.0, (10_000, 10)) # H0 false: mean 0.5 sigma
p = stats.ttest_1samp(y, 0, axis=1).pvalue
print((p >= 0.05).mean()) # about 0.71: Type II error rate
Every number in this article is computed by tools/experiments/applied-statistics/type-i-and-type-ii-errors.py in the site's repository, which samples from the NHANES 2017 to 2018 files and implements the $t$ and noncentral $t$ distributions directly; they were checked against scipy 1.17.1.
References
- Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., and Altman, D. G. (2016). Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. European Journal of Epidemiology, 31(4), 337–350. record · 5 pages
- ICH (1998). E9: Statistical Principles for Clinical Trials. International Council for Harmonisation. https://database.ich.org/sites/default/files/E9_Guideline.pdf record
- National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018: Demographic Variables and Sample Weights (DEMO_J) and Blood Pressure (BPX_J) data files. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 45 pages
- Neyman, J., and Pearson, E. S. (1933). On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London, Series A, 231, 289–337. record · 2 pages
- NIST/SEMATECH. e-Handbook of Statistical Methods, §7.1.3 (What are statistical tests?). National Institute of Standards and Technology. https://www.itl.nist.gov/div898/handbook/ record · 30 pages
- Wasserstein, R. L., and Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129–133. https://doi.org/10.1080/00031305.2016.1154108 record · 6 pages