Hypothesis Testing

Deciding whether data are consistent with a stated value of a population mean. The logic of a test and the correct reading of its p-value, following the American Statistical Association's 2016 statement; one- and two-sided z and t tests and their equivalence to confidence intervals; Type I and Type II errors measured by simulation on NHANES blood pressures; and power curves and the sample size a test needs.

Ten patients in Student's 1908 sleep trial slept on average 1.58 hours longer with one drug than with another. A confidence interval says how large that difference plausibly is. Often a narrower question comes first: is a difference that large the kind of thing chance alone would produce if the drugs were equally effective?

A hypothesis test answers that question. It supposes the drugs are equally effective, works out what that supposition predicts, and measures how far the data fall from the prediction. The measure is the $p$-value. It is perhaps the most widely reported number in medical research, and among the most widely misread.

This unit follows a test from its logic to its design, in three steps.

  1. How does a test reach its verdict, and what does the $p$-value mean? The test for a population mean is built from the same sampling distributions that gave the confidence intervals of the previous unit. Tests and intervals turn out to be two readings of one calculation.
  2. How often is the verdict wrong? A test can reject a hypothesis that is true or fail to reject one that is false. The rates of these two errors, not the $p$-value of any single study, are what a test's design controls.
  3. How large must a study be? Power is the probability of detecting a difference that is really there. Planning for it gives a study a good chance of finding what it is looking for.

The examples use two datasets: Student's sleep data, and the systolic blood pressures of adults from the National Health and Nutrition Examination Survey (NHANES), treated as a population whose true mean is known so that each decision can be checked against the truth. The reading of $p$-values follows the American Statistical Association's 2016 statement and its six principles. Every number comes from a script in the site's repository.

Articles

  • Hypothesis Tests for a Mean: are the data consistent with a stated mean? The logic of a test, the $z$ test with known $\sigma$ and the $t$ test with estimated $\sigma$, worked on NHANES samples and Student's sleep data. Also Fisher's 1925 treatment of the same data, and a real difference that a study of 25 failed to detect. Then one- and two-sided alternatives, the proof that a test and an interval always agree, and the $p$-value function. Last, the ASA's six principles and the common misreadings of a $p$-value.
  • Type I and Type II Errors: how often does a test decide wrongly? Neyman and Pearson's view of a test as a rule with controlled error rates. Type I error rates measured for normal and skewed populations, and why $p$-values are uniform when the null hypothesis is true. Type II error rates and their trade-off with $\alpha$. How optional stopping, choosing the side and many tests inflate the Type I rate, and what share of significant results are false.
  • Power and Sample Size for a Test of a Mean: how many observations does a test need? The power function of the $z$ test and power curves, the noncentral $t$ distribution for the $t$ test, and the sample size formula derived and corrected for $t$. Also planning a study of older adults and a new sleep trial, and what moves the required sample size. Last, why power computed after the study adds nothing to the $p$-value.

The tests use the pivots of Confidence Intervals. Confidence Intervals for a Mean derives the $t$ distribution. One-Sided Confidence Bounds and Sample Size shows why the side of a one-sided analysis must be fixed in advance, and how to size a study for a target interval width.

The next unit applies the same tests to two groups: the paired t-test, the two-sample t-test and sample size for comparing two means.

In information retrieval, Significance Testing works the paired $t$-test, the sign test, the Wilcoxon test, the randomization test and the bootstrap on comparisons of search engines. It also covers the number of test queries needed and the problem of many comparisons.

A test's errors are close cousins of a diagnostic test's false positives and false negatives. That analogy is developed in Sensitivity and Specificity and Predictive Values and Bayes' Theorem.

References

  • Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., and Altman, D. G. (2016). Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. European Journal of Epidemiology, 31(4), 337–350. record · 5 pages
  • Neyman, J., and Pearson, E. S. (1933). On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London, Series A, 231, 289–337. record · 2 pages
  • NIST/SEMATECH. e-Handbook of Statistical Methods, §7.1.3 (What are statistical tests?) and §7.2.2 (Are the data consistent with the assumed process mean?). National Institute of Standards and Technology. https://www.itl.nist.gov/div898/handbook/ record · 30 pages
  • Pagano, M., Gauvreau, K., and Mattie, H. (2022). Principles of Biostatistics (3rd ed.). CRC Press. The course text. record · 35 pages
  • Wasserstein, R. L., and Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129–133. https://doi.org/10.1080/00031305.2016.1154108 record · 6 pages