Hypothesis Tests for a Mean

How a test of a population mean works and what its p-value says. The logic of a test, the z test with known standard deviation and the t test with estimated standard deviation, worked on NHANES blood pressures and Student's sleep data, one- and two-sided alternatives, why a test and a confidence interval always agree, the American Statistical Association's six principles for reading a p-value, and the common misreadings they warn against.

Ten patients each took two sleeping drugs, and on average they slept 1.58 hours longer with the second than with the first. Confidence Intervals for a Mean turned that number into a 95% confidence interval, 0.70 to 2.46 hours, which says how large the difference plausibly is. But ten patients is a small sample, and some of the 1.58 hours is surely chance. So a different question is often asked first: are the data consistent with the two drugs being equally effective?

A hypothesis test answers that question. It asks whether a difference this large is the kind of thing chance alone would be expected to produce. This article is the course's home for the logic of a test and the $p$-value, and later articles build on it.

Here is the route. First we name the parts of a test. Then we work the $z$ test on NHANES blood pressures, where the truth is known, and the $t$ test on Student's sleep data. Next we treat one- and two-sided alternatives, and prove that a test and an interval are one calculation. Finally we read the $p$-value as the American Statistical Association's 2016 statement asks, and name the misreadings it warns against. The article assumes the $t$ distribution of Confidence Intervals for a Mean.

The logic of a test

A test reasons the way a skeptic does. It runs in three moves.

  1. Suppose the uninteresting explanation is true: the drugs are equally effective, and any difference in sleep is chance.
  2. Work out what data that supposition predicts.
  3. Compare the data actually observed with the prediction.

If the observed data would be very unusual under the supposition, that is evidence against it. If they are ordinary, the supposition survives, though it has not been shown true.

The parts of this argument have names.

  • The null hypothesis $H_0$ is the statement being tested, usually that a parameter equals a particular value: $H_0: \mu = \mu_0$. For the sleep data, $\mu$ is the mean extra sleep from drug 2 over drug 1 in a population of such patients, and $\mu_0 = 0$.
  • The alternative hypothesis $H_A$ says which departures from $H_0$ the test is meant to detect: $H_A: \mu \ne \mu_0$ for departures in either direction, or $H_A: \mu > \mu_0$ or $H_A: \mu < \mu_0$ for one direction only.
  • The test statistic measures how far the data fall from what $H_0$ predicts, in units that make its distribution known when $H_0$ is true. For a mean, it is the distance from the sample mean to $\mu_0$ in standard errors.
  • The $p$-value is the probability, computed supposing $H_0$ and every other assumption of the model are true, of a test statistic at least as far from what $H_0$ predicts as the one observed.
  • The significance level $\alpha$ is the threshold, chosen before the data are seen. $H_0$ is rejected when $p \le \alpha$.
  • Equivalently, $H_0$ is rejected when the test statistic falls in the rejection region, the set of values that has probability $\alpha$ under $H_0$. Its boundaries are the critical values.

The American Statistical Association's statement on $p$-values gives the informal definition this way: "a p-value is the probability under a specified statistical model that a statistical summary of the data (e.g., the sample mean difference between two compared groups) would be equal to or more extreme than its observed value" (Wasserstein and Lazar, 2016). The NIST/SEMATECH e-Handbook of Statistical Methods (§7.1.3.1) gives the same definition in terms of the test statistic. It recommends deciding in advance how small a $p$-value will lead to rejection, which is the same thing as choosing $\alpha$.

The vocabulary of the decision is deliberately lopsided. A test either rejects $H_0$ or fails to reject it. It never accepts or proves $H_0$. Data that are consistent with $\mu = 0$ are also consistent with many values near 0, and the test was not designed to tell them apart.

Now the logic needs a statistic whose distribution under $H_0$ we can compute. The simplest case is when $\sigma$ is known.

The z test: when sigma is known

The test statistic comes from the pivot used for confidence intervals in Confidence Intervals and Coverage. Let $X_1, \dots, X_n$ be independent draws from a population with mean $\mu$ and known standard deviation $\sigma$. Then

$$Z = \frac{\bar{X} - \mu}{\sigma/\sqrt{n}}$$

has a standard normal distribution, exactly for a normal population and approximately for a large sample from any population (the sampling distribution of the mean). If $H_0: \mu = \mu_0$ is true, we can substitute $\mu_0$ for $\mu$. That gives a quantity computable from the data whose distribution is known:

$$z = \frac{\bar{x} - \mu_0}{\sigma/\sqrt{n}} \sim N(0, 1) \text{ under } H_0.$$

This is the one-sample $z$ test. For $H_A: \mu \ne \mu_0$, values of $z$ far from zero in either direction count against $H_0$. So the two-sided $p$-value adds both tails:

$$p = P(|Z| \ge |z|) = 2\left[1 - \Phi(|z|)\right],$$

where $\Phi$ is the standard normal distribution function. At level $\alpha$ the rejection region is $|z| \ge z_{1-\alpha/2}$, which is $|z| \ge 1.960$ at $\alpha = 0.05$.

A test where the answer is known

The NHANES blood pressure population of Confidence Intervals and Coverage lets us watch a test at work with the truth in view. The 5,247 adults examined in the 2017 to 2018 survey have a mean systolic blood pressure of $\mu = 126.36$ mm Hg and a standard deviation of $\sigma = 20.12$ mm Hg. That article's first random sample of 25 adults had mean $\bar{x} = 128.88$.

Check your understanding. The sample mean is 2.52 mm Hg above 126.36, and the standard error is $20.12/\sqrt{25} = 4.025$ mm Hg. Before computing anything further, is that distance unusual?

It is not. The distance is less than one standard error, and the rejection region begins at 1.96 standard errors. The full calculation, testing $H_0: \mu = 126.36$, which here we know to be true, confirms it:

$$z = \frac{128.88 - 126.36}{20.12/\sqrt{25}} = \frac{2.52}{4.025} = 0.626, \qquad p = 2[1 - \Phi(0.626)] = 0.531.$$

The sample mean is about six tenths of a standard error from the hypothesized value. When $H_0$ is true, a distance at least that large happens in 53% of samples. There is no evidence against $H_0$, and the test does not reject it. The 95% interval from the same sample, $[120.99, 136.77]$, contains 126.36, as the section on intervals below shows it must.

In practice, of course, nobody knows $\sigma$. What changes when it must be estimated?

The t test: when sigma is estimated

When $\sigma$ is unknown, it is replaced by the sample standard deviation $s$. Confidence Intervals for a Mean derived the consequence: for a normal population, $(\bar{X} - \mu)/(S/\sqrt{n})$ has a $t$ distribution with $n - 1$ degrees of freedom. Under $H_0: \mu = \mu_0$,

$$t = \frac{\bar{x} - \mu_0}{s/\sqrt{n}} \sim t_{n-1} \text{ under } H_0.$$

The two-sided $p$-value is $2\,P(T_{n-1} \ge |t|)$, with rejection region $|t| \ge t_{n-1,\,1-\alpha/2}$. This is the one-sample $t$ test. The NIST handbook (§7.2.2) gives it in this form, writing $\bar{Y}$ and $N$ for $\bar{x}$ and $n$. It is the test used in practice for a mean, since $\sigma$ is rarely known.

Student's sleep data

Back to the ten sleep patients. Their differences in hours of sleep, drug 2 minus drug 1, were worked in Confidence Intervals for a Mean: $\bar{d} = 1.580$, $s_d = 1.2300$, and the estimated standard error is $s_d/\sqrt{10} = 0.3890$ hours. The test of equal effectiveness follows in four steps.

Step 1, the hypotheses. $H_0: \mu = 0$, the drugs give the same sleep on average; $H_A: \mu \ne 0$, they do not. The alternative is two-sided because, before the trial, a difference in either direction would have mattered.

Step 2, the test statistic.

$$t = \frac{\bar{d} - 0}{s_d/\sqrt{n}} = \frac{1.580}{0.3890} = 4.062,$$

with $n - 1 = 9$ degrees of freedom. The observed mean difference is about four standard errors from zero.

Step 3, the $p$-value. Under $H_0$, $t$ follows the $t$ distribution with 9 degrees of freedom, and

$$p = 2\,P(T_9 \ge 4.062) = 0.0028.$$

Step 4, the decision. At $\alpha = 0.05$ the critical value is $t_{9,\,0.975} = 2.2622$, and $4.062$ exceeds it. Equivalently, $0.0028 < 0.05$. $H_0$ is rejected. It would also be rejected at $\alpha = 0.01$, whose critical value is $t_{9,\,0.995} = 3.2498$.

What the null hypothesis predicts for Student's sleep data: the t distribution with 9 degrees of freedom. The orange tails beyond plus or minus 2.262 hold 5% of the probability and form the rejection region at alpha = 0.05. Notice how far out in the right tail the observed t of 4.06 lies. The p-value is the area beyond plus and minus 4.06, too small to see at this scale (0.0028).
What the null hypothesis predicts for Student's sleep data: the t distribution with 9 degrees of freedom. The orange tails beyond plus or minus 2.262 hold 5% of the probability and form the rejection region at alpha = 0.05. Notice how far out in the right tail the observed t of 4.06 lies. The p-value is the area beyond plus and minus 4.06, too small to see at this scale (0.0028).

The data are hard to reconcile with equal effectiveness. Suppose the drugs were equally effective and the model's assumptions held. Then a mean difference at least four standard errors from zero would occur in about 3 samples in 1,000. The confidence interval adds what the test does not: how large the difference plausibly is, 0.70 to 2.46 hours.

Check your understanding. Suppose we had ignored the estimation of $\sigma$ and used the normal distribution with $s_d$ in its place. Would the $p$-value come out larger or smaller than 0.0028?

Smaller: $p = 2[1 - \Phi(4.062)] = 0.00005$, nearly sixty times smaller. The $t$ distribution's heavier tails are the price of estimating $\sigma$ from ten numbers. Ignoring them overstates the evidence, just as it understates the width of an interval.

Fisher's version

R. A. Fisher's Statistical Methods for Research Workers (1925) was a handbook written for working scientists. It used the same data as its example of the significance of the mean of a small sample (§24, Ex. 19), quoted from Student's paper. Fisher compared the statistic with his table of $t$. For 9 degrees of freedom only one value in a hundred exceeds 3.250 by chance, so "the difference between the results is clearly significant."

He then checked the conclusion a second way. Of the nine differences that are not zero, all are positive. If the drugs were equally effective, each sign would be like a coin toss, so all nine would agree by chance only twice in $2^9 = 512$ trials. This is the sign test, developed in The Sign Test and the Wilcoxon Signed-Rank Test. Fisher preferred the $t$ test because it uses "the actual values and not merely of their signs" (Fisher, 1925, §24).

The book is also the source usually cited for the convention of 0.05. Discussing the normal distribution, Fisher wrote that the deviation for which $P = .05$ "is 1.96 or nearly 2; it is convenient to take this point as a limit in judging whether a deviation is to be considered significant or not" (Fisher, 1925, §12). The reason he gave was convenience.

So far both tests reached the right verdict. What happens when the null hypothesis is false but the sample is small?

A sample where the effect is real

Adults aged 60 and over in the NHANES population, 1,923 of them, have a mean systolic pressure of 136.85 mm Hg. That is 10.49 mm Hg above the all-adult mean. Suppose a study draws 25 of them and asks whether older adults' mean differs from the all-adult value. It tests $H_0: \mu = 126.36$ against $H_A: \mu \ne 126.36$, and here we know $H_0$ is false. The script's sample (seed 1) gave $\bar{x} = 133.19$ and $s = 18.76$, so

$$t = \frac{133.19 - 126.36}{18.76/\sqrt{25}} = \frac{6.83}{3.751} = 1.820, \qquad p = 2\,P(T_{24} \ge 1.820) = 0.081.$$

The critical value is $t_{24,\,0.975} = 2.0639$, so the test does not reject $H_0$ at the 0.05 level. Yet the difference it was looking for is real and clinically meaningful.

The 95% interval, $[125.44, 140.93]$, shows why. It includes the null value, and it also includes differences of 14 mm Hg. Twenty-five people were too few to settle the question. A study that reported this result as "no difference between older adults and the general population" would be reporting the opposite of the truth. How often a study of this size misses a real difference of this size is a question about its power. Type I and Type II Errors answers it: about 31% of the time.

One-sided and two-sided tests

Sometimes only one direction of departure matters. A one-sided test puts all of $\alpha$ in one tail. The NIST handbook (§7.2.2) lists the three forms of the test for a mean:

Null hypothesis Alternative Reject $H_0$ at level $\alpha$ when $p$-value
$\mu = \mu_0$ $\mu \ne \mu_0$ $\lvert t \rvert \ge t_{n-1,\,1-\alpha/2}$ $2\,P(T_{n-1} \ge \lvert t \rvert)$
$\mu \le \mu_0$ $\mu > \mu_0$ $t \ge t_{n-1,\,1-\alpha}$ $P(T_{n-1} \ge t)$
$\mu \ge \mu_0$ $\mu < \mu_0$ $t \le -t_{n-1,\,1-\alpha}$ $P(T_{n-1} \le t)$

The $z$ test has the same table with $z$ and the normal distribution in place of $t$ and $T_{n-1}$.

The one-sided nulls are composite: $H_0: \mu \le \mu_0$ contains many values of $\mu$. The test statistic is still computed at the boundary value $\mu_0$, and that is enough. If the true mean is below $\mu_0$, the sample mean tends to fall even further from the rejection region than it would at $\mu_0$. So the chance of rejecting is at most $\alpha$. The boundary is the worst case, and controlling the error there controls it everywhere in $H_0$.

Check your understanding. For the sleep data, the one-sided alternative is $H_A: \mu > 0$, that drug 2 gives more sleep. Without a table, what is its $p$-value?

Half the two-sided value: $p = P(T_9 \ge 4.062) = 0.0014$. The $t$ distribution is symmetric, and the observed $t$ lies on the side the alternative names, so the one tail holds half of the two-tailed area. This is the number that One-Sided Confidence Bounds and Sample Size found as the confidence level 0.99858 at which the lower bound reaches zero. It agrees with the 0.9985 that Student (1908) reported as the probability that drug 2 is the better soporific. Student's reading of it is not the modern one, as the section on reading a $p$-value explains.

A one-sided test is more sensitive in its chosen direction and blind in the other. Two cautions follow.

  • The side must be chosen before the data are seen. For the older adults above, the two-sided $p$ was 0.081. A test of $H_A: \mu > 126.36$ would give $p = P(T_{24} \ge 1.820) = 0.041$, below 0.05. Choosing the one-sided test after seeing which way the sample mean fell would turn a nonsignificant result into a significant one. One-Sided Confidence Bounds and Sample Size showed by simulation that doing so doubles the error rate from 5% to 10%. A one-sided test is legitimate when the protocol specifies it in advance. The question must genuinely have one direction, such as whether a new treatment is at least as good as the standard.
  • Two-sided is not always right either. Greenland and colleagues (2016, misinterpretation 14) list "one should always use two-sided $p$-values" among common errors. When the question of interest is one-sided, a one-sided $p$-value is the appropriate one, provided the report says so and why.

Tests and intervals are the same calculation

The sleep test rejected $\mu = 0$, and the 95% interval, 0.70 to 2.46, excludes 0. The NHANES test did not reject 126.36, and its interval contains 126.36. That is no coincidence. Here is the idea in one sentence: both come from the same pivot, and distance is symmetric, so $\mu_0$ is close to $\bar{x}$ exactly when $\bar{x}$ is close to $\mu_0$.

The formal version. The two-sided $t$ test at level $\alpha$ fails to reject $H_0: \mu = \mu_0$ exactly when $|t| < t_{n-1,\,1-\alpha/2}$, that is, when

$$-t_{n-1,\,1-\alpha/2} < \frac{\bar{x} - \mu_0}{s/\sqrt{n}} < t_{n-1,\,1-\alpha/2}.$$

Multiply through by $s/\sqrt{n}$, which is positive, and rearrange each inequality to isolate $\mu_0$:

$$\bar{x} - t_{n-1,\,1-\alpha/2}\frac{s}{\sqrt{n}} < \mu_0 < \bar{x} + t_{n-1,\,1-\alpha/2}\frac{s}{\sqrt{n}}.$$

That is exactly the statement that $\mu_0$ lies inside the $100(1-\alpha)\%$ confidence interval. So the $100(1-\alpha)\%$ interval is the set of values $\mu_0$ that a two-sided level-$\alpha$ test would not reject. A value is rejected exactly when it lies outside the interval. The NIST handbook (§7.2.2.1) states the same equivalence: the 95% interval includes the null value if, and only if, the hypothesis would not be rejected at the 5% level. The one-sided version, a test against a lower confidence bound, is derived in One-Sided Confidence Bounds and Sample Size.

The equivalence can be drawn. Compute the two-sided $p$-value for every hypothesized value $\mu_0$, not only for zero. The result is a curve that peaks at $p = 1$ when $\mu_0$ equals the sample mean and falls away on both sides.

The two-sided p-value of the t test for Student's sleep data, computed for every hypothesized mean mu0 rather than only for zero. Notice that it equals 1 at the sample mean, 1.58 hours, and crosses p = 0.05 exactly at 0.70 and 2.46, the ends of the 95% confidence interval. The null value of zero, far out on the left, has p = 0.0028.
The two-sided p-value of the t test for Student's sleep data, computed for every hypothesized mean mu0 rather than only for zero. Notice that it equals 1 at the sample mean, 1.58 hours, and crosses p = 0.05 exactly at 0.70 and 2.46, the ends of the 95% confidence interval. The null value of zero, far out on the left, has p = 0.0028.

Check your understanding. Would a two-sided test at 0.05 reject the hypothesis that drug 2 adds a mean of 2.0 hours?

No. The value 2.0 lies inside the 95% interval, so the test cannot reject it. The table gives its $p$-value, 0.308.

Hypothesized mean $\mu_0$ (hours) 0 0.5 0.70 1.0 1.58 2.0 2.46 3.0
Two-sided $p$ 0.0028 0.0215 0.050 0.170 1.000 0.308 0.050 0.0053

The curve makes two points that a single test hides.

  • The value zero is not special to the data. It is one of many hypotheses, and the data are about as incompatible with a mean of 3 hours ($p = 0.0053$) as with a mean of zero.
  • The values inside the interval are not all equally compatible with the data. The value 1.58 fits perfectly and 0.70 only barely.

Greenland and colleagues (2016) recommend thinking of $p$-values this way, as a measure of the compatibility between the data and each candidate value under the model. They note that the exclusive focus on the null value obscures the close relationship between $p$-values and intervals.

A practical consequence is that an interval carries everything the test does and more. Suppose a report gives $\bar{d} = 1.58$ hours, 95% interval 0.70 to 2.46. A reader learns that a test of zero rejects at 0.05, and how large the effect plausibly is. With the 99% interval, $[0.316, 2.844]$, which excludes zero too, the reader also learns that a test at 0.01 would reject.

Reading a p-value

The $p$-value is among the most misread numbers in science. In 2016 the American Statistical Association issued, for the first time, a statement on a matter of statistical practice. It set out six principles for the use and interpretation of $p$-values, for researchers, practitioners and science writers who are not primarily statisticians (Wasserstein and Lazar, 2016). The accompanying editorial explains that the board was responding in part to concern about the reproducibility of published research. It deliberately left out alternative hypotheses, error types and power, which the next two articles of this topic take up.

Before reading the principles, try the most common misreading.

Check your understanding. A report on the sleep trial says: "$p = 0.0028$, so there is only a 0.28% chance that the two drugs are equally effective." Is that right?

It is not. The $p$-value was computed by supposing the drugs are equally effective. It is a probability about the data, given that supposition, so it cannot also be the probability of the supposition. The second principle below says so directly.

Here are the six principles, with what each means for a test of a mean.

1. "P-values can indicate how incompatible the data are with a specified statistical model." The smaller the $p$-value, the greater the incompatibility of the data with the null hypothesis, if the assumptions used to compute it hold. For the sleep data the model includes $\mu = 0$. It also includes that the ten differences are independent draws from a normal population. A small $p$-value casts doubt on the model as a whole, of which the null hypothesis is one part.

2. "P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone." For the sleep data, $p = 0.0028$ does not mean there is a 0.28% chance that the drugs are equally effective. Nor, read the other way, is $1 - p = 0.9972$ the probability that drug 2 is better. Student's 1908 reading of the one-sided value was "the odds are about 666 to one that 2 is the better soporific." That is exactly this inversion, and it is the first misinterpretation in Greenland and colleagues' list (2016, misinterpretation 1). Turning a $p$-value into a probability for a hypothesis needs a prior probability for the hypothesis and Bayes' theorem. It has the same structure as turning a test result into the probability of disease in Predictive Values and Bayes' Theorem.

3. "Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold." Nothing happens at 0.05 except a convention. The older-adult sample above has $p = 0.081$. By the threshold, it is "not significant." By the interval, the mean lies between 0.9 mm Hg below and 14.6 mm Hg above the all-adult value. The interval describes what the data say. The word "nonsignificant" describes only which side of a convenient line the $p$-value fell.

4. "Proper inference requires full reporting and transparency." A $p$-value means what it says only if it was not selected for being small. Three practices make the reported $p$-value uninterpretable:

  • running several analyses and reporting the one that reached significance;
  • choosing the side of a test after looking;
  • stopping data collection as soon as $p$ dips below 0.05.

Type I and Type II Errors measures how much each practice inflates the error rate.

5. "A p-value, or statistical significance, does not measure the size of an effect or the importance of a result." The $t$ statistic is the estimated effect divided by its standard error, and the standard error shrinks like $1/\sqrt{n}$. So any nonzero difference becomes significant with enough data, and any difference can fail to be significant with too little. The script checked both directions on the NHANES population.

Situation True difference from $\mu_0$ $n$ Median $p$ Share of samples with $p < 0.05$
All adults, $H_0: \mu = 125$ 1.36 mm Hg 5,000 $1.9 \times 10^{-6}$ 0.998
Aged 60 and over, $H_0: \mu = 126.36$ 10.49 mm Hg 10 0.142 0.248

A difference of 1.36 mm Hg is too small to matter to any patient. With 5,000 people it gives a typical $p$-value of two in a million. A difference of 10.5 mm Hg is large enough to matter a great deal. With 10 people it gives a typical $p$-value of 0.14 and is detected in only a quarter of samples. The $p$-value mixes the size of the effect with the size of the study; the estimate and its interval keep them apart.

6. "By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis." In particular, a large $p$-value is not evidence for the null hypothesis. The NHANES sample of 25 from all adults gave $p = 0.53$ for $H_0: \mu = 126.36$, which happens to be true. The older-adult sample gave $p = 0.081$ for the same $H_0$, which is false. Neither number, alone, says which case it came from. Greenland and colleagues (2016, misinterpretations 4 to 6) list the forms this error takes. They run from reading a nonsignificant result as showing that the test hypothesis is true, to concluding that "no effect was observed." Their remedy is the interval: it shows at once whether the data exclude effects that would matter.

The statement closes by noting that some statisticians supplement or replace $p$-values with other approaches. Some emphasize estimation, such as confidence intervals; others use Bayesian methods and likelihood ratios. Its last sentence is the shortest summary of all six principles: "No single index should substitute for scientific reasoning."

The common misreadings, in their usual words

In reports, the errors the principles guard against usually appear in a few familiar sentences.

  • "The $p$-value is the probability that the null hypothesis is true." For the sleep data that would put equal effectiveness at 0.28%. The $p$-value assumes $H_0$; it is not a probability for $H_0$ (principle 2; Greenland et al., 2016, misinterpretation 1).
  • "$1 - p$ is the probability that the effect is real." Student's "666 to one" is this inversion (principle 2).
  • "The $p$-value is the probability that the result is due to chance alone." The calculation supposes chance alone, so it cannot measure the probability of that supposition (principle 2; misinterpretation 2).
  • "$p > 0.05$, so there is no effect." The older-adult sample had $p = 0.081$ and a real difference of 10.49 mm Hg (principle 6; misinterpretations 4 to 6).
  • "$p$ is tiny, so the effect is large or important." The 1.36 mm Hg difference had a typical $p$ of two in a million (principle 5; misinterpretation 7).
  • "$p$ is large, so the effect is small." The 10.49 mm Hg difference had a typical $p$ of 0.14 with ten people (principle 5; misinterpretation 8).
  • "We rejected at 0.05, so the chance we are wrong is 5%." This confuses a long-run error rate with a single result (misinterpretation 10), as Type I and Type II Errors shows.

Assumptions

The $t$ test inherits the assumptions of the $t$ interval:

  • independent observations from a single population;
  • for exact error rates in small samples, a normal population.

Confidence Intervals for a Mean showed that the $t$ interval tolerates moderate skewness and fails for strong skewness. The test, being the same calculation, behaves the same way.

One detail matters more for tests than for intervals. Take a right-skewed population such as blood pressure. A one-sided test of $H_A: \mu < \mu_0$ rejects a true null much more often than $\alpha$, and the opposite one-sided test much less often. The cause is that samples that miss the long tail have both a low mean and a small standard deviation. Type I and Type II Errors measures the size of the effect.

The paired design of the sleep trial reduces to a one-sample test on the differences. It is developed further in The Paired Samples t-Test. For skewed data or outliers, the sign and signed-rank tests make weaker assumptions.

The same tests, applied to the per-topic differences between two search engines, are worked in the information retrieval wiki's Significance Testing. That article sets them alongside the sign test, the Wilcoxon test, the randomization test and the bootstrap, on one example where they disagree.

What we have learned

A test measures, in standard errors, how far the data fall from what the null hypothesis predicts. The $p$-value is the chance of a distance at least that large if the null hypothesis and the model hold: 0.0028 for the ten sleep patients, 0.53 for a true null in NHANES, 0.081 for a real difference studied with 25 people. A two-sided test rejects exactly the values outside the matching confidence interval, so the interval carries the verdict and the size of the effect. The $p$-value is not the probability that the null hypothesis is true, nor a measure of effect size.

The older-adult study left an uncomfortable fact: a test can reach the wrong verdict with no error in the arithmetic. How often, in each direction? That is the next article, Type I and Type II Errors. What sample size makes a miss rare is in Power and Sample Size for a Test of a Mean.

In Python

import numpy as np
from scipy import stats

d = np.array([1.2, 2.4, 1.3, 1.3, 0.0, 1.0, 1.8, 0.8, 4.6, 1.4])
res = stats.ttest_1samp(d, popmean=0)
print(res.statistic, res.pvalue)                  # 4.062, 0.00283
print(stats.ttest_1samp(d, 0, alternative='greater').pvalue)   # 0.00142
print(res.confidence_interval(0.95))              # about (0.700, 2.460)

# z test with known sigma: no library call needed
from statistics import NormalDist
z = (128.88 - 126.36) / (20.12 / 25 ** 0.5)
print(z, 2 * (1 - NormalDist().cdf(abs(z))))      # 0.626, 0.531

ttest_1samp reports a two-sided $p$-value unless alternative is set; the side should come from the study's question, not from the result. Every number in this article is computed by tools/experiments/applied-statistics/hypothesis-tests-for-a-mean.py in the site's repository, which implements the $t$ distribution directly and was checked against scipy 1.17.1.

References