The Paired Samples t-Test

Comparing two means when each subject is measured twice or subjects come in matched pairs. Paired and independent designs, why pairing removes the variation between subjects (the variance of a difference derived), the paired t-test as a one-sample test on the differences, worked on Student's sleep data with its interval and one-sided bound, what ignoring the pairing does to the same data, a large-scale check on repeated NHANES blood pressure readings, and when pairing costs more than it gains.

Ten patients in a trial of two sleeping drugs each took both drugs, on different nights, and slept on average 1.58 hours more on the second. Confidence Intervals for a Mean and Hypothesis Tests for a Mean treated the ten differences as one sample. They found a 95% interval of 0.70 to 2.46 hours and a two-sided $p$-value of 0.0028. Now analyze the same twenty numbers as if they came from twenty different patients, ten on each drug. The answer becomes $p = 0.079$, with an interval that includes zero. Why should the way a study was run change the analysis of the very same numbers?

The trial was run by Cushny and Peebles (1905), and Student (1908) used its data. Senn and Richardson (1994) discuss its design and how its data were later quoted. A design like this one, where each observation in one sample belongs with exactly one observation in the other, calls for the paired samples $t$-test. It is the one-sample $t$-test applied to the within-pair differences. The gap between $p = 0.0028$ and $p = 0.079$ is not a technicality. It is the difference between using the design of a study and ignoring it.

Here is the route. First we define paired and independent designs. Then the variance of a difference shows where the gain from pairing comes from. Next we set out the paired $t$-test in the notation of the course's formula sheet, and see what ignoring the pairing does. A large-scale check on repeated blood pressure readings from NHANES follows. Finally we find the one situation in which pairing loses. The article assumes the one-sample $t$-test of Hypothesis Tests for a Mean.

Paired and independent designs

Two samples are paired when each observation in one sample corresponds to exactly one observation in the other. The two members of a pair share something that the other observations do not. The NIST/SEMATECH e-Handbook of Statistical Methods (§1.3.5.3) describes this as a one-to-one correspondence between the values in the two samples. Pairs arise in two ways.

  • Self-pairing. The same subject is measured twice. Examples are before and after a treatment, under each of two treatments in a crossover trial (the sleep trial), with each of two instruments, or on the left and right eye. Each subject serves as their own comparison.
  • Matching. Different subjects are deliberately paired so that the two members are alike in characteristics that affect the outcome, such as age and sex. One member of each pair receives each treatment. Twins are the extreme case.

Two samples are independent when there is no such correspondence. The subjects in one group have nothing to do with the subjects in the other, and the groups may have different sizes. Any observation in the first group could be compared with any in the second equally well. A trial that randomizes patients to one drug or the other produces independent samples, and so does a survey comparing men with women.

Which kind of data a study has is settled by how the data were collected, not by looking at the numbers. Two columns of equal length are not paired unless the rows mean something.

Check your understanding. Suppose the drug 1 values of the sleep trial were shuffled among the ten patients, leaving the drug 2 values in place. What happens to the mean of the ten differences, and what happens to the pairing?

The mean of the differences is still 1.58 hours, because it equals the difference of the two column means, and shuffling a column does not change its mean. But the pairing, and the gain from it, would be gone: each drug 1 value would now sit next to a stranger's drug 2 value. The next section shows that the gain lives entirely in the correlation between a patient's two values, which a shuffle reduces to chance.

The reverse mistake is as real. A paired analysis of two independent groups that happen to have equal sizes pairs observations arbitrarily, and its result then depends on the arbitrary order.

Where the gain comes from

Here is the idea in one sentence. A difference between two measurements on the same subject cancels whatever that subject brings to both, so it varies less than a difference between two different subjects.

The formal version. Let $X_1$ and $X_2$ be the two measurements on a randomly chosen subject. They have means $\mu_1$ and $\mu_2$, standard deviations $\sigma_1$ and $\sigma_2$, and correlation $\rho$. The difference $D = X_1 - X_2$ has mean $\delta = \mu_1 - \mu_2$, the quantity of interest. Its variance follows from the rule for the variance of a sum:

$$\operatorname{Var}(D) = \operatorname{Var}(X_1) + \operatorname{Var}(X_2) - 2\operatorname{Cov}(X_1, X_2) = \sigma_1^2 + \sigma_2^2 - 2\rho\,\sigma_1\sigma_2.$$

Two subjects measured once each, one under each condition, give a difference whose variance is $\sigma_1^2 + \sigma_2^2$. Measurements on different subjects are independent, so the covariance term is zero. The same subject measured twice gives a difference whose variance is smaller by $2\rho\sigma_1\sigma_2$.

When subjects differ a great deal from one another, a subject who is high on one measurement tends to be high on the other. Then $\rho$ is large and positive, and most of the variance cancels. What cancels is exactly the variation between subjects: the patient who sleeps badly on both drugs, or the person whose blood pressure is high on every reading.

Check your understanding. Could pairing ever make the variance of a difference larger than $\sigma_1^2 + \sigma_2^2$?

Yes, if $\rho$ is negative: then $-2\rho\sigma_1\sigma_2$ is positive and adds to the variance. That would happen if subjects who are high on one measurement tended to be low on the other. It is rare for two measurements of the same quantity on the same person, but the identity shows why pairing helps only through a positive correlation.

Checking the identity on Student's data

The identity holds for samples too, with $s_1$, $s_2$ and the sample correlation $r$ in place of the population values. For Student's data, with $d_i$ the drug 2 value minus the drug 1 value:

Drug 1 Drug 2 Difference $d_i$
Mean (hours) 0.750 2.330 1.580
Standard deviation (hours) 1.7890 2.0022 1.2300

The correlation between the two drugs' values is $r = 0.7952$. Then

$$s_d^2 = s_1^2 + s_2^2 - 2 r s_1 s_2 = 3.2006 + 4.0090 - 2(0.7952)(1.7890)(2.0022) = 1.5129,$$

and $s_d = \sqrt{1.5129} = 1.2300$, the standard deviation of the ten differences computed directly. Without the covariance term the standard deviation would have been $\sqrt{3.2006 + 4.0090} = 2.6851$ hours, more than twice as large.

The patients differed far more from one another than the drugs did. The drug 1 values run from $-1.6$ to $3.7$ hours, while nine of the ten differences lie between 0.0 and 2.4.

Student's sleep data drawn two ways. Left: each patient's two values joined by a line. Notice that nine lines rise and one is flat, and most run roughly parallel, because patients who sleep well on one drug tend to sleep well on the other. Right: the same twenty numbers as two unrelated groups. The groups overlap heavily, and the shift between their means, 1.58 hours, looks small next to the spread within each.
Student's sleep data drawn two ways. Left: each patient's two values joined by a line. Notice that nine lines rise and one is flat, and most run roughly parallel, because patients who sleep well on one drug tend to sleep well on the other. Right: the same twenty numbers as two unrelated groups. The groups overlap heavily, and the shift between their means, 1.58 hours, looks small next to the spread within each.

The left panel is what a paired test sees, and the right panel is what an unpaired test sees. The next section turns the left panel into a test.

The paired t-test

Since $D$ is a single random variable, a paired comparison is a one-sample problem. The $n$ differences $d_1, \dots, d_n$ are a sample from the population of differences. The question about two means becomes a question about one, $\delta = \mu_1 - \mu_2$. Everything in Confidence Intervals for a Mean and Hypothesis Tests for a Mean applies to the differences unchanged. In the course's notation:

  • the differences are $d_i = x_{i1} - x_{i2}$, with mean $\bar{d} = \frac{1}{n}\sum d_i$ and standard deviation $s_d = \sqrt{\sum (d_i - \bar{d})^2/(n-1)}$, where $n$ is the number of pairs, not the number of measurements;
  • the hypotheses are $H_0: \delta = \mu_1 - \mu_2 = 0$ (equivalently $\mu_1 = \mu_2$) against $H_A: \delta \ne 0$, or a one-sided alternative;
  • the test statistic is

$$t = \frac{\bar{d} - \delta}{s_d/\sqrt{n}},$$

evaluated at the null value of $\delta$. It follows a $t$ distribution with $n - 1$ degrees of freedom when $H_0$ is true and the differences come from a normal population;

  • the $100(1-\alpha)\%$ confidence interval for $\delta$ is

$$\bar{d} \pm t_{n-1,\,1-\alpha/2}\,\frac{s_d}{\sqrt{n}},$$

and the one-sided lower bound is $\bar{d} - t_{n-1,\,1-\alpha}\,s_d/\sqrt{n}$ when only an increase is of interest.

The null value need not be zero. The NIST handbook (§1.3.5.3) notes that a new treatment may be worth adopting only if it beats the current one by some threshold. The null hypothesis then states that the difference equals that threshold, and $\delta$ in the numerator is the threshold.

Student's sleep data

Confidence Intervals for a Mean worked the calculation step by step, so only the results are collected here. With $n = 10$ pairs, $\bar{d} = 1.580$ and $s_d = 1.2300$, the standard error is $1.2300/\sqrt{10} = 0.3890$ hours and

$$t = \frac{1.580 - 0}{0.3890} = 4.062, \qquad p = 2\,P(T_9 \ge 4.062) = 0.0028.$$

The 95% interval is $1.580 \pm 2.2622 \times 0.3890 = [0.700, 2.460]$ hours. Suppose instead the trial had been designed to ask only whether drug 2 adds sleep. Then the one-sided 95% lower bound is

$$1.580 - t_{9,\,0.95} \times 0.3890 = 1.580 - 1.8331 \times 0.3890 = 0.867 \text{ hours},$$

and the one-sided $p$-value is 0.0014. The data are consistent with drug 2 adding at least about 0.87 hours. The side of the bound has to come from the protocol, not from the data, as One-Sided Confidence Bounds and Sample Size explains.

That is the analysis as designed. What would a careless analysis of the same numbers report?

Ignoring the pairing

Suppose the twenty numbers had been analyzed as two independent groups of ten, as if twenty different patients had taken part. The two-sample $t$-test of the next article compares the difference in means, which is still 1.580 hours, with a standard error built from the spread within each group:

$$s_p^2 = \frac{9(1.7890)^2 + 9(2.0022)^2}{18} = 3.6048, \qquad SE = s_p\sqrt{\frac{1}{10} + \frac{1}{10}} = 1.8986 \times 0.4472 = 0.8491.$$

Then $t = 1.580/0.8491 = 1.861$ on 18 degrees of freedom, and $p = 0.079$. The 95% interval for the difference is $[-0.204, 3.364]$ hours. Welch's version of the test does not assume equal variances. It gives the same $t$ (the groups have equal sizes) on 17.78 degrees of freedom, and $p = 0.079$.

Analysis Standard error (hours) $t$ df $p$ 95% interval (hours)
Paired $t$ (as designed) 0.3890 4.062 9 0.0028 $[0.700, 2.460]$
Two-sample $t$ (pairing ignored) 0.8491 1.861 18 0.079 $[-0.204, 3.364]$

The estimate is identical, and only its standard error differs, by a factor of $0.8491/0.3890 = 2.18$. The unpaired analysis is not significant at 0.05.

The unpaired standard error is not wrong for an experiment with twenty different patients. It is wrong for this one, because it charges the comparison with the variation between patients. Every patient here took both drugs, so that variation never entered the comparison. The unpaired analysis also gets the degrees of freedom wrong in the opposite direction: 18 instead of 9, as if there were twenty independent pieces of information about the drugs rather than ten.

Errors of this kind are easy to make with software, because the paired and unpaired tests take the same two columns of numbers. The information retrieval wiki's Significance Testing shows the same mistake on a comparison of two search engines over the same queries. There, ignoring the pairing turns $p = 0.001$ into $p = 0.27$.

Ten patients are one sample. To see what ignoring the pairing does in general, we need a population of pairs whose truth we know.

A large-scale check: repeated blood pressure readings

The NHANES examination of 2017 to 2018 is the source of the blood pressure population used throughout this course. It recorded several systolic readings for each participant in succession at the same examination. The 4,782 adults who have both a first and a third reading form a population of pairs: a self-paired design with a known truth.

Reading 1 Reading 3 Difference (1 minus 3)
Mean (mm Hg) 125.79 125.00 0.786
Standard deviation (mm Hg) 19.65 19.35 5.629

The correlation between the two readings is 0.9584. In this population the first reading is higher than the third by 0.79 mm Hg on average, and the population splits three ways:

  • 46.7% of adults have a higher first reading;
  • 15.8% have identical readings (the readings are recorded in steps of 2 mm Hg);
  • 37.5% have a lower first reading.

The difference is small next to the variation between people, a standard deviation of about 19.5 mm Hg for a single reading. But the two readings of one person agree closely, and the variance identity shows how much that matters. Without pairing, the standard deviation of a difference would be $\sqrt{19.65^2 + 19.35^2} = 27.58$ mm Hg. With it, the standard deviation is 5.63 mm Hg, about one fifth.

One study of 40 adults

A random sample of 40 of these adults (the script's seed 1) shows the effect on a single study.

  • Paired analysis: $\bar{d} = 0.900$, $s_d = 7.203$, standard error $1.139$, $t = 0.790$ on 39 degrees of freedom, $p = 0.43$, and a 95% interval of $[-1.40, 3.20]$ mm Hg.
  • Unpaired analysis of the same 80 readings: standard error $3.596$, $t = 0.250$, $p = 0.80$, and an interval of $[-6.26, 8.06]$ mm Hg, more than three times as wide.

Neither rejects, because the true difference is too small for 40 people to detect. But the paired interval already rules out a difference larger than about 3 mm Hg in either direction. The unpaired one rules out almost nothing of practical interest.

Many studies

Check your understanding. The unpaired test uses the wrong standard error. When there is no true difference, will it reject too often, or too rarely?

Too rarely, and by a wide margin. The script drew 10,000 samples of each size from the population of pairs, for each of 30 seeds, and applied both tests at $\alpha = 0.05$ (medians over the seeds). With the population shifted so that the mean difference was exactly zero, the paired test rejected in 4.95% of samples of 25 and 5.01% of samples of 100, as a test at $\alpha = 0.05$ should. The unpaired test never rejected at all.

The same simulation, on the unshifted population, measures the gap in power:

Adults in the sample Power, paired $t$ Power, two-sample $t$ (pairing ignored)
25 0.098 0.000
50 0.160 0.000
100 0.281 0.000
200 0.503 0.000
400 0.797 0.000

With 400 adults the paired test detects the 0.79 mm Hg difference four times in five. The unpaired test, at these sizes, never does.

The variance identity explains why. The unpaired standard error is built as if the spread of a difference were 27.58 mm Hg. The actual spread of the difference in means comes from the paired differences, 5.63 mm Hg, about a fifth as much. So the unpaired $t$ statistic varies from sample to sample with a standard deviation of about $5.63/27.58 = 0.20$ instead of 1. With no true difference, it would have to land about ten of its own standard deviations from zero to pass 1.96. Even with 400 adults, where the true difference puts its center near 0.57, it would need about seven. Neither happened in any sample.

Ignoring a positive correlation does not make a test reject too often. It makes the test so cautious that it rarely rejects anything.

When pairing costs more than it gains

So far pairing has only helped. Is there a design in which it hurts?

Pairing has a price, paid in degrees of freedom. With $n$ pairs the paired test has $n - 1$ degrees of freedom, while two independent groups of $n$ subjects give a two-sample test with $2n - 2$. When $\rho$ is close to zero, the paired design gains nothing in variance and loses half its degrees of freedom. It is then slightly less powerful than an independent-groups design with the same number of measurements.

The size of the effect can be computed exactly. For a paired design with $n$ pairs and equal standard deviations $\sigma$, the differences have standard deviation $\sigma\sqrt{2(1 - \rho)}$. The power of the paired test follows from the noncentral $t$ distribution with $n - 1$ degrees of freedom, defined in Power and Sample Size for a Test of a Mean. For two independent groups of $n$, the power of the two-sample test comes from the noncentral $t$ with $2n - 2$ degrees of freedom.

Check your understanding. With 10 pairs, a true difference of one standard deviation and $\rho = 0$, which design has more power: 10 pairs, or two independent groups of 10?

The two groups, 0.562 against 0.514. With no correlation the variances match, and the extra degrees of freedom decide it. The table shows how quickly a correlation reverses the order:

Correlation $\rho$ 0 0.05 0.10 0.20 0.50 0.80
Power, 10 pairs 0.514 0.535 0.557 0.606 0.803 0.993
Power, two groups of 10 0.562 0.562 0.562 0.562 0.562 0.562
Power of the paired t-test with 10 pairs (solid) against the two-sample t-test with two independent groups of 10 (dashed), for a true difference of one standard deviation at alpha = 0.05, as the correlation between a subject's two measurements varies. At zero correlation pairing loses (0.51 against 0.56) because it has 9 degrees of freedom instead of 18. Notice where the curves cross, at a correlation of about 0.11: above it pairing wins by a wide margin.
Power of the paired t-test with 10 pairs (solid) against the two-sample t-test with two independent groups of 10 (dashed), for a true difference of one standard deviation at alpha = 0.05, as the correlation between a subject's two measurements varies. At zero correlation pairing loses (0.51 against 0.56) because it has 9 degrees of freedom instead of 18. Notice where the curves cross, at a correlation of about 0.11: above it pairing wins by a wide margin.

The break-even correlation is 0.11 for this design and 0.03 for a larger one (30 pairs, a difference of half a standard deviation), because the loss of degrees of freedom matters less as $n$ grows. Measurements on the same person are usually correlated far more strongly than that: 0.80 for the sleep data and 0.96 for repeated blood pressure readings.

The practical lesson is that self-pairing almost always pays, while matching pays only when the matching variables are related to the outcome. Matching on characteristics that do not predict the outcome produces pairs with $\rho$ near zero. The analysis then pays the price in degrees of freedom for nothing.

A paired design can also be less convenient in ways that have nothing to do with variance.

  • A crossover trial assumes that the first treatment's effect is gone before the second begins, and that the order of treatment does not matter. A drug with lasting effects, or a disease that changes over the course of the trial, breaks the comparison.
  • A before-and-after study without a control group measures change, not its cause. Anything else that happened between the two measurements is mixed into the mean difference. So is any tendency of unusually high first measurements to be followed by more ordinary ones.
  • A pair with one measurement missing contributes nothing to a paired analysis.

Assumptions

The paired $t$-test assumes that the pairs are independent of one another. For exact error rates in small samples, it also assumes that the differences come from a normal population.

The normality assumption is about the differences, not about the original measurements. The sleep values for each drug are spread over several hours, but the test only needs the ten differences to look like a sample from a normal distribution. Differences between two similar measurements are often more symmetric than the measurements themselves. For strongly skewed differences or outliers, the sign test and the Wilcoxon signed-rank test apply to the same differences with weaker assumptions. Fisher's check of the sleep data by the signs of the differences is described in Hypothesis Tests for a Mean.

What we have learned

A paired design turns a comparison of two means into a one-sample problem on the differences. The variance of a difference, $\sigma_1^2 + \sigma_2^2 - 2\rho\sigma_1\sigma_2$, shows that the gain comes entirely from the correlation within pairs. On Student's data that correlation, 0.80, cut the standard error by a factor of 2.18 and turned $p = 0.079$ into $p = 0.0028$. On repeated blood pressure readings, ignoring the pairing left a test that almost never rejected. Pairing loses only when the correlation is near zero, below about 0.11 for ten pairs. The number of pairs a paired study needs, and how much smaller it is than a parallel-group study, is worked out in Sample Size for Comparing Two Means.

Many comparisons cannot be paired at all: men and women, or patients randomized to one drug or the other. Without a correlation to remove, the variation in both groups enters the standard error. The Two-Sample t-Test: Pooled and Welch builds the test for that case, and asks what to do when the two groups vary by different amounts.

In Python

import numpy as np
from scipy import stats

drug1 = np.array([0.7, -1.6, -0.2, -1.2, -0.1, 3.4, 3.7, 0.8, 0.0, 2.0])
drug2 = np.array([1.9, 0.8, 1.1, 0.1, -0.1, 4.4, 5.5, 1.6, 4.6, 3.4])

res = stats.ttest_rel(drug2, drug1)            # paired: same as ttest_1samp(drug2 - drug1, 0)
print(res.statistic, res.pvalue)                # 4.062, 0.00283
print(res.confidence_interval(0.95))            # about (0.700, 2.460)
print(stats.ttest_rel(drug2, drug1, alternative='greater').confidence_interval(0.95))  # lower bound 0.867

wrong = stats.ttest_ind(drug2, drug1)           # pairing ignored
print(wrong.statistic, wrong.pvalue)            # 1.861, 0.079

ttest_rel and ttest_ind accept the same two arrays, so the software cannot tell which design produced them; the analyst has to. Every number in this article is computed by tools/experiments/applied-statistics/paired-samples-t-test.py in the site's repository. The script implements the $t$ and noncentral $t$ distributions directly and was checked against scipy 1.17.1.

References