Confidence Intervals and Coverage

What a confidence interval for a mean is and what its confidence level promises. The interval derived from the sampling distribution of the mean, a population of 5,247 adults' blood pressures from NHANES used to draw 100 intervals and then 300,000, coverage at three confidence levels, why the 95% describes the method rather than any one interval, the price of higher confidence, and what happens to coverage when the sample is not random.

A health department wants to know the average systolic blood pressure of the adults it serves. It cannot measure everyone, so it measures a random sample of 25 and finds a mean of 128.9 mm Hg. That is its best single guess. It is also almost certainly wrong: another 25 adults would have given a different mean. So the useful question is not "what is the average?" but "how far from the average could this guess be?"

The standard answer is a confidence interval: a range computed from the sample, such as 121.0 to 136.8 mm Hg, reported with a confidence level, such as 95%. This article is about what that pair of numbers means. The 95% is easy to state and easy to misread, so we will not take it on trust. We will check it.

The plan is simple. First we set up a population whose true mean we know, so that every interval can be graded right or wrong. Then we derive the interval from the behavior of sample means. Next we test the promise behind the 95%, first with a hundred intervals and then with 300,000. Finally we look at the common misreadings, at what higher confidence costs, and at the one failure no formula can fix.

This article assumes the population standard deviation is known, which keeps the mathematics clean. The realistic case, where it must be estimated from the same sample, needs Student's $t$ distribution and is the subject of Confidence Intervals for a Mean.

A population where we know the answer

To check a method of estimation, we need a case where the answer is known. The National Health and Nutrition Examination Survey (NHANES) supplies one. It measures the blood pressure of a sample of US residents in a mobile examination center: three consecutive readings after five minutes of seated rest, with a fourth attempt if one is interrupted (National Center for Health Statistics, 2020).

In the 2017 to 2018 cycle, 5,247 adults aged 18 and over had at least one of the three systolic readings. Averaging each adult's available readings gives one value per person. This article treats those 5,247 values as a complete population.

That is a teaching device. NHANES is itself a sample, designed so that national estimates use sampling weights, and nothing here estimates national blood pressure. What matters is that this population is real, it is not a textbook bell curve, and its mean is known exactly.

Population summary Value
Size $N$ 5,247 adults
Mean $\mu$ 126.36 mm Hg
Standard deviation $\sigma$ 20.12 mm Hg
Median 123.3 mm Hg
Range 72.7 to 238.0 mm Hg
Skewness 1.04

The distribution is skewed to the right. Most adults are between 100 and 150, and a long tail of high values pulls the mean above the median (Measures of Center and Spread describes its shape in detail). A real study samples from a population like this without knowing $\mu$. Here we sample knowing $\mu$, so we can count how often the method's answer is right.

How far does a sample mean wander?

Draw $n = 25$ adults at random and average their blood pressures. Call the result $\bar{X}$. The capital letter marks a random quantity: before the sample is drawn, $\bar{X}$ could take many values. The value from one particular sample is written $\bar{x}$.

How $\bar{X}$ varies over repeated samples is its sampling distribution. That article proves three facts about it, and they are all we need. If the $n$ observations are drawn independently from a population with mean $\mu$ and standard deviation $\sigma$, then:

  1. the sampling distribution is centered at the population mean: $E(\bar{X}) = \mu$;
  2. its standard deviation, called the standard error of the mean, is $\sigma/\sqrt{n}$;
  3. for large enough $n$ it is approximately normal, whatever the shape of the population (the central limit theorem); for a normal population it is exactly normal for every $n$.

For samples of 25 from the blood pressure population, the standard error is $20.12/\sqrt{25} = 4.025$ mm Hg. Sample means are five times less spread out than individual adults.

The figure shows this directly. The script drew 10,000 samples of 25 (with replacement, so the draws are independent) and recorded each mean. The means averaged 126.31 mm Hg with a standard deviation of 4.009, against the theoretical 126.36 and 4.025. Their histogram is close to symmetric even though the population is not.

Top: systolic blood pressure of the 5,247 NHANES adults, skewed to the right with a standard deviation of 20.1 mm Hg. Bottom: the means of 10,000 random samples of 25 from the same population, on the same axis. Notice how much narrower the bottom histogram is (standard deviation 4.0 mm Hg) and how much more symmetric. That narrow, nearly normal spread is what the interval is built from.
Top: systolic blood pressure of the 5,247 NHANES adults, skewed to the right with a standard deviation of 20.1 mm Hg. Bottom: the means of 10,000 random samples of 25 from the same population, on the same axis. Notice how much narrower the bottom histogram is (standard deviation 4.0 mm Hg) and how much more symmetric. That narrow, nearly normal spread is what the interval is built from.

Turning "the mean is usually close" into an interval

Here is the idea in one sentence. If the sample mean lands within 7.89 mm Hg of $\mu$ in 95% of samples, then $\mu$ lies within 7.89 mm Hg of the sample mean in those same samples. Distance is symmetric. So the range $\bar{x} \pm 7.89$ will contain $\mu$ in 95% of samples, and we can compute that range without knowing $\mu$.

The number 7.89 is $1.96$ standard errors, and the rest of this section makes the argument exact.

The formal version. Standardizing the sample mean gives a quantity whose distribution does not depend on $\mu$ or $\sigma$:

$$Z = \frac{\bar{X} - \mu}{\sigma/\sqrt{n}} \sim N(0, 1)\ \text{(exactly or approximately)}.$$

Write $z_{1-\alpha/2}$ for the point with probability $1 - \alpha/2$ below it under the standard normal curve. The middle of the curve, between $-z_{1-\alpha/2}$ and $z_{1-\alpha/2}$, then holds probability $1 - \alpha$, with $\alpha/2$ in each tail. For 95% confidence, $\alpha = 0.05$ and $z_{0.975} = 1.960$. Before the sample is drawn,

$$P\left(-z_{1-\alpha/2} \le \frac{\bar{X} - \mu}{\sigma/\sqrt{n}} \le z_{1-\alpha/2}\right) = 1 - \alpha.$$

Now rewrite the event inside the parentheses without changing it. Multiply through by $\sigma/\sqrt{n}$, which is positive: $-z_{1-\alpha/2}\,\sigma/\sqrt{n} \le \bar{X} - \mu \le z_{1-\alpha/2}\,\sigma/\sqrt{n}$. Subtract $\bar{X}$, then multiply by $-1$, which reverses both inequalities:

$$P\left(\bar{X} - z_{1-\alpha/2}\frac{\sigma}{\sqrt{n}} \le \mu \le \bar{X} + z_{1-\alpha/2}\frac{\sigma}{\sqrt{n}}\right) = 1 - \alpha.$$

It is the same event, so it has the same probability. Read it carefully, because it is easy to read backwards. The population mean $\mu$ is a fixed number. What is random here are the endpoints, because they are built from $\bar{X}$. The statement says that the random interval

$$\left[\bar{X} - z_{1-\alpha/2}\frac{\sigma}{\sqrt{n}},\ \ \bar{X} + z_{1-\alpha/2}\frac{\sigma}{\sqrt{n}}\right]$$

will land on $\mu$ with probability $1 - \alpha$. Once a sample has been drawn and $\bar{x}$ substituted, the result is the $100(1-\alpha)\%$ confidence interval for $\mu$:

$$\bar{x} \pm z_{1-\alpha/2}\frac{\sigma}{\sqrt{n}}.$$

The NIST/SEMATECH e-Handbook of Statistical Methods (§7.1.4) gives this interval for a normal population with known standard deviation, writing $\bar{Y}$ and $N$ for the sample mean and sample size. Its parts have names:

  • the point estimate is $\bar{x}$;
  • the margin of error is $z_{1-\alpha/2}\,\sigma/\sqrt{n}$, the critical value times the standard error;
  • the interval is point estimate plus or minus margin of error.

That last form recurs for almost every interval in this course, so it is worth recognizing on sight.

The pivot. The trick of the derivation also deserves a name, because we will use it again and again. The quantity $Z$ involves the unknown $\mu$, yet its distribution is completely known. Such a quantity is called a pivot. A probability statement about the pivot is turned around into a statement about the parameter. Confidence Intervals for a Mean does the same with a different pivot when $\sigma$ is unknown.

One sample, three levels

Back to the health department. The script's first sample of 25 adults (seed 1) has mean $\bar{x} = 128.88$ mm Hg. With $\sigma = 20.12$ treated as known, the standard error is 4.025. The three common confidence levels give:

Level $1 - \alpha$ $z_{1-\alpha/2}$ Margin of error Interval (mm Hg) Width
90% 1.645 6.62 [122.26, 135.50] 13.24
95% 1.960 7.89 [120.99, 136.77] 15.78
99% 2.576 10.37 [118.51, 139.25] 20.74

All three contain the true mean of 126.36. The department, of course, would not know that. It has one interval and no way to grade it. That is why the next question is about the method rather than the interval: how often does it succeed?

What 95% promises: a hundred intervals

The 95% is a property of the procedure. Draw many samples, compute an interval from each, and about 95% of the intervals will contain $\mu$. The NIST handbook (§1.3.5.2) states it this way and adds the warning that matters: a particular computed interval either contains the true mean or does not, and the confidence level belongs to the method of calculation.

The figure makes this long-run statement visible. It shows the 95% intervals from the first 100 of the script's 10,000 samples of 25. Every interval has the same width, 15.78 mm Hg, because $\sigma$ is known. They differ only in where they are centered.

95% confidence intervals from 100 random samples of 25 adults, each centered on its own sample mean (dot). The horizontal line is the true population mean, 126.4 mm Hg. Nine intervals, drawn thicker in orange, miss it: five lie entirely above, four entirely below. Without the horizontal line, nothing would set the misses apart from the hits: each has the same width, and each sample's analyst would have had no way to tell whether theirs was one of the nine.
95% confidence intervals from 100 random samples of 25 adults, each centered on its own sample mean (dot). The horizontal line is the true population mean, 126.4 mm Hg. Nine intervals, drawn thicker in orange, miss it: five lie entirely above, four entirely below. Without the horizontal line, nothing would set the misses apart from the hits: each has the same width, and each sample's analyst would have had no way to tell whether theirs was one of the nine.

Nine of the 100 intervals miss. A 95% level suggests five, so is something wrong?

No. The number of misses among 100 independent intervals is itself random. When the true miss rate is 5%, it is a binomial count with mean 5 and standard deviation 2.18, and nine or more misses happens with probability 0.063. One hundred intervals are enough to show the idea and too few to measure the rate. Measuring it needs many more.

Measuring coverage

The share of intervals that contain the true value, over all possible samples, is the method's coverage probability. When the assumptions behind the derivation hold, coverage equals the confidence level. Whether it does in a given situation is a question simulation can answer.

The script draws 10,000 samples for each of 30 random seeds, computes the interval from each, and records the share that contain $\mu = 126.36$. The table reports the median of the 30 shares and their interquartile range (IQR), the middle half of the 30 results.

$n$ Confidence level Coverage, median of 30 runs IQR
25 90% 0.9013 0.8986 to 0.9029
25 95% 0.9510 0.9496 to 0.9518
25 99% 0.9891 0.9886 to 0.9900
5 95% 0.9524 0.9514 to 0.9536
100 95% 0.9498 0.9485 to 0.9512

The intervals do what they claim, to within the simulation's own noise of about half a percentage point. Two details are worth a closer look.

First, the 99% interval covers slightly less often than promised: 98.9%. The population's long right tail leaves the sampling distribution of a mean of 25 slightly skewed, with slightly heavier tails than a normal curve. A 99% interval relies on the far tails, and that is where the small departure shows.

Second, even samples of 5 reach 95% coverage. That may be surprising, given how skewed the population is. With $\sigma$ known, only the shape of the sampling distribution of $\bar{X}$ matters, and for this moderately skewed population five observations already make it nearly normal. When $\sigma$ has to be estimated, small samples are much less forgiving, as Confidence Intervals for a Mean shows.

Reading an interval correctly

Because the probability attaches to the procedure, some natural-sounding statements about a computed interval are wrong. Greenland and colleagues (2016) catalog common misinterpretations of statistical results, and several concern confidence intervals. Before reading on, try one.

Check your understanding. The department reports its 95% interval, 120.99 to 136.77 mm Hg. A board member summarizes: "So there is a 95% chance that the true average is between 121 and 137." Is that right?

It is not, though it is the most common reading of all. Once the numbers are computed, nothing in the statement is random under the model. Either 126.36 lies between them or it does not (here it does). The 95% is the frequency with which intervals computed this way contain the true value over very many studies, provided the assumptions used to compute them hold (Greenland et al., 2016, misinterpretation 19). In the figure of a hundred intervals, the board member's interval is one bar; the 95% describes the whole column of bars. An interval that does carry a probability for the parameter needs an additional ingredient, a prior distribution, and is a Bayesian credible interval, a different object.

Two other misreadings are common enough to name.

  • "95% of future samples will have means inside this interval." The interval is built to catch the population mean, not other sample means. A second study's mean wanders around $\mu$ independently of the first. The two means therefore differ by a quantity with standard error $\sqrt{2}\,\sigma/\sqrt{n}$. The chance that the second falls within $1.96\,\sigma/\sqrt{n}$ of the first is $2\Phi(1.96/\sqrt{2}) - 1 = 0.834$, where $\Phi$ is the standard normal distribution function. Greenland et al. (2016, misinterpretation 22) give the same 83% figure, and the script's simulation agrees: in 30 runs of 10,000 pairs of samples, the median share was 0.835.
  • "Values outside the interval have been ruled out." The interval summarizes which values of $\mu$ are reasonably consistent with the data under the model. Calling a value outside it incompatible with the data depends on the model's assumptions as well as the data. It also depends on a 95% cut-off that is arbitrary (Greenland et al., 2016, misinterpretation 20). A value just outside the interval is only slightly less consistent with the data than one just inside.

One way to hold all of this together: we have 95% confidence in the procedure, and the computed interval is the procedure's answer for this sample. The confidence describes the track record, not the case.

The price of confidence

If 95% is good, why not 99.9%? The table of three levels shows what it would cost. The interval's width is $2z_{1-\alpha/2}\,\sigma/\sqrt{n}$, and three things set it.

  • The confidence level, through $z_{1-\alpha/2}$. Moving from 90% to 99% raises the critical value from 1.645 to 2.576 and the width from 13.24 to 20.74 mm Hg. A 100% interval would have to be infinitely wide, and so useless. Width is the price paid for reliability.
  • The spread of the population, $\sigma$. More variable populations give wider intervals, and no analysis can change that.
  • The sample size, through $\sqrt{n}$. Quadrupling the sample halves the width. This is the one factor a study controls. Choosing $n$ to reach a target width is the subject of One-Sided Confidence Bounds and Sample Size.

The choice of 95% is a convention, not a law. The NIST handbook (§1.3.5.2) notes that the choice is somewhat arbitrary, and that 90%, 95% and 99% are the levels most used, with 95% the most common.

What coverage depends on

So far every simulation has honored the derivation's three assumptions: the observations are a random sample, they are independent of one another, and $\bar{X}$ has a normal sampling distribution with the stated $\sigma$. That is why coverage came out at the nominal level. What happens when an assumption fails? The first one is the most dangerous, because no formula can repair it.

A sample from the wrong population. Suppose the department's sample came not from all adults but from a clinic whose patients are all 60 or older. In the NHANES population, the 1,923 adults aged 60 and over have a mean systolic pressure of 136.85 mm Hg, about 10.5 mm Hg above the all-adult mean of 126.36. The script repeated the coverage experiment, drawing every sample of 25 from this older group and asking whether the 95% interval contained the all-adult mean.

The median coverage over 30 runs was 0.274. The intervals are just as narrow as before and look just as authoritative. Yet nearly three quarters of them miss, because they are centered on the wrong population. The margin of error describes random sampling error only. It says nothing about bias from who was sampled, which only the design of the study can prevent.

Correlated observations. Independence matters too. Repeated measurements on the same person, or patients clustered within hospitals, are correlated. Then the standard error $\sigma/\sqrt{n}$ understates the real uncertainty and the interval is too narrow. Linear Regression Assumptions shows the same failure for the interval around a regression slope, where correlated errors cut coverage roughly in half.

A non-normal sampling distribution. The third assumption is the most forgiving of the three for a mean, as the samples of 5 above showed. It becomes important once $\sigma$ must be estimated.

A short history

Results were long reported as an estimate plus or minus a multiple of its "probable error", the half-width of a 50% interval for a normal distribution. Student's 1908 paper, which introduced the $t$ distribution, is titled "The probable error of a mean" and still states its results that way.

Confidence intervals in the modern sense were proposed by Jerzy Neyman (1937), for exactly the property this article has checked by simulation: computed repeatedly in valid applications, 95% intervals contain the true value 95% of the time on average (Greenland et al., 2016). Their competitor, the Bayesian interval, does give a probability for the parameter, at the cost of a prior distribution. Much of the confusion that Greenland et al. (2016) catalog comes from reading one kind of interval as if it were the other.

What we have learned

A confidence interval is point estimate plus or minus margin of error, and it comes from turning around a probability statement about a pivot. Its confidence level is a promise about the method: across many samples, that share of intervals will contain the true mean. The NHANES simulations confirmed the promise when the assumptions hold, showed what one interval cannot tell us, and showed that no margin of error protects against sampling the wrong population.

One assumption remains unrealistic: we treated $\sigma$ as known. Replacing it with the sample standard deviation looks like a small step, but for small samples it quietly breaks the 95%. Repairing it is what Student did in 1908, and it is the subject of the next article, Confidence Intervals for a Mean: z and t.

In Python

The interval needs only the normal quantile, available in the standard library.

from statistics import NormalDist
import math

xbar, sigma, n = 128.88, 20.12, 25
for level in (0.90, 0.95, 0.99):
    z = NormalDist().inv_cdf(1 - (1 - level) / 2)
    margin = z * sigma / math.sqrt(n)
    print(level, round(z, 4), round(xbar - margin, 2), round(xbar + margin, 2))
# 0.9  1.6449 122.26 135.5
# 0.95 1.96   120.99 136.77
# 0.99 2.5758 118.51 139.25

Every number in this article, including the coverage simulations, is computed by tools/experiments/applied-statistics/confidence-intervals-and-coverage.py in the site's repository, which downloads the two NHANES files it uses from the CDC and checks their checksums.

References

  • Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., and Altman, D. G. (2016). Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. European Journal of Epidemiology, 31(4), 337–350. record · 5 pages
  • National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018: Demographic Variables and Sample Weights (DEMO_J) and Blood Pressure (BPX_J) data files. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 45 pages
  • Neyman, J. (1937). Outline of a theory of statistical estimation based on the classical theory of probability. Philosophical Transactions of the Royal Society of London, Series A, 236(767), 333–380. record
  • NIST/SEMATECH. e-Handbook of Statistical Methods, §1.3.5.2 (Confidence limits for the mean) and §7.1.4 (What are confidence intervals?). National Institute of Standards and Technology. https://www.itl.nist.gov/div898/handbook/ record · 30 pages
  • Student (1908). The probable error of a mean. Biometrika, 6(1), 1–25. record · 11 pages