The Sampling Distribution of the Mean

How a sample mean varies from sample to sample. The population, the sample and the sampling distribution distinguished on NHANES blood pressures; every sample of a five-person population enumerated; proofs that the sample mean is centered on the population mean and has standard error sigma over root n; the finite population correction for sampling without replacement, derived and checked; the standard error versus the standard deviation; how the shape of the sampling distribution changes with n; and the sampling distribution of the median for comparison.

A health department measures the systolic blood pressure of 25 adults drawn at random from the population it serves, and reports their mean. Had it drawn a different 25 adults, it would have reported a different mean. So the reported mean is one draw from a whole range of possible means. Where is that range centered, and how wide is it?

We can watch the range directly. Treat the 5,247 adults of the National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018 as the population. Their mean is 126.36 mm Hg. Five samples of 25 drawn from them give (all values in mm Hg):

Sample Mean Median Standard deviation Smallest Largest
1 122.39 124.00 17.08 92.7 176.0
2 125.20 122.67 17.38 94.0 158.7
3 131.49 124.67 27.43 101.3 217.3
4 130.36 127.33 22.84 99.3 188.7
5 127.81 128.00 21.24 93.3 175.3

None of the five means equals 126.36, and they range over 9 mm Hg. Each is a reasonable estimate, and each is wrong by an amount the department cannot know. What can be known is how far a mean of 25 tends to fall from the truth, because the variation from sample to sample follows rules.

The distribution of a statistic over all possible samples is its sampling distribution. The standard deviation of the sampling distribution of a mean is the standard error of the mean. This article is the course's home for both ideas. The confidence intervals and hypothesis tests of later units are statements about this sampling distribution, and they build on the results proved here.

First we separate three distributions that are easy to confuse, and list every sample of a population of five. Then we prove where sample means are centered and how widely they spread, with and without replacement. Finally we separate the standard error from the standard deviation and look at the shape of the sampling distribution. The proofs use the expectation and variance rules of The Binomial Distribution.

Three distributions

Three different distributions are involved, and most confusion about sampling comes from mixing them up.

  • The population distribution is the distribution of the measurement over everyone of interest. Here it is the systolic pressure of the 5,247 adults. It has mean $\mu = 126.36$ mm Hg, standard deviation $\sigma = 20.12$ mm Hg and a long right tail, with skewness 1.04. Because this is a complete population, $\sigma$ uses the divisor $N$ (see Measures of Center and Spread). The summaries $\mu$ and $\sigma$ are parameters: fixed numbers, usually unknown.
  • The distribution of a sample is the distribution of the $n$ values actually observed. It is a rough, small-scale copy of the population distribution. Its summaries $\bar{x}$ and $s$ are statistics, numbers computed from data.
  • The sampling distribution of a statistic is the distribution of the values the statistic would take over all possible samples of size $n$. It is not a distribution of people at all. Each point in it is the mean of a whole sample.

These adults are treated as a complete population to draw samples from, as in Confidence Intervals and Coverage. That is a teaching device. NHANES is itself a sample of the US population, designed so that national estimates use sampling weights, and nothing here estimates national blood pressure. Its value is that the population is real, not a textbook curve, and its parameters are known exactly. So every claim about the sampling distribution can be checked against the truth.

The three distributions on one axis. Top: the population of 5,247 adults, skewed to the right, with standard deviation 20.1 mm Hg. Middle: one random sample of 25 from it, a ragged copy of the population with two adults above 195 mm Hg; its mean is 135.4. Bottom: the means of 10,000 such samples. Notice that the bottom histogram is centered on the population mean (gray line), five times less spread out than the adults, and nearly symmetric although the population is not.
The three distributions on one axis. Top: the population of 5,247 adults, skewed to the right, with standard deviation 20.1 mm Hg. Middle: one random sample of 25 from it, a ragged copy of the population with two adults above 195 mm Hg; its mean is 135.4. Bottom: the means of 10,000 such samples. Notice that the bottom histogram is centered on the population mean (gray line), five times less spread out than the adults, and nearly symmetric although the population is not.

Check your understanding. Suppose the department samples 64 adults instead of 25. Which of the three distributions changes, and how?

The population does not change. The sample becomes a less ragged copy of it, still skewed and still spread over about 20 mm Hg. Only the sampling distribution changes predictably: it stays centered on 126.36 but narrows, from a standard deviation of about 4.0 mm Hg to about 2.5, as the simulation below confirms.

Notation. Before a sample is drawn, its observations are random variables $X_1, \ldots, X_n$. Their mean

$$\bar{X} = \frac{1}{n}\sum_{i=1}^{n} X_i$$

is a random variable too, written with a capital letter. The value it takes for one particular sample is $\bar{x}$. The mean and standard deviation of the sampling distribution of $\bar{X}$ are written $\mu_{\bar{X}}$ and $\sigma_{\bar{X}}$.

How the sample is drawn matters. In a simple random sample, every subset of $n$ people is equally likely to be the sample. Drawn one person at a time, that means each draw is equally likely to be any of the people not yet drawn. This is sampling without replacement. If instead each draw is equally likely to be anyone in the population, including people already drawn, the sampling is with replacement. Then the draws are independent, each with the population distribution. The two schemes have the same mean, and almost the same standard error when the sample is a small fraction of the population. To see the difference exactly, we need a population small enough to list every sample.

Every sample of a small population

For a small enough population, the sampling distribution can be written down completely. Take five of the NHANES adults as a population in their own right, with systolic pressures

$$134.67, \quad 114.00, \quad 136.00, \quad 89.33, \quad 112.67 \text{ mm Hg}.$$

Their mean is $\mu = 117.33$ and their variance, with divisor 5, is $\sigma^2 = 293.16$, so $\sigma = 17.12$ mm Hg. Drawing two of them without replacement, there are $\binom{5}{2} = 10$ possible samples, each with probability $1/10$:

Sample Values Mean $\bar{x}$
adults 1, 2 134.67, 114.00 124.33
adults 1, 3 134.67, 136.00 135.33
adults 1, 4 134.67, 89.33 112.00
adults 1, 5 134.67, 112.67 123.67
adults 2, 3 114.00, 136.00 125.00
adults 2, 4 114.00, 89.33 101.67
adults 2, 5 114.00, 112.67 113.33
adults 3, 4 136.00, 89.33 112.67
adults 3, 5 136.00, 112.67 124.33
adults 4, 5 89.33, 112.67 101.00

These ten means, each with probability $1/10$, are the sampling distribution of $\bar{X}$ for this population and this design. Their average is 117.33, exactly the population mean. Their variance is 109.93.

Check your understanding. Drawing with replacement adds samples in which one adult is drawn twice, such as adult 4 twice, with mean 89.33. That is more extreme than any mean in the table. Will the variance of the sample means be larger or smaller than 109.93?

Larger. With replacement there are $5^2 = 25$ equally likely ordered samples. Their 25 means again average 117.33, but their variance is 146.58, which is exactly $\sigma^2/2$. Without replacement the variance, 109.93, is not $\sigma^2/2$ but three quarters of it. The next three sections prove that none of this is a coincidence, and find where the factor of three quarters comes from.

The center: the sample mean is unbiased

Here is the idea in one sentence: each draw is, on average, the population mean, so the average of the draws is too.

Proposition. If each $X_i$ has the population distribution, so that $E[X_i] = \mu$ for every $i$, then $E[\bar{X}] = \mu$.

Proof. The expected value of a sum is the sum of the expected values, and a constant factor comes outside (The Binomial Distribution proves both rules). So

$$E[\bar{X}] = \frac{1}{n}\sum_{i=1}^{n} E[X_i] = \frac{1}{n}\cdot n\mu = \mu. \qquad \square$$

The proof does not use independence. So it covers sampling without replacement as well, provided each draw has the population distribution. It does, by symmetry. Every ordering of the population is equally likely to be the order of drawing, and each person occupies position $i$ in the same number of orderings. So the $i$th person drawn is equally likely to be any of the $N$ people: $P(X_i = x_k) = 1/N$ for each person $k$, and $E[X_i] = \frac{1}{N}\sum_k x_k = \mu$. The five adults confirm it for both designs: the means average 117.33 with and without replacement.

A statistic whose expected value equals the parameter it estimates is unbiased. The sample mean is unbiased for the population mean under random sampling of either kind. Averaged over all possible samples, it is right.

A common misreading takes "unbiased" to mean "close". It says nothing about any one sample, whose mean may be well off: sample 3 of the opening table missed by more than 5 mm Hg. It also says nothing about samples that are not random. Confidence Intervals and Coverage shows what happens when samples come only from adults aged 60 and over: every mean is centered on the wrong value, and no formula in this article can detect it.

But the department has one sample, not the average of all of them. It needs to know how far a single mean typically falls from $\mu$.

The spread: the standard error is sigma over root n

Here is the idea. A sum of $n$ independent values has $n$ times the variance of one value. Dividing the sum by $n$ divides the variance by $n^2$. So the variance of the mean is $\sigma^2/n$, and its standard deviation falls only with $\sqrt{n}$.

Proposition. If $X_1, \ldots, X_n$ are independent, each with variance $\sigma^2$, then

$$\operatorname{Var}(\bar{X}) = \frac{\sigma^2}{n}, \qquad \sigma_{\bar{X}} = \frac{\sigma}{\sqrt{n}}.$$

Proof. For independent variables the variance of a sum is the sum of the variances. (The Binomial Distribution proves this from the vanishing of the covariance.) Multiplying a variable by a constant $c$ multiplies its variance by $c^2$. So

$$\operatorname{Var}(\bar{X}) = \frac{1}{n^2}\operatorname{Var}\Big(\sum_{i=1}^n X_i\Big) = \frac{1}{n^2}\sum_{i=1}^n \operatorname{Var}(X_i) = \frac{1}{n^2}\cdot n\sigma^2 = \frac{\sigma^2}{n},$$

and the standard deviation is the square root. $\square$

Grinstead and Snell (1997, Theorem 6.9) state it for an independent trials process, and it is the most used formula of the course. The standard deviation of the sampling distribution is called the standard error of the mean. The name keeps the word "deviation" for individuals and "error" for estimates.

Unlike the center, the spread needs independence. The proof drops the covariances between the $X_i$, which are zero only for independent draws. With replacement the draws are independent, which is why the 25 ordered samples of the five adults had variance exactly $\sigma^2/2$.

The formula says three things.

  • The standard error is proportional to $\sigma$. A more variable population gives more variable means.
  • It does not depend on the size of the population, only on the size of the sample.
  • It shrinks with the square root of $n$, not with $n$. A sample four times as large halves the standard error, and one a hundred times as large cuts it by a factor of ten. That square-root law is why precision is expensive.

Check your understanding. For the department's samples of 25, the standard error is about 4.0 mm Hg. How many adults would it take to bring it down to 1.0?

About 400. Cutting the standard error by a factor of four needs a sample $4^2 = 16$ times as large, and $16 \times 25 = 400$.

The simulation checks the formula on the full NHANES population. For each sample size, 10,000 samples were drawn with replacement for each of 30 random seeds. The table gives the median over the seeds of the average and standard deviation of the 10,000 means, with the interquartile range (IQR) of the 30 results.

$n$ Average of the means Standard deviation of the means (IQR over seeds) $\sigma/\sqrt{n}$
1 126.35 20.141 (20.102 to 20.171) 20.125
4 126.38 10.056 (10.016 to 10.096) 10.062
16 126.36 5.021 (5.005 to 5.037) 5.031
25 126.35 4.016 (3.990 to 4.047) 4.025
64 126.37 2.521 (2.504 to 2.531) 2.516

Every row is centered on 126.36, and every spread matches $\sigma/\sqrt{n}$ to within the noise of the simulation.

Sampling distributions of the mean for samples of 4, 16 and 64 adults, 10,000 samples each, all centered on the population mean of 126.4 mm Hg (gray line). Read down the panels: each quadrupling of the sample size halves the spread, with standard errors 10.1, 5.0 and 2.5 mm Hg. Notice also the shape: the distribution for n = 4 still shows the population's right skew, and for n = 64 it is close to symmetric.
Sampling distributions of the mean for samples of 4, 16 and 64 adults, 10,000 samples each, all centered on the population mean of 126.4 mm Hg (gray line). Read down the panels: each quadrupling of the sample size halves the spread, with standard errors 10.1, 5.0 and 2.5 mm Hg. Notice also the shape: the distribution for n = 4 still shows the population's right skew, and for n = 64 it is close to symmetric.

Sampling without replacement: the finite population correction

Real samples are drawn without replacement, so that nobody is measured twice. The draws are then dependent, and the proof above no longer applies. Here is the idea in one sentence: if the first person drawn has a high pressure, fewer high pressures remain for the second draw, so high and low draws tend to offset each other. The dependence is negative, and it makes the sample mean less variable than $\sigma/\sqrt{n}$, as the five adults showed.

Proposition. For a simple random sample of $n$ drawn without replacement from a population of $N$ values with mean $\mu$ and variance $\sigma^2 = \frac{1}{N}\sum_k (x_k - \mu)^2$,

$$\operatorname{Var}(\bar{X}) = \frac{\sigma^2}{n}\cdot\frac{N - n}{N - 1}.$$

The plan of the proof: keep the covariances that the independent case dropped, show that every pair of draws has the same covariance, and compute it from the fact that deviations from the mean sum to zero.

Proof. The variance of a sum, without assuming independence, is the sum of the variances plus the covariances of all ordered pairs:

$$\operatorname{Var}\Big(\sum_{i=1}^n X_i\Big) = \sum_{i=1}^n \operatorname{Var}(X_i) + \sum_{i \ne j} \operatorname{Cov}(X_i, X_j).$$

Each $X_i$ has the population distribution, as shown in the previous section, so $\operatorname{Var}(X_i) = \sigma^2$. For the covariance, the same symmetry argument applied to two positions $i \ne j$ shows that the pair $(X_i, X_j)$ is equally likely to be any of the $N(N-1)$ ordered pairs of two different people. So every pair of draws has the same covariance,

$$c = \operatorname{Cov}(X_i, X_j) = \frac{1}{N(N-1)}\sum_{k \ne l}(x_k - \mu)(x_l - \mu).$$

The sum over pairs of different people is the sum over all pairs minus the pairs with $k = l$:

$$\sum_{k \ne l}(x_k - \mu)(x_l - \mu) = \Big(\sum_k (x_k - \mu)\Big)^2 - \sum_k (x_k - \mu)^2 = 0 - N\sigma^2,$$

because deviations from the mean sum to zero. Hence $c = -\sigma^2/(N - 1)$. There are $n(n-1)$ ordered pairs among the $n$ draws, so

$$\operatorname{Var}\Big(\sum_{i=1}^n X_i\Big) = n\sigma^2 - n(n-1)\frac{\sigma^2}{N-1} = n\sigma^2\Big(1 - \frac{n-1}{N-1}\Big) = n\sigma^2\,\frac{N - n}{N - 1},$$

and dividing by $n^2$ gives the result. $\square$

The square root of the factor, $\sqrt{(N - n)/(N - 1)}$, is the finite population correction.

Back to the five adults. The covariance of two draws is $c = -293.16/4 = -73.29$. That is exactly the average of $(x_k - \mu)(x_l - \mu)$ over the 20 ordered pairs of different adults. The variance of a mean of two is $146.58 \times 3/4 = 109.93$, as enumerated: the three quarters is $(N - n)/(N - 1) = 3/4$.

Sampling texts often use the population variance with divisor $N - 1$, $S^2 = N\sigma^2/(N-1)$, and write the result as $\operatorname{Var}(\bar{X}) = (1 - n/N)\,S^2/n$ (Cochran, 1977, §2.5); substituting $S^2$ shows that the two forms agree. The same factor multiplies the binomial variance when a count is taken without replacement, which gives the hypergeometric distribution of The Normal Approximation to the Binomial.

How much does the correction matter? It depends on the sampling fraction $n/N$. The script drew 2,000 samples without replacement from the 5,247 adults, for each of 30 seeds and each sample size:

$n$ $n/N$ Standard deviation of the means (IQR over seeds) $\sigma/\sqrt{n}$ Corrected
25 0.005 4.045 (4.001 to 4.068) 4.025 4.016
500 0.095 0.852 (0.844 to 0.864) 0.900 0.856
1,000 0.191 0.573 (0.567 to 0.577) 0.636 0.573
2,624 0.500 0.277 (0.273 to 0.280) 0.393 0.278
4,000 0.762 0.155 (0.154 to 0.157) 0.318 0.155
5,000 0.953 0.062 (0.061 to 0.062) 0.285 0.062
5,247 1 0 0.278 0
The standard error of a mean of n of the 5,247 adults against the sampling fraction n/N. Blue: sigma over root n, correct for sampling with replacement. Orange: with the finite population correction, correct without replacement. Notice that the simulated dots follow the orange curve, which falls to zero when the sample is the whole population, and that for small fractions the two curves meet.
The standard error of a mean of n of the 5,247 adults against the sampling fraction n/N. Blue: sigma over root n, correct for sampling with replacement. Orange: with the finite population correction, correct without replacement. Notice that the simulated dots follow the orange curve, which falls to zero when the sample is the whole population, and that for small fractions the two curves meet.

Check your understanding. One department samples 25 adults from a town of 5,247 adults. Another samples 25 from a whole country. If the two populations have the same $\sigma$, whose sample mean is more precise?

Neither, for practical purposes. For 25 out of 5,247 the correction changes the standard error by 0.2%, less than the simulation's own noise, and for a country it is smaller still. Until the sample is a large share of the population, only the sample size matters. At half the population, by contrast, the correction shrinks the standard error by 29%, and a sample of everyone has no sampling error at all, which the uncorrected formula misses.

The usual practice follows. The correction is ignored when the sampling fraction is small, as in nearly every medical study, where the population is a whole country or an open-ended stream of future patients. It is applied when a large share of a finite population is sampled, such as the records of one hospital or the beds of one ward. When the population is a process rather than a list, there is no $N$, and the independent-draws formula applies.

The standard error and the standard deviation

The two are easy to confuse: both are standard deviations, and both are reported next to a mean. They answer different questions.

  • The standard deviation $\sigma$ (estimated by $s$) describes how much individuals vary. It does not shrink as the sample grows; a larger sample only estimates it more precisely.
  • The standard error $\sigma/\sqrt{n}$ describes how much the sample mean varies from sample to sample, that is, how precisely the mean is estimated. It shrinks toward zero as the sample grows.

In practice $\sigma$ is unknown. The standard error is then estimated by $s/\sqrt{n}$, the estimated standard error, which is often also just called the standard error.

Check your understanding. Sample 1 of the opening table has mean 122.4, standard deviation 17.1 and estimated standard error $17.1/\sqrt{25} = 3.4$ mm Hg. About how many of its 25 adults lie within 17.1 mm Hg of the mean, and how many within 3.4?

Twenty of the 25 lie within one standard deviation of the mean, and only five within one standard error. So a report of "122.4 ± 3.4 mm Hg", read as a description of the adults, would make them look far more alike than they are.

Altman and Bland (2005) make the same distinction for medical papers. To say how widely scattered measurements are, use the standard deviation. The standard error measures the precision of the mean and is used mainly to compute confidence intervals and $P$ values. They also point out that "mean ± value" without saying which value is ambiguous, and cite a review of 88 articles published in 2002 in which 12 did not say. The safe practice is to report the standard deviation to describe data and a confidence interval to describe the precision of a mean, and to label both.

The estimated standard error has a sampling distribution too. Over the script's 10,000 samples of 25 (median of 30 seeds), the true standard error is 4.025 mm Hg. The estimates had:

  • median 3.893;
  • quartiles 3.414 and 4.428;
  • 5th and 95th percentiles 2.811 and 5.292.

So one sample in ten misjudges the precision of its own mean by 30% or more, in one direction or the other. The estimates are low more often than high, for two reasons. First, $s^2$ is unbiased for $\sigma^2$, but its square root is not unbiased for $\sigma$. Second, in a right-skewed population most samples of 25 miss the long tail that contributes much of $\sigma$. Replacing $\sigma$ by $s$ in a standardized mean is what makes Student's $t$ distribution necessary, the subject of Confidence Intervals for a Mean: z and t.

The shape of the sampling distribution

So far we have the center and the spread, and they hold for every population. The shape does not, though it follows a pattern.

If the population is normal, the sample mean is exactly normal for every $n$. A sum of independent normal variables is normal (Grinstead and Snell, 1997, §10.3, prove it with moment generating functions), and so is a sum divided by a constant. In the notation of the course's formula sheet, where the second argument is the standard deviation, $\bar{X} \sim N(\mu, \sigma/\sqrt{n})$.

If the population is not normal, the sampling distribution inherits some of its shape and loses it as $n$ grows. The figures above show it: the means of 4 adults keep a visible right tail, the means of 64 almost none. The simulation measured the skewness of the means, which falls in proportion to $1/\sqrt{n}$:

$n$ Skewness of the means (IQR over seeds) Population skewness $/\sqrt{n}$
1 1.034 (1.011 to 1.063) 1.038
4 0.523 (0.502 to 0.542) 0.519
16 0.251 (0.239 to 0.270) 0.259
25 0.207 (0.189 to 0.217) 0.208
64 0.137 (0.119 to 0.149) 0.130

Check your understanding. Means of 4 adults have skewness 0.52. What sample size should halve it?

Sixteen. Skewness falls with $\sqrt{n}$, so halving it takes four times the sample. The table agrees: 0.251 at $n = 16$.

The skewness of a mean of $n$ independent observations is the population skewness divided by $\sqrt{n}$. The Central Limit Theorem proves it, and proves that the whole distribution, not just its skewness, approaches the normal. For blood pressure the approach is quick. For strongly skewed measurements it can be slow enough to matter in studies of ordinary size, as the same article shows for C-reactive protein.

Putting the three results together, for independent observations and a large enough $n$,

$$\bar{X} \approx N\!\left(\mu,\ \frac{\sigma}{\sqrt{n}}\right), \qquad Z = \frac{\bar{X} - \mu}{\sigma/\sqrt{n}} \approx N(0, 1).$$

This standardized mean is the starting point of the confidence interval in Confidence Intervals and Coverage.

Other statistics have sampling distributions too

Every statistic computed from a random sample has a sampling distribution, and the ideas above apply to each. Three come up in the course.

  • The sample proportion. A proportion is the mean of observations coded 0 and 1, so it is a special case of this article, with $\mu = p$ and $\sigma^2 = p(1 - p)$. The Normal Approximation to the Binomial works it out, including the finite population correction.
  • The sample variance. For a normal population, its sampling distribution is a scaled chi-squared distribution, derived in Confidence Intervals for a Mean: z and t.
  • The sample median, treated next.

The sample median

Check your understanding. For samples of 25 of these adults, which varies less from sample to sample: the mean or the median?

The mean. Over the script's samples of 25, the medians had a standard deviation of 4.454 mm Hg, against 4.030 for the means of the same samples. The ratio is 1.108 (IQR 1.103 to 1.113 over the 30 seeds). The medians averaged 123.30 mm Hg, close to the population median of 123.33. So the median estimates a different parameter, the population median, and estimates it less precisely.

For a normal population the gap is wider. The same simulation, with samples of 25 from a standard normal population, gave a ratio of 1.242, approaching $\sqrt{\pi/2} = 1.2533$ for large samples. Where does $\sqrt{\pi/2}$ come from?

Here is the plan. The median of a sample is below a point exactly when at least half the observations are. A count of observations below a point is binomial, so the normal approximation to the binomial gives the distribution of the median.

The formal version. Take a continuous population with median $m$ and density $f(m) > 0$ there, and an odd sample size $n$.

  1. The sample median is at most $m + d$ exactly when at least half the observations are.
  2. Each observation is at most $m + d$ with probability close to $\tfrac12 + f(m)\,d$ when $d$ is small.
  3. So the proportion of such observations has mean close to $\tfrac12 + f(m)\,d$ and standard deviation close to $1/(2\sqrt{n})$.
  4. By the normal approximation to the binomial, the probability that this proportion is at least one half is close to $\Phi\big(2f(m)\,d\sqrt{n}\big)$.

That is the distribution function of a normal variable centered on $m$ with standard deviation $1/(2f(m)\sqrt{n})$. A normal population has $f(m) = 1/(\sigma\sqrt{2\pi})$, so the median's standard error is $\sigma\sqrt{\pi/2}/\sqrt{n}$. Its ratio to $\sigma/\sqrt{n}$ is $\sqrt{\pi/2}$.

That ratio is one reason the mean is preferred when the population is close to normal. Its square, $\pi/2$, reappears later in the course. The sign test, a test about the median, pays that price in sample size relative to the $t$-test for normal data: its efficiency is $2/\pi = 0.637$.

What the formulas assume

Every result in this article assumed a random sample. The standard error formula also assumed independent draws or, without replacement, the correction. Two kinds of departure are common.

  • Clustered observations. Patients within hospitals, children within schools and repeated measurements on one person are positively correlated. Then the standard error of the mean is larger than $\sigma/\sqrt{n}$, by a factor that The Binomial Distribution derives as the design effect and measures on NHANES's repeated blood pressure readings. The Central Limit Theorem shows a case where correlation keeps the standard error from shrinking to zero however large the sample.
  • Samples that are not random. No formula repairs them. The mean of a sample from the wrong population is unbiased for the wrong mean.

Corrections to the course notes

The unit's notes give the mean and standard error of the sampling distribution, $\mu_{\bar{X}} = \mu$ and $\sigma_{\bar{X}} = \sigma/\sqrt{n}$. They distinguish the population standard deviation, the sample standard deviation and the standard error, all as here. Three statements can be made more precise.

  • What the standard error measures. The notes describe it as how much different sample means are expected to vary from each other. It measures the typical distance of a sample mean from the population mean. Two independent sample means differ from each other by a quantity with the larger standard error $\sqrt{2}\,\sigma/\sqrt{n}$. That is why a second study's mean falls inside the first study's 95% interval only about 83% of the time, as Confidence Intervals and Coverage shows.
  • When $\sigma/\sqrt{n}$ holds. The formula holds for independent observations. For sampling without replacement it is multiplied by the finite population correction derived above, which matters when the sample is a large share of the population.
  • The estimated standard error. $s/\sqrt{n}$ is an estimate of the standard error rather than the standard error itself. It has a sampling distribution of its own, wide enough in samples of 25 that one sample in ten misjudges its precision by 30% or more.

Summary

Quantity Formula
Sample mean $\bar{X} = \frac{1}{n}\sum_{i=1}^n X_i$; observed value $\bar{x}$
Center $\mu_{\bar{X}} = E[\bar{X}] = \mu$ (any random sample)
Standard error, independent draws $\sigma_{\bar{X}} = \sigma/\sqrt{n}$
Without replacement from $N$ $\sigma_{\bar{X}} = \dfrac{\sigma}{\sqrt{n}}\sqrt{\dfrac{N - n}{N - 1}}$
Estimated standard error $s/\sqrt{n}$
Normal population $\bar{X} \sim N(\mu, \sigma/\sqrt{n})$ exactly
Skewness of $\bar{X}$ population skewness $/\sqrt{n}$
Large $n$, any population with finite $\sigma$ $\bar{X} \approx N(\mu, \sigma/\sqrt{n})$

What we have learned

A sample mean is a random quantity with a distribution of its own. It is centered on the population mean, with standard error $\sigma/\sqrt{n}$ for independent draws and a little less without replacement. The five adults showed both facts by listing every sample, and NHANES simulations confirmed them on a real, skewed population. The standard error describes the precision of a mean, not the spread of people, and in practice it is itself estimated, with error.

One question remains. Why were the means of 64 skewed blood pressures nearly symmetric, and how large must $n$ be before we can rely on that? The next article, The Central Limit Theorem, answers both.

In Python

The sampling distribution of a mean from a finite population can be simulated directly; rng.choice draws with replacement by default.

import numpy as np

rng = np.random.default_rng(0)
pop = np.array([134.67, 114.00, 136.00, 89.33, 112.67])   # any population
N, n = len(pop), 2
sigma2 = pop.var()                                        # divisor N

with_repl = rng.choice(pop, (100_000, n)).mean(axis=1)
without = np.array([rng.choice(pop, n, replace=False).mean() for _ in range(100_000)])
print(with_repl.var(), sigma2 / n)                        # about 146.6
print(without.var(), sigma2 / n * (N - n) / (N - 1))      # about 109.9

Every number in this article is computed by tools/experiments/applied-statistics/sampling-distribution-of-the-mean.py in the site's repository, which downloads the NHANES files from the CDC and checks their checksums; the figures are drawn by tools/figures/sampling-distribution-of-the-mean.py.

References

  • Altman, D. G., and Bland, J. M. (2005). Standard deviations and standard errors. BMJ, 331(7521), 903. record
  • Cochran, W. G. (1977). Sampling Techniques (3rd ed.). Wiley. record
  • Grinstead, C. M., and Snell, J. L. (1997). Introduction to Probability (2nd rev. ed.). American Mathematical Society. Theorem 6.9 (mean and variance of an average) and §10.3 (sums of independent normal variables). record · 10 pages
  • National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018: Demographic Variables and Sample Weights (DEMO_J) and Blood Pressure (BPX_J) data files. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 45 pages
  • Pagano, M., Gauvreau, K., and Mattie, H. (2022). Principles of Biostatistics (3rd ed.). CRC Press. The course text. record · 35 pages