The 5,182 adults whose blood pressure the National Health and Nutrition Examination Survey (NHANES) measured in 2017 and 2018 have a mean systolic pressure of 126.48 mm Hg. Treat them as a population, and the number is a parameter: fixed, and in this unusual case known. Draw 25 of them at random and the mean of the sample, a statistic, is 131.32 for one particular draw (seed 11). A different 25 would give a different mean. Statistical inference is the reasoning that gets from the one sample in hand to a statement about the parameter that admits how far off the sample might be: an interval of plausible values, or a judgement on whether a stated value is consistent with the data.
The theory is developed in the Applied Statistics course pages, with proofs: Sampling Distributions and The Central Limit Theorem for how statistics vary, Confidence Intervals and Hypothesis Testing for the two forms inference takes. This article is the working layer: what each Python function computes, the defaults that change its answer, and simulations against a population whose answers are known, so that each tool can be checked rather than trusted. Inference for Proportions and Means applies the same tools to the four standard comparisons.
The sampling distribution
A statistic's sampling distribution is the distribution of its values over all the samples the design could have produced. Its standard deviation is the standard error. For the mean of a simple random sample of $n$ drawn without replacement from $N$ values with standard deviation $\sigma$ (divisor $N$), the standard error is
$$\text{SE}(\bar{x}) = \frac{\sigma}{\sqrt{n}}\sqrt{\frac{N-n}{N-1}},$$
which The Sampling Distribution of the Mean derives. For the 5,182 adults, $\sigma = 20.064$ mm Hg, and for $n = 25$ the formula gives 4.0036. Simulation agrees: the standard deviation of 4,000 sample means was 3.9976 (median over 30 seeds; IQR 3.9758 to 4.0226), and their average 126.4765, the population mean to within simulation error. The mean is unbiased, and its typical error for samples of 25 is about 4 mm Hg.
In practice there is one sample and $\sigma$ is unknown, so the standard error is estimated from the sample standard deviation $s$:
$$\widehat{\text{SE}} = \frac{s}{\sqrt{n}}.$$
For the sample of 25, $s = 19.916$ and the estimate is 3.9832, close to the true 4.0036. scipy.stats.sem(x) computes it with divisor $n - 1$ by default; with NumPy's default x.std(), which divides by $n$, $s$ would be 19.514 instead (Summarizing Data with pandas tabulates which functions use which divisor). For statistics without a formula, the bootstrap estimates the standard error by resampling the sample (The Bootstrap Method); for this sample, scipy.stats.bootstrap gave 3.9007.
Confidence intervals
A 95% confidence interval is a range computed from the sample by a method that captures the parameter in 95% of samples. For a mean, the standard method is the $t$ interval
$$\bar{x} \pm t_{n-1,\,0.975}\,\frac{s}{\sqrt{n}},$$
whose derivation and the reason for $n - 1$ degrees of freedom are in Confidence Intervals for a Mean. The 95% refers to the method, not to any one interval: once computed, an interval either contains the parameter or does not, a distinction Confidence Intervals and Coverage makes concrete by drawing many intervals from a known population.
Python offers several routes to the same interval, and several near misses. For the sample of 25 (mean 131.32):
| Call | Interval | What it is |
|---|---|---|
stats.t.interval(0.95, n - 1, loc=xbar, scale=stats.sem(x)) |
123.099 to 139.541 | the $t$ interval, assembled by hand |
stats.ttest_1samp(x, 120).confidence_interval(0.95) |
123.099 to 139.541 | the same, from the test's result object |
DescrStatsW(x).tconfint_mean() (statsmodels) |
123.099 to 139.541 | the same |
stats.norm.interval(0.95, loc=xbar, scale=stats.sem(x)) |
123.513 to 139.127 | a $z$ interval: too narrow for small $n$ |
DescrStatsW(x).zconfint_mean() |
123.513 to 139.127 | the same $z$ interval |
stats.bootstrap((x,), np.mean, rng=rng) |
124.507 to 140.016 | BCa bootstrap, 9,999 resamples (the defaults) |
stats.bootstrap(..., method="percentile") |
123.933 to 139.161 | percentile bootstrap |
All of these contain the population mean, 126.48. Three details are worth noting. First, the result object returned by scipy's $t$-tests in current versions has a confidence_interval method, so the interval need not be assembled by hand, and ttest_1samp(x, 120, alternative="greater") returns the matching one-sided bound, here 124.505 to infinity. Second, statsmodels' DescrStatsW(x).std is the standard deviation with divisor $n$, 19.514, while its intervals correctly use $n - 1$; the attribute is not the sample standard deviation an analyst would report. Third, scipy.stats.bootstrap uses the bias-corrected and accelerated (BCa) method with 9,999 resamples unless told otherwise; the intervals it reports are not the percentile intervals most textbooks describe first.
When the nominal level is not the actual level
A 95% method that covers the parameter 95% of the time on paper may not do so for real data. The $t$ interval assumes the sample mean is close to normally distributed, which the central limit theorem delivers for large enough samples, and "large enough" depends on the population's shape (The Central Limit Theorem measures how fast skewness fades). Two populations show the range. Systolic pressure is mildly right-skewed; C-reactive protein (CRP), a marker of inflammation measured on 5,146 of the adults, has skewness 7.9 and a mean of 4.17 mg/L. For samples of 20:
| Population | Interval | Coverage of nominal 95%, median over 30 seeds (IQR) |
|---|---|---|
| Systolic pressure | $t$ | 0.940 (0.937 to 0.942), 2,000 samples per seed |
| CRP | $t$ | 0.793 (0.765 to 0.815), 200 samples per seed |
| CRP | bootstrap percentile (1,999 resamples) | 0.790 (0.755 to 0.809) |
| CRP | bootstrap BCa (1,999 resamples) | 0.825 (0.795 to 0.848) |
For blood pressure the $t$ interval is close to its promise. For CRP every method falls far short: a "95%" interval misses the mean about one time in five. BCa does a little better than the percentile interval, as it is designed to, but no method can reconstruct a long right tail that a sample of 20 usually does not contain. The Bootstrap Method finds the same pattern for an exponential population. The practical consequences are to look at the data's shape before trusting an interval for a mean, to consider whether the mean is the right parameter at all (the median, or the mean of the logarithms, is better behaved for CRP), and to check a method by simulation when the stakes justify it.
Tests and p-values
A hypothesis test asks whether the data are consistent with a specific value of the parameter, the null hypothesis $H_0$. The p-value is the probability, computed assuming $H_0$ is true, of a test statistic at least as extreme as the one observed. For the sample of 25 and $H_0: \mu = 120$, the one-sample $t$ statistic is
$$t = \frac{\bar{x} - 120}{s/\sqrt{n}} = \frac{131.32 - 120}{3.9832} = 2.8419$$
on 24 degrees of freedom, and stats.ttest_1samp(x, popmean=120) reports a two-sided p-value of 0.0090. The logic, the link between tests and intervals (the test rejects at level 0.05 exactly when 120 is outside the 95% $t$ interval, as it is here), and the meaning of one- and two-sided alternatives are worked through in Hypothesis Tests for a Mean.
The alternative argument. scipy's tests are two-sided by default; alternative="greater" or "less" gives a one-sided p-value, here 0.0045. statsmodels spells the same options "larger" and "smaller" in its proportion and weighted-statistics functions. The side must come from the question asked before the data were seen. Choosing it after seeing which way the data point halves the p-value and doubles the error rate, one of the practices Type I and Type II Errors measures.
What a p-value is not. The American Statistical Association's statement on p-values (Wasserstein and Lazar, 2016) lists six principles, among them that "P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone" and that "A p-value, or statistical significance, does not measure the size of an effect or the importance of a result." The second point is shown on these data in Inference for Proportions and Means, where all 5,182 adults give a highly significant difference between men and women of 1.8 mm Hg.
What p-values look like
When the null hypothesis is true and the test's assumptions hold, the p-value is uniformly distributed between 0 and 1, so it falls below 0.05 in 5% of samples; that is what the significance level means. Two groups of 30 drawn at random (with replacement) from the same 5,182 adults, compared by Welch's $t$-test, gave $p < 0.05$ in 4.88% of 2,000 comparisons (median over 30 seeds; IQR 4.70% to 5.19%). When the null hypothesis is false, p-values pile up near zero, and the share below 0.05 is the test's power. Ten adults aged 60 and over against ten aged 18 to 39, whose population means differ by 21.73 mm Hg, gave $p < 0.05$ in 76.1% of comparisons (IQR 75.3% to 76.7%).
Permutation tests
A permutation test builds the null distribution from the data themselves. If two groups come from the same population, the group labels are arbitrary, so every reassignment of the labels to the pooled values is equally likely; the p-value is the share of reassignments whose statistic is at least as extreme as the observed one. It needs no assumption about the shape of the population, only that the labels are exchangeable under the null hypothesis. Comparing Systems with Significance Tests works the paired version, the randomization test, by hand.
In scipy, stats.permutation_test takes the samples and any statistic. For 15 men and 15 women drawn from the adults (seed 5), with sample means 119.38 and 126.29 mm Hg:
def diff(u, v, axis):
return np.mean(u, axis=axis) - np.mean(v, axis=axis)
stats.permutation_test((men, women), diff, n_resamples=99_999, vectorized=True, rng=rng)
The permutation p-value is 0.3350, against 0.3318 from Welch's test. With equal group sizes the pooled and Welch statistics coincide ($t = -0.9926$) and only their degrees of freedom differ (28 and 21.89), giving 0.3294 for the pooled test. All three agree that a difference of this size is unremarkable for groups of 15. In the population, men's mean is higher by 1.80 mm Hg; this sample happens to point the other way, which is the kind of variation the p-value accounts for.
Many tests at once
A test at level 0.05 has a 5% chance of a false alarm. An analysis that runs many tests has a much larger chance of at least one. If all $m$ null hypotheses are true and the tests are independent, the family-wise error rate, the probability of one or more false rejections, is $1 - 0.95^m$, which is 0.5123 for $m = 14$.
To see this on real variables, take 100 adults from the 4,669 with complete records on 14 variables (blood pressures, body measurements, laboratory values, age and prescription count), split them into two groups of 50 at random, and test each variable for a difference with Welch's $t$-test. Since the groups are random, every null hypothesis is true. Over 1,000 random splits per seed and 30 seeds:
| Procedure | Share of splits with any rejection at 0.05, median (IQR) |
|---|---|
| No correction | 0.427 (0.418 to 0.435) |
| Bonferroni | 0.034 (0.029 to 0.036) |
| Holm | 0.034 (0.029 to 0.036) |
| Benjamini–Hochberg | 0.036 (0.032 to 0.040) |
Without correction, 43% of these random splits "find" a difference. That is less than the 51% of the independence formula because the variables are correlated (weight, BMI and waist circumference almost measure the same thing), so their false alarms tend to come together. The corrections bring the rate below 0.05.
The corrections themselves are covered in Multiple Comparisons and the Bonferroni Correction: Bonferroni tests each hypothesis at $\alpha/m$, and Holm's step-down version rejects at least as much while still controlling the family-wise error rate. Both make no assumption about how the tests depend on each other.
The false discovery rate
Controlling the chance of any false rejection is strict, and when many hypotheses are tested in the hope of finding several real effects, as in screening genes, features or variants of a product, it costs a lot of power. Benjamini and Hochberg (1995) proposed controlling a different quantity. If $R$ hypotheses are rejected and $V$ of them are true nulls (false discoveries), the false discovery rate is the expected proportion of false discoveries among the rejections,
$$\text{FDR} = E\!\left[\frac{V}{R}\right], \quad \text{with } V/R = 0 \text{ when } R = 0.$$
Their procedure orders the p-values $p_{(1)} \le \dots \le p_{(m)}$, finds the largest $k$ for which
$$p_{(k)} \le \frac{k}{m}\, q,$$
and rejects the hypotheses with the $k$ smallest p-values. They proved that for independent test statistics it controls the FDR at level $q$ whatever the configuration of true and false null hypotheses (their Theorem 1). Two properties they note explain the table above. When all null hypotheses are true, any rejection is a false one and $V/R = 1$, so the FDR equals the family-wise error rate; that is why Benjamini–Hochberg held the any-rejection rate near 0.036 on random labels. When some null hypotheses are false, the FDR is at most the family-wise error rate, so a procedure that controls only the FDR can be less strict and reject more. The tests here are correlated, not independent, so the theorem does not strictly apply; the simulation shows the rate still controlled in this case.
The same variables, with real differences
Comparing 45 men and 55 women from the same 100-adult design (seed 7), on the same 14 variables, gives a mix of real and absent differences:
| Variable | p | Bonferroni | Holm | Benjamini–Hochberg |
|---|---|---|---|---|
| Height | < 0.00001 | < 0.0001 | < 0.0001 | < 0.0001 |
| HDL cholesterol | 0.00017 | 0.0023 | 0.0022 | 0.0012 |
| Blood cadmium | 0.0319 | 0.447 | 0.383 | 0.149 |
| Weight | 0.0579 | 0.810 | 0.636 | 0.181 |
| Age | 0.0647 | 0.906 | 0.647 | 0.181 |
| Diastolic pressure | 0.0932 | 1 | 0.839 | 0.218 |
The table shows adjusted p-values, which can be compared directly with 0.05: the smallest significance level at which each procedure would reject that hypothesis. The other eight variables have p-values from 0.21 to 0.96. Uncorrected testing rejects three hypotheses; all three procedures reject two, height and HDL cholesterol. On all 4,669 adults, however, men and women differ at $p < 0.001$ on 11 of the 14 variables. Most of the "non-significant" results in the sample of 100 are not true nulls but differences too small for that sample to detect. A correction for multiple testing protects against false discoveries; it does nothing about missed ones, and a non-significant result is never evidence that a difference is zero.
Defaults in the libraries
- statsmodels'
multipletests(pvals, alpha=0.05, method="hs")uses Holm–Šidák by default, not Bonferroni or Holm: a step-down procedure like Holm's that replaces $\alpha/m$ by Šidák's $1 - (1-\alpha)^{1/m}$, slightly less conservative, and derived for independent tests. Passmethod="holm","bonferroni"or"fdr_bh"explicitly. It returns the reject decisions, the adjusted p-values and two corrected significance levels. - scipy's
stats.false_discovery_control(ps)returns Benjamini–Hochberg adjusted p-values by default (method="bh"); on the 14 p-values above they equal statsmodels'fdr_bh. - The course wiki's Comparing Systems with Significance Tests discusses multiple comparisons when many retrieval systems are compared on the same topics.
Corrections to the course notes
The unit's notes give an accurate outline of the framework and are stated more precisely here in three places. They say the probability of a Type I error equals the significance level; it does for tests with continuous statistics whose assumptions hold, but for discrete data the achievable rates jump, and the exact binomial test at a nominal 5% can have a true rate of 3.6% (Inference for Proportions and Means computes it). They cite $n \ge 30$ as the common rule of thumb for the central limit theorem and add that skewed populations need more; the CRP intervals above show how much more a strongly skewed population can need. And they say confidence intervals and tests always agree; they do when both are built from the same statistic and standard error, as the $t$ test and $t$ interval are, but not in general: the default interval for a proportion in statsmodels and the score test do not share a standard error and can disagree near the boundary.
In Python
import numpy as np
from scipy import stats
from statsmodels.stats.weightstats import DescrStatsW
from statsmodels.stats.multitest import multipletests
rng = np.random.default_rng(11)
x = pop_sbp[rng.choice(len(pop_sbp), 25, replace=False)]
stats.sem(x) # s / sqrt(n), divisor n - 1
res = stats.ttest_1samp(x, popmean=120) # two-sided unless alternative= is set
res.statistic, res.pvalue, res.df, res.confidence_interval(0.95)
DescrStatsW(x).tconfint_mean() # same interval; DescrStatsW(x).std uses divisor n
stats.bootstrap((x,), np.mean, method="BCa", n_resamples=9999, rng=rng).confidence_interval
stats.permutation_test((a, b), lambda u, v, axis: u.mean(axis) - v.mean(axis),
vectorized=True, n_resamples=99_999, rng=rng).pvalue
reject, p_adj, _, _ = multipletests(pvals, alpha=0.05, method="holm") # name the method
stats.false_discovery_control(pvals) # Benjamini-Hochberg adjusted p-values
The numbers in this article are computed by tools/experiments/data-science/statistical-inference.py in the site's repository, which needs statsmodels in addition to NumPy, pandas and scipy (the results shown used NumPy 2.5.4, pandas 3.1.1, scipy 1.18.1 and statsmodels 0.15.0; the rng= keyword of scipy's resampling functions dates from scipy 1.15, and earlier versions call it random_state), from the NHANES tables built by psdata.py and exploredata.py; the figure is drawn by tools/figures/statistical-inference.py.
References
- Benjamini, Y., and Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B, 57(1), 289–300. https://doi.org/10.1111/j.2517-6161.1995.tb02031.x record · 2 pages
- National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018 data files and documentation. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 45 pages
- SciPy developers (2026).
scipy.stats.ttest_1samp,bootstrap,permutation_test,false_discovery_control. SciPy reference guide. https://docs.scipy.org/doc/scipy/reference/stats.html record - statsmodels developers (2026).
statsmodels.stats.weightstats.DescrStatsW;statsmodels.stats.multitest.multipletests. https://www.statsmodels.org/stable/ record - Wasserstein, R. L., and Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129–133. https://doi.org/10.1080/00031305.2016.1154108 (the six principles quoted are as listed in the American Statistical Association's release of the statement, March 7, 2016) record · 6 pages