Most questions that end in a test are one of four kinds: is a proportion equal to a stated value, do two groups have the same proportion, is a mean equal to a stated value, do two groups have the same mean. Each has a standard test and interval, each is available in scipy and statsmodels, and the libraries do not agree on the defaults. The same 53 adults with hypertension out of 80 give a two-sided p-value of 0.0262, 0.0348 or 0.0427 depending on which function is called with its default arguments, and the same two samples of blood pressures give 0.0001 or 0.0034 depending on one keyword. None of these is a bug. Each function computes a well-defined procedure, and choosing among them is part of the analysis.
The procedures themselves are derived, with their assumptions and exact error rates, in the Applied Statistics course pages: Inference on Proportions (the normal approximation, intervals for a proportion, tests and sample size, two proportions), Confidence Intervals and Hypothesis Testing for a mean, and Comparison of Two Means (the pooled and Welch $t$-tests, paired samples, sample size). The general ideas, standard errors, p-values and multiple testing, are in Inferential Statistics in Python. This article maps each of the four questions to the functions that answer it and shows, on samples from the 5,182 adults whose blood pressure the National Health and Nutrition Examination Survey (NHANES) measured in 2017 and 2018, where their answers part ways. Hypertension follows the 2017 definition of the American College of Cardiology and American Heart Association, as in Data Collection and Sampling.
One proportion
NCHS estimates that 54.5% of US adults aged 40 to 59 had hypertension in 2017 to 2018 (Ostchega et al., 2020). Is the share the same among the NHANES participants of that age? A random sample of 80 of them (seed 3) contains 53 with hypertension, $\hat{p} = 0.6625$. The null hypothesis is $H_0: p = 0.545$.
Three tests are in common use. The score test standardizes by the standard error under the null hypothesis,
$$z = \frac{\hat{p} - p_0}{\sqrt{p_0(1 - p_0)/n}} = \frac{0.6625 - 0.545}{\sqrt{0.545 \times 0.455 / 80}} = 2.1105,$$
the Wald test by the standard error estimated from the sample, $\sqrt{\hat{p}(1-\hat{p})/n}$, and the exact binomial test uses the binomial distribution itself instead of a normal approximation. In Python:
| Call | Test | Statistic | p (two-sided) |
|---|---|---|---|
proportions_ztest(53, 80, value=0.545) (statsmodels default) |
Wald | 2.2226 | 0.0262 |
proportions_ztest(53, 80, value=0.545, prop_var=0.545) |
score | 2.1105 | 0.0348 |
chisquare([53, 27], [80 * 0.545, 80 * 0.455]) (scipy) |
score, squared | 4.4541 | 0.0348 |
binomtest(53, 80, p=0.545) (scipy) |
exact | 0.0427 |
statsmodels' one-sample proportions_ztest computes the variance from the sample proportion unless prop_var supplies another value; with prop_var=p0 it becomes the score test, which is also the chi-squared goodness-of-fit test with one degree of freedom. All three reject at 0.05 here, and the null hypothesis is in fact false for this population: the share among the 5,182 examined adults aged 40 to 59 is 0.5839. NHANES oversamples groups with higher prevalence, so its participants are not a miniature of the country, and the national 54.5% is a weighted figure (Data Collection and Sampling explains the weights). The test has found the difference between the sample design and the population, which is a real finding about these data even if not about health.
Which test holds its level
The differences between the tests matter most in small samples. Because the count is discrete, the exact probability of rejecting a true null hypothesis can be computed by adding the binomial probabilities of the counts each test rejects. For $n = 30$ and $p_0 = 0.2$ at the 5% level:
| Test | Counts rejected (out of 30) | True Type I error rate |
|---|---|---|
Wald (proportions_ztest default) |
0, 1, 2 and 12 or more | 0.0537 |
Score (prop_var=p0) |
0, 1 and 11 or more | 0.0361 |
Exact (binomtest) |
0, 1 and 11 or more | 0.0361 |
The Wald test rejects too often, partly because a sample with 1 or 2 events has a tiny estimated standard error and an enormous $z$: for one event in 30, statsmodels' default reports $z = -5.085$ and a p-value of $3.7 \times 10^{-7}$. The score and exact tests reject the same counts here and keep the rate below 5%, at the price of rejecting less often than the nominal level, the conservatism of exact tests that Tests and Sample Size for a Proportion examines. The general advice from that page and from Confidence Intervals for a Proportion carries over to code: avoid the Wald forms for small samples or proportions near 0 or 1.
Intervals
For the interval, statsmodels' proportion_confint defaults to method="normal", the Wald interval, while scipy's binomtest(...).proportion_ci() defaults to the exact (Clopper–Pearson) interval:
| Method | statsmodels method= |
scipy method= |
95% interval | Width |
|---|---|---|---|---|
| Wald | "normal" (default) |
0.5589 to 0.7661 | 0.2072 | |
| Wilson score | "wilson" |
"wilson" |
0.5536 to 0.7565 | 0.2030 |
| Agresti–Coull | "agresti_coull" |
0.5533 to 0.7568 | 0.2035 | |
| Jeffreys | "jeffreys" |
0.5546 to 0.7589 | 0.2043 | |
| Clopper–Pearson | "beta" |
"exact" (default) |
0.5481 to 0.7645 | 0.2164 |
| Wilson with continuity correction | "wilsoncc" |
0.5472 to 0.7621 |
With 80 observations and $\hat{p}$ in the middle of the range, the intervals differ by less than a percentage point at either end. The coverage of each method across sample sizes and true proportions, computed exactly, is the subject of Confidence Intervals for a Proportion, which recommends Wilson's interval for general use. Note also that statsmodels takes alpha=0.05 for a 95% interval, while scipy's proportion_ci takes confidence_level=0.95.
Two proportions
Among 100 men and 100 women sampled from the adults (seed 4), 63 men and 48 women have hypertension, a difference of 0.15; in the whole population the shares are 0.5856 and 0.5108. The functions:
| Call | Method | Statistic | p |
|---|---|---|---|
proportions_ztest([63, 48], [100, 100]) |
pooled $z$ (score) | 2.1343 | 0.0328 |
test_proportions_2indep(63, 100, 48, 100) |
default: Agresti–Caffo | 2.1362 | 0.0327 |
test_proportions_2indep(..., method="score") |
score | 2.1289 | 0.0333 |
chi2_contingency(table, correction=False) |
Pearson chi-squared | 4.5551 ($= z^2$) | 0.0328 |
chi2_contingency(table) (scipy default) |
with Yates's correction | 3.9680 | 0.0464 |
fisher_exact(table) |
exact conditional | odds ratio 1.8446 | 0.0461 |
Unlike the one-sample case, statsmodels' two-sample proportions_ztest uses the pooled proportion under the null hypothesis, so it equals the Pearson chi-squared test on the $2 \times 2$ table, whose statistic is $z^2$; Comparing Two Proportions proves the identity. scipy's chi2_contingency applies Yates's continuity correction to $2 \times 2$ tables unless correction=False, which moves the p-value from 0.0328 to 0.0464, close to Fisher's exact test. For the interval, confint_proportions_2indep defaults to Newcombe's method, 0.0124 to 0.2799 for the difference, against 0.0138 to 0.2862 for the Wald interval (method="wald"). Both exclude zero, consistent with the tests.
One mean
For a mean the standard procedure is the one-sample $t$-test and interval (Hypothesis Tests for a Mean). A sample of 40 adults (seed 6) has mean systolic pressure 125.08 mm Hg and standard deviation 17.60. Against $H_0: \mu = 120$, scipy's ttest_1samp(x, 120) gives $t = 1.8267$ on 39 degrees of freedom and $p = 0.0754$, with 95% interval 119.455 to 130.712 from its confidence_interval() method; statsmodels' DescrStatsW(x).ttest_mean(120) and tconfint_mean() return the same. Here the libraries agree, and the remaining traps are elsewhere.
Missing values. With one NaN appended to the sample, ttest_1samp returns a p-value of NaN under its default nan_policy="propagate"; nan_policy="omit" drops the missing value and returns 0.0754 again. pandas' mean() would have skipped it silently. The scipy behavior is the safer one, since it forces a decision, but code that passes pandas columns to scipy needs dropna() or nan_policy="omit" made explicit.
Return values. scipy's tests return a result object that unpacks into the statistic and p-value (t, p = ttest_1samp(...)) and also carries df and confidence_interval(). statsmodels' ttest_mean and its two-sample ttest_ind return a plain tuple of three values: statistic, p-value and degrees of freedom. Code that unpacks two values from a statsmodels test fails.
Two means
Twelve adults aged 60 and over and 36 aged 18 to 39 (seed 8) have mean systolic pressures of 130.78 and 113.69 mm Hg, with standard deviations 15.77 and 10.84. In the population the groups' standard deviations are 20.93 and 12.89 mm Hg: the older group is both smaller and more variable.
| Call | Test | $t$ | df | p | 95% interval for the difference |
|---|---|---|---|---|---|
stats.ttest_ind(a, b) (scipy default) |
pooled | 4.2001 | 46 | 0.000121 | 8.896 to 25.271 |
stats.ttest_ind(a, b, equal_var=False) |
Welch | 3.4887 | 14.63 | 0.003410 | 6.623 to 27.544 |
ttest_ind(a, b) (statsmodels default) |
pooled | 4.2001 | 46 | 0.0001 | |
ttest_ind(a, b, usevar="unequal") (statsmodels) |
Welch | 3.4887 | 14.63 | 0.0034 |
Both libraries pool the variances by default; Welch's test must be requested, with equal_var=False in scipy and usevar="unequal" in statsmodels (whose CompareMeans.tconfint_diff takes the same argument for the interval). R's t.test does the opposite and uses Welch's test unless told otherwise, as The Two-Sample t-Test: Pooled and Welch notes. Here both tests reject, but the pooled test is overconfident by a factor of almost 30 in its p-value, because it averages the two variances in proportion to the group sizes and so gives the variance of the smaller, more variable group of older adults too little weight.
How much that matters can be measured by simulation. With normal populations of equal means and the population standard deviations above, 20.9 in a group of 12 and 12.9 in a group of 36, the pooled test rejected the true null hypothesis in 12.0% of 4,000 replications (median over 30 seeds; IQR 11.7% to 12.4%), and Welch's test in 5.1% (IQR 5.0% to 5.4%). The pooled test is safe when the variances are equal or the groups are of equal size; when the smaller group has the larger variance, as here, it is not. The Two-Sample t-Test: Pooled and Welch maps the error rates across designs and shows that testing the variances first and then choosing is worse than using Welch's test throughout. The unit's notes describe Welch's test as scipy's default; in scipy it is the pooled test that is the default.
Paired data
When each person is measured twice, the two measurements are not independent samples, and the analysis has to know it. NHANES takes up to three systolic readings per person in one sitting, and the first is on average higher: across the 4,782 adults with a first and third reading, by 0.786 mm Hg. In a sample of 300 of them (seed 9) the mean difference is 0.613 mm Hg:
| Call | $t$ | p | 95% interval |
|---|---|---|---|
stats.ttest_rel(first, third) (paired) |
2.0865 | 0.0378 | 0.035 to 1.192 |
stats.ttest_ind(first, third) (pairing ignored) |
0.4332 | 0.6650 | $-2.167$ to 3.394 |
The paired test works on the 300 differences, whose standard deviation is 5.09 mm Hg; the unpaired test compares two columns whose standard deviations are about 17.3, because people differ from each other far more than one reading differs from the next (the two readings correlate at 0.957). Ignoring the pairing throws that information away, and an interval almost five times as wide cannot see a difference of less than 1 mm Hg. Both functions accept the same two arrays, so the software cannot tell which design produced them; The Paired Samples t-Test works the same contrast on Student's sleep data. The signed-rank test on the same differences gives 0.0556; by default it drops the 45 zero differences, pairs whose two readings were equal, a choice its zero_method argument controls (The Sign Test and the Wilcoxon Signed-Rank Test).
Statistical and practical significance
With all 5,182 adults, men's mean systolic pressure, 127.40 mm Hg, exceeds women's, 125.60, by 1.80 mm Hg. Welch's test gives $t = 3.239$ and $p = 0.0012$, and the 95% interval for the difference is 0.71 to 2.89 mm Hg. The difference is real in the sense that it is unlikely to be chance. It is also small: about a tenth of a standard deviation (standardized difference 0.090), much less than the variation between two readings taken minutes apart in the same person. A p-value measures how well the data separate the difference from zero, which depends on the sample size as much as on the difference; it does not measure whether the difference matters (Wasserstein and Lazar, 2016). The interval says both things at once, which is the main reason to report it alongside, or instead of, the p-value.
Choosing the function
| Question | Recommended in scipy | Recommended in statsmodels | Default to watch |
|---|---|---|---|
| One proportion, test | binomtest(x, n, p0) |
proportions_ztest(x, n, p0, prop_var=p0) |
statsmodels' Wald variance |
| One proportion, interval | binomtest(...).proportion_ci(method="wilson") |
proportion_confint(x, n, method="wilson") |
statsmodels' "normal" |
| Two proportions | chi2_contingency(t, correction=False), fisher_exact(t) |
test_proportions_2indep, confint_proportions_2indep |
scipy's Yates correction |
| One mean | ttest_1samp(x, mu0).confidence_interval() |
DescrStatsW(x).ttest_mean(mu0) |
nan_policy="propagate" |
| Two means | ttest_ind(a, b, equal_var=False) |
ttest_ind(a, b, usevar="unequal") |
pooled variances in both |
| Paired means | ttest_rel(a, b) |
DescrStatsW(a - b).ttest_mean(0) |
none: but the design must be known |
The recommendations follow the course pages linked above: Wilson's interval for a proportion, the exact or score test for small samples, and Welch's test whenever the variances may differ. Whatever is chosen, the function and its arguments belong in the report, since the same data can give different p-values under different defaults, and a reader cannot reproduce a number without knowing which one was used.
Corrections to the course notes
The unit's notes describe the four scenarios and the functions clearly and are stated more precisely here in four places. They describe equal_var=False as the default of scipy's ttest_ind; the default is equal_var=True, the pooled test, and Welch's test must be requested. They say scipy's $t$-tests do not return confidence intervals; current versions return them through the result's confidence_interval() method, as shown above. They present statsmodels' default prop_var=False as a minor variant of the standard test; for one sample it makes the test a Wald test, which holds its level less well than the score test in small samples. And they say every test and interval agree; that holds when both come from the same statistic, but statsmodels' default interval for a proportion (Wald) and the score test do not share a standard error and can disagree near the boundary. The notes' worked examples use datasets from their course text; the examples here use public NHANES data.
In Python
import numpy as np
from scipy import stats
from statsmodels.stats.proportion import (proportions_ztest, proportion_confint,
test_proportions_2indep, confint_proportions_2indep)
from statsmodels.stats.weightstats import DescrStatsW, CompareMeans
stats.binomtest(53, 80, p=0.545).pvalue # exact
proportions_ztest(53, 80, value=0.545, prop_var=0.545) # score; omit prop_var for Wald
proportion_confint(53, 80, alpha=0.05, method="wilson")
test_proportions_2indep(63, 100, 48, 100, method="score")
stats.chi2_contingency([[63, 37], [48, 52]], correction=False)
stats.ttest_1samp(x, 120, nan_policy="omit").confidence_interval()
res = stats.ttest_ind(old, young, equal_var=False) # Welch
res.statistic, res.pvalue, res.df, res.confidence_interval()
CompareMeans(DescrStatsW(old), DescrStatsW(young)).tconfint_diff(usevar="unequal")
stats.ttest_rel(first, third) # paired
The numbers in this article are computed by tools/experiments/data-science/inference-for-proportions-and-means.py in the site's repository, which needs statsmodels in addition to NumPy, pandas and scipy (the results shown used NumPy 2.5.4, pandas 3.1.1, scipy 1.18.1 and statsmodels 0.15.0), from the NHANES tables built by psdata.py and the Applied Statistics helper statdata.py; the figure is drawn by tools/figures/inference-for-proportions-and-means.py.
References
- National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018 data files and documentation (DEMO_J, BPX_J, BPQ_J). Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 45 pages
- Ostchega, Y., Fryar, C. D., Nwankwo, T., and Nguyen, D. T. (2020). Hypertension Prevalence Among Adults Aged 18 and Over: United States, 2017–2018. NCHS Data Brief No. 364. National Center for Health Statistics. https://www.cdc.gov/nchs/products/databriefs/db364.htm record · 6 pages
- SciPy developers (2026).
scipy.stats.binomtest,chi2_contingency,fisher_exact,ttest_1samp,ttest_ind,ttest_rel. SciPy reference guide. https://docs.scipy.org/doc/scipy/reference/stats.html record - statsmodels developers (2026).
statsmodels.stats.proportion(proportions_ztest,proportion_confint,test_proportions_2indep,confint_proportions_2indep) andstatsmodels.stats.weightstats(DescrStatsW,CompareMeans,ttest_ind). https://www.statsmodels.org/stable/ record - Wasserstein, R. L., and Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129–133. https://doi.org/10.1080/00031305.2016.1154108 (principles as listed in the American Statistical Association's release of the statement, March 7, 2016) record · 6 pages