Choose one adult at random from the 5,182 whose blood pressure the National Health and Nutrition Examination Survey (NHANES) measured in 2017 and 2018. The probability that this person has hypertension is the share of the 5,182 who do, 2,837 of them, or 0.5475. For a finite population and a uniformly random choice, every probability is a share of people, and every rule of probability becomes a statement about counts that a few lines of pandas can check. That makes a data table a good place to learn the rules. It also makes it a place where they are easy to misapply, because the counts include missing values, overlapping groups and conditional shares computed in the wrong direction.
Probability enters data science in two further ways. Data are modeled as outcomes of random processes, which is the subject of Probability Distributions in SciPy; and probabilities that are hard to compute are estimated by simulating the process many times, which is the subject of the second half of this article. The axioms and the rules derived from them are proved in the Applied Statistics pages The Rules of Probability and Conditional Probability and the Law of Total Probability, with the topic overview at Probability; here they are stated briefly and put to work.
Events as masks
A random process has outcomes that cannot be predicted individually; the set of all outcomes is the sample space, and an event is a set of outcomes. When the process is "pick one row of the table at random", the outcomes are the rows, an event is a set of rows, and in pandas a set of rows is a Boolean mask. Its mean is the share of True values, which is the probability of the event:
H = adults.htn == 1 # hypertension
O = adults.bmi >= 30 # obesity
H.mean(), O.mean()
(H & O).mean(), (H | O).mean(), (~H).mean() # intersection, union, complement
The operators &, | and ~ are intersection ("and"), union ("or") and complement ("not"). Python's and, or and not do not work element by element on Series and raise an error, which is the better outcome; a common silent mistake is the next one.
Missing values in masks. A comparison with a missing value is False in pandas: NaN >= 30 is False. Body mass index (BMI) is missing for 71 of the 5,182 adults, and adults.bmi >= 30 quietly classifies them as not obese, giving $P(O) = 0.4052$. Among the 5,111 adults whose BMI is known the share is 0.4109. The first number answers no sensible question: it is the probability of "known to be obese", which mixes obesity with whether the measurement was taken. Restricting to rows where every variable in the events is known, or using pandas' nullable types, in which a comparison with a missing value stays missing, keeps the events honest. The rest of this section uses the 5,111 adults with a known BMI.
The rules, checked in code
For these 5,111 adults, $P(H) = 0.5447$, $P(O) = 0.4109$ and $P(H \cap O) = 0.2653$; in counts, 2,784 have hypertension, 2,100 are obese, and 1,356 are both.
Complement rule. $P(H^c) = 1 - P(H) = 0.4553$. It is used constantly in the form "at least one" equals one minus "none", as in the birthday problem below.
Addition rule. For any two events $P(H \cup O) = P(H) + P(O) - P(H \cap O)$, since adding the two probabilities counts the overlap twice. Here $0.5447 + 0.4109 - 0.2653 = 0.6903$, and the direct share (H | O).mean() is also 0.6903 (3,528 adults). For disjoint events, which cannot happen together, the overlap is empty and the probabilities simply add.
Independence. Two events are independent when $P(H \cap O) = P(H)\,P(O)$, that is, when knowing that one occurred does not change the probability of the other. Here $P(H)\,P(O) = 0.2238$, well below the observed 0.2653, so hypertension and obesity are not independent among these adults. Two cautions apply. In a sample, the product rule will almost never hold exactly even for independent events, so a small discrepancy in data is not evidence against independence; deciding whether a discrepancy is larger than chance would produce is the job of a test such as the chi-squared test in Comparing Two Proportions. And independence is not the same as being disjoint: disjoint events with positive probabilities are as dependent as events can be, since one rules out the other (The Rules of Probability proves this).
Conditional probability
The conditional probability of $H$ given $O$ is the share of the $O$ rows that are also $H$ rows:
$$P(H \mid O) = \frac{P(H \cap O)}{P(O)} = \frac{0.2653}{0.4109} = 0.6457.$$
In code it is H[O].mean(), or for every combination at once a cross-tabulation normalized by row:
pd.crosstab(O, H, normalize="index")
| No hypertension | Hypertension | |
|---|---|---|
| Not obese | 0.5257 | 0.4743 |
| Obese | 0.3543 | 0.6457 |
Adults with obesity have hypertension 64.6% of the time, those without 47.4%. The ratio $P(H \mid O)/P(H) = 1.185$ measures how much knowing $O$ changes the probability of $H$; it equals 1 exactly when the events are independent. The two directions of conditioning answer different questions: normalize="index" gives $P(H \mid O)$, normalize="columns" gives $P(O \mid H)$, and confusing them is one of the commonest errors in reading tables, as Exploring Data with Groups and Pivot Tables shows with blood pressure categories. The ratio of two conditional risks such as 0.6457 and 0.4743 is the relative risk of Relative Risk and the Odds Ratio.
An association in a table can come from a third variable. Hypertension and obesity both vary with age, so one might suspect age explains their association. Conditioning on age group as well shows that it does not, at least not entirely: within each age group, adults with obesity are more likely to have hypertension (0.3573 against 0.1479 at ages 18 to 39, 0.6901 against 0.4948 at 40 to 59, 0.8478 against 0.7647 at 60 and over).
Total probability and Bayes' theorem
If events $A_1, \dots, A_k$ split the sample space into disjoint parts, such as three age groups, the law of total probability recovers $P(H)$ from the conditional probabilities within the parts:
$$P(H) = \sum_i P(H \mid A_i)\,P(A_i).$$
With age shares $P(A_i)$ of 0.3291, 0.3048 and 0.3661 and hypertension shares within them of 0.2277, 0.5822 and 0.7985, the sum is 0.5447, the overall share. Bayes' theorem reverses the direction of conditioning:
$$P(A_i \mid H) = \frac{P(H \mid A_i)\,P(A_i)}{P(H)}.$$
It gives the age distribution of the adults with hypertension as 0.1376, 0.3258 and 0.5366, exactly the shares obtained by counting them directly with adults[H].age_group.value_counts(normalize=True). In a single table, Bayes' theorem is a check on arithmetic, because both directions can be counted. It earns its place when the pieces come from different sources: a test's sensitivity measured in a study of patients, combined with a disease's prevalence from a survey, to find the probability that a person who tests positive has the disease. That calculation, and why its answer is so often lower than intuition expects, is worked in Predictive Values and Bayes' Theorem. The same reversal is the basis of the naive Bayes classifier.
Probability by simulation
Many probabilities are easier to estimate than to derive: simulate the random process many times and count how often the event happens. The estimate is a sample proportion, so its precision is known. If the true probability is $p$ and the process is simulated $m$ times independently, the estimate has standard error
$$\text{SE} = \sqrt{\frac{p(1-p)}{m}},$$
the Monte Carlo error. It falls only as $1/\sqrt{m}$: a hundred times more simulations buy one more decimal place.
Random numbers in NumPy
NumPy's recommended interface is a Generator object, created by np.random.default_rng(seed), which by default draws from the PCG64 bit generator (NumPy documentation, default_rng). Three properties matter in practice.
Seeds make results reproducible, and different interfaces give different streams. np.random.default_rng(0).random(3) returns 0.636962, 0.269787 and 0.040974 on every run and every machine. The older global interface, np.random.seed(0) followed by np.random.random(3), returns 0.548814, 0.715189 and 0.602763: the same seed, a different generator. Code that mixes the two, or relies on the global state that any library call may advance, is hard to reproduce. Passing a Generator explicitly to every function that draws random numbers avoids this; pandas' sample and the rvs method of scipy's distributions accept one as random_state.
Several independent streams come from one seed by spawning. Experiments repeated over 30 seeds, as on this site, use seeds 0 to 29; for parallel work, np.random.SeedSequence(2026).spawn(30) derives 30 child seeds from one, so that the whole experiment is reproducible from a single number and no two workers share a stream.
Sampling with and without replacement have different defaults. Generator.choice(a, size) samples with replacement unless told otherwise (replace=True is its default), while pandas' DataFrame.sample samples without replacement by default. Drawing 100 indices from the 5,182 adults with rng.choice(5182, 100) repeated some adult in most seeds (median 1 repeated draw over the 30 seeds, IQR 0 to 1.75); the expected number of repeated pairs is $\binom{100}{2}/5{,}182 = 0.955$. Drawing 5,182 indices from 5,182 with replacement leaves out about a third of the rows: the distinct share was 0.6319 (IQR 0.6275 to 0.6340), close to $1 - 1/e = 0.6321$, the fraction behind the out-of-bag estimates of The Bootstrap Method and Bagging. A "random sample" drawn with the wrong default is a different design.
The birthday problem
How likely is it that among 23 people, two share a birthday? With 365 equally likely birthdays and no twins, the complement is easier: all 23 birthdays differ with probability
$$\frac{365}{365} \times \frac{364}{365} \times \cdots \times \frac{343}{365} = \prod_{i=0}^{22} \frac{365 - i}{365},$$
since the $i$-th person must avoid the $i$ birthdays already taken. One minus this product is 0.5073. The probability passes one half at 23 people, reaches 0.7063 at 30, 0.9704 at 50 and 0.9901 at 57. It is high because 23 people form $\binom{23}{2} = 253$ pairs, each a chance of a match.
The simulation draws all the birthdays at once as a $100{,}000 \times 23$ array of integers from 0 to 364, sorts each row and looks for equal neighbors:
rng = np.random.default_rng(seed)
b = np.sort(rng.integers(0, 365, size=(100_000, 23)), axis=1)
shared = (np.diff(b, axis=1) == 0).any(axis=1)
shared.mean()
Over 30 seeds the estimate was 0.5073 (IQR 0.5068 to 0.5082), ranging from 0.5046 to 0.5112. The Monte Carlo standard error for $m = 100{,}000$ is 0.00158, and 29 of the 30 estimates fell within two standard errors of the exact value, as about 95% should.
Writing the simulation as array operations rather than a Python loop over groups is what makes 100,000 repetitions take a fraction of a second; The NumPy Package explains why.
Simulating the right process
A simulation estimates the probability of the process it simulates, which may not be the process intended. Consider a random sample of 20 adults, drawn without replacement from the 5,182. The number with hypertension follows the hypergeometric distribution, and the probability that at least 15 of the 20 have it is 0.0525. A binomial model, which treats the 20 draws as independent with probability $2{,}837/5{,}182$ each, as if drawn with replacement, gives 0.0529, very close because 20 is a small fraction of 5,182. Simulating 20,000 samples per seed with rng.choice(5182, 20, replace=False) gave 0.0518 (IQR 0.0511 to 0.0530), within the Monte Carlo error of 0.0016 of the exact value. Probability Distributions in SciPy computes both exact values with scipy.
For a rare event the Monte Carlo error is small in absolute terms and large relative to $p$. The probability that all of 10 adults sampled without replacement have hypertension is 0.002402 (hypergeometric), and 0.002419 if the draws are made with replacement. A simulation using fast binomial draws, which correspond to sampling with replacement, gave:
| Simulations per seed | Estimate, median over 30 seeds (IQR) | Relative standard error |
|---|---|---|
| 10,000 | 0.002300 (0.001950 to 0.002600) | 0.203 |
| 1,000,000 | 0.002417 (0.002372 to 0.002440) | 0.020 |
The relative standard error, $\sqrt{(1-p)/(mp)}$, is 20% with 10,000 simulations: the estimate rests on about 24 occurrences of the event. A million simulations bring it to 2%, and at that precision the estimate settles on 0.002419, the probability of the process simulated, not the 0.002402 of sampling without replacement. Here the difference is negligible; in a small population it would not be. Rare events in general need either very many simulations or a smarter method, such as importance sampling, which simulates from a distribution in which the event is common and reweights.
When to simulate
Exact calculation is better when it is available: it has no Monte Carlo error and shows how the answer depends on its inputs. Simulation is better when the process is too complicated to analyze, such as a sampling design with several stages (Data Collection and Sampling compares four by simulation), a statistic without a known distribution (The Bootstrap Method) or a test whose error rates under a violated assumption are in question (Type I and Type II Errors). Doing both, where possible, is the best check on each: a simulation that disagrees with an exact answer by more than its Monte Carlo error has found a bug in one of them.
Corrections to the course notes
The unit's notes cover the definitions and rules accurately and are stated more precisely here in three places. They say the classical definition (outcomes counted) and the long-run frequency interpretation lead to the same value; they do when the outcomes are in fact equally likely, which the classical definition assumes rather than establishes. They say a probability of 0 means an event is impossible; that holds for finitely many outcomes, but a continuous variable takes each single value with probability 0, and it does take one of them. And they say the way to verify independence is to check whether the multiplication rule holds for the observed probabilities; in data it will not hold exactly even when the events are independent, so the check needs a test, as noted above. The notes' screening example uses figures from their course text; Predictive Values and Bayes' Theorem works the same reasoning on public data.
In Python
import numpy as np
import pandas as pd
from scipy import stats
f = adults[adults.bmi.notna()] # events need every variable known
H, O = f.htn == 1, f.bmi >= 30
H.mean(), (H & O).mean(), (H | O).mean()
H[O].mean() # P(H | O)
pd.crosstab(O, H, normalize="index") # all conditional probabilities by row
rng = np.random.default_rng(0) # one Generator, passed everywhere
f.sample(n=20, random_state=rng) # without replacement (pandas default)
rng.choice(len(f), 20, replace=False) # NumPy needs replace=False
children = np.random.SeedSequence(2026).spawn(30)
streams = [np.random.default_rng(c) for c in children]
p, m = 0.0525, 20_000
np.sqrt(p * (1 - p) / m) # Monte Carlo standard error
stats.hypergeom(5182, 2837, 20).sf(14) # exact P(X >= 15)
The numbers in this article are computed by tools/experiments/data-science/probability-fundamentals.py in the site's repository, from the NHANES table built by psdata.py beside it, which downloads the files from the CDC and checks their SHA-256 hashes; the figure is drawn by tools/figures/probability-fundamentals.py.
References
- NumPy developers (2025). Random sampling (
numpy.random):default_rng,Generator.choice,SeedSequence. NumPy reference. https://numpy.org/doc/stable/reference/random/index.html record - National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018 data files and documentation (DEMO_J, BPX_J, BPQ_J, BMX_J). Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 45 pages
- pandas development team (2026).
DataFrame.sample,crosstab. pandas API reference. https://pandas.pydata.org/docs/reference/ record