Of the 5,107 adults in this unit's NHANES population, 2,783 have hypertension, 2,099 are obese and 777 have been told by a doctor that they have diabetes. Choose one of them at random. What is the chance that the person has at least one of the three conditions?
The obvious answer is to add the three shares: $0.545 + 0.411 + 0.152 = 1.108$. That cannot be a probability. The sum counts the 1,356 obese adults with hypertension twice, and the 361 adults with all three conditions three times. Counting each adult once gives 3,617, a probability of 0.708. The correction for the overlaps is the inclusion–exclusion rule. It is one of a handful of rules that follow from three short statements, the axioms of probability.
First we set up the language of events and ask what a probability means. Then we state the axioms and prove from them the rules a statistician uses every day, checking each on the NHANES adults. Finally we define independence, which makes most probability calculations tractable, and separate it from mutual exclusivity, with which it is often confused.
Outcomes, events and their operations
A random experiment is any process whose result is not known in advance: rolling two dice, drawing an adult at random from a population, recording whether a patient recovers. Its sample space $S$ is the set of all possible results, called outcomes. Rolling two dice, one red and one green, has 36 outcomes, the ordered pairs $(r, g)$ with $r$ and $g$ from 1 to 6. Drawing one adult from the NHANES population has 5,107 outcomes, one per adult.
An event is a set of outcomes, a subset of $S$, and it occurs when the outcome of the experiment is one of its members. "The dice sum to 7" is the event $\{(1,6), (2,5), (3,4), (4,3), (5,2), (6,1)\}$. "The adult has hypertension," written $H$, is the set of 2,783 hypertensive adults. Events combine as sets do:
| Operation | Notation | Occurs when |
|---|---|---|
| Union | $A \cup B$ | $A$ occurs, or $B$ occurs, or both |
| Intersection | $A \cap B$ | both $A$ and $B$ occur |
| Complement | $A^c$ | $A$ does not occur |
The complement is also written $\bar{A}$ or $A'$. The empty set $\varnothing$ is the impossible event, and $S$ itself is the certain event. Two events are mutually exclusive, or disjoint, when they have no outcome in common: $A \cap B = \varnothing$. "Sum 7" and "doubles" are mutually exclusive, since a double sums to an even number. "Never smoked" and "currently smokes" are mutually exclusive for any adult.
Two identities of set algebra, De Morgan's laws, are used constantly:
$$(A \cup B)^c = A^c \cap B^c, \qquad (A \cap B)^c = A^c \cup B^c.$$
The first says that "not ($A$ or $B$)" is "neither $A$ nor $B$"; the second, that "not both" is "at least one fails." Both follow by checking that an outcome belongs to one side exactly when it belongs to the other.
Events, then, are sets. What number should each one get?
What a probability is
A probability assigns to each event a number between 0 and 1 that measures how likely it is. Two interpretations give the numbers their meaning.
Equally likely outcomes. Suppose the sample space is finite and there is no reason to favor one outcome over another, as with fair dice or a draw that gives every adult the same chance. Then the probability of an event is the share of outcomes it contains:
$$P(A) = \frac{\text{number of outcomes in } A}{\text{number of outcomes in } S}.$$
For two dice, $P(\text{sum } 7) = 6/36 = 1/6$. For a random adult, $P(H) = 2{,}783/5{,}107 = 0.5449$. So the probability of hypertension for a randomly drawn adult is the proportion of the population with hypertension. This is the link between probability and the descriptive statistics of Descriptive Statistics. It is also how probability enters every later unit of the course: a population proportion is the probability that a randomly sampled member has the characteristic.
Long-run relative frequency. If the experiment is repeated many times independently, the share of repetitions in which $A$ occurs settles toward $P(A)$. Drawing adults one at a time with replacement, the running share with hypertension wanders widely at first and then closes in on 0.5449.
How fast does it close in? Over 30 independent sequences of draws, the median distance between the running share and 0.5449 was 0.035 after 100 draws.
Check your understanding. After 10,000 draws, a hundred times as many, how far from 0.5449 would we expect the running share to be: a hundredth of 0.035, or something larger?
Larger: about a tenth. The median distance was 0.0085 after 1,000 draws and 0.003 after 10,000, shrinking roughly in proportion to $1/\sqrt{n}$, the rate that the standard error $\sqrt{p(1-p)/n}$ of the normal approximation predicts. A hundred times the draws buys only ten times the accuracy. The precise statement of this convergence, the law of large numbers, is proved from the rules of this article once random variables are defined.
Neither interpretation serves as a definition for every case. Many events have no equally likely outcomes, and many cannot be repeated. Modern probability therefore defines a probability by the properties it must have, and lets either interpretation, or a model, supply the numbers.
The axioms
Here is the idea in one sentence: whatever probabilities mean, they must never be negative, must give the certain event 1, and must add up correctly over events that cannot happen together.
The formal version. A probability on a sample space $S$ is a function $P$ that assigns a number $P(A)$ to each event $A$ and satisfies three axioms:
- Nonnegativity. $P(A) \ge 0$ for every event $A$.
- Normalization. $P(S) = 1$.
- Additivity. If $A_1, A_2, \ldots$ are mutually exclusive (no two share an outcome), then $$P(A_1 \cup A_2 \cup \cdots) = P(A_1) + P(A_2) + \cdots.$$
This axiomatic approach is due to Kolmogorov (1933). The third axiom is stated for a finite or an infinite sequence of events. For a finite sample space, as in every example of this article, finite sequences are all that is ever needed.
The axioms say nothing about where the numbers come from. Equally likely outcomes satisfy them:
- counts are nonnegative;
- $S$ contains all the outcomes;
- the count of a union of disjoint sets is the sum of their counts.
So do long-run relative frequencies, and so does any proportion of a finite population. Everything below is proved from the three axioms alone, so it holds for every one of these interpretations at once.
Rules proved from the axioms
What can three such short statements tell us? Each rule below is proved from the axioms and checked on the dice or the NHANES adults, and together they answer the opening question.
The complement rule
$$P(A^c) = 1 - P(A).$$
Proof. $A$ and $A^c$ are mutually exclusive and their union is $S$. By additivity and normalization, $P(A) + P(A^c) = P(S) = 1$. $\square$
Two consequences follow at once. Taking $A = S$, whose complement is $\varnothing$, gives $P(\varnothing) = 1 - 1 = 0$. And since $P(A^c) \ge 0$ by nonnegativity, $P(A) = 1 - P(A^c) \le 1$: no probability exceeds 1, a fact the axioms do not state but imply.
The complement rule is the standard route to "at least one" questions, which are hard to count directly and easy to count through their complement, "none." For two dice, "at least one six" has complement "no six," which happens in $5 \times 5 = 25$ of the 36 outcomes, so
$$P(\text{at least one six}) = 1 - \frac{25}{36} = \frac{11}{36}.$$
Back to the opening question. In the NHANES population, 1,490 adults have none of the three conditions, so the probability of at least one is $1 - 1{,}490/5{,}107 = 1 - 0.2918 = 0.7082$, the value found above by counting.
Monotonicity
If every outcome of $A$ is also in $B$ ($A \subseteq B$), then $P(A) \le P(B)$.
Proof. $B$ is the union of the disjoint events $A$ and $B \cap A^c$ (the outcomes of $B$ that are not in $A$). By additivity, $P(B) = P(A) + P(B \cap A^c) \ge P(A)$, since the second term is nonnegative. $\square$
An intersection is contained in each of its events, so $P(A \cap B) \le \min\{P(A), P(B)\}$; a union contains each of its events, so $P(A \cup B) \ge \max\{P(A), P(B)\}$. The share of adults with both hypertension and diabetes, 0.1204, is below the share with diabetes, 0.1521, as it must be.
The addition rule
For any two events,
$$P(A \cup B) = P(A) + P(B) - P(A \cap B).$$
The plan of the proof is to cut the union into pieces that do not overlap, so that the third axiom can add them.
Proof. Split $A \cup B$ into three disjoint pieces: the outcomes in $A$ only ($A \cap B^c$), in $B$ only ($A^c \cap B$), and in both ($A \cap B$). By additivity,
$$P(A \cup B) = P(A \cap B^c) + P(A^c \cap B) + P(A \cap B).$$
$A$ itself is the disjoint union of $A \cap B^c$ and $A \cap B$, so $P(A \cap B^c) = P(A) - P(A \cap B)$. Likewise $P(A^c \cap B) = P(B) - P(A \cap B)$. Substituting,
$$P(A \cup B) = P(A) - P(A \cap B) + P(B) - P(A \cap B) + P(A \cap B) = P(A) + P(B) - P(A \cap B). \qquad \square$$
The subtracted term corrects for the outcomes counted twice, once in $P(A)$ and once in $P(B)$. When $A$ and $B$ are mutually exclusive, $P(A \cap B) = P(\varnothing) = 0$, and the rule reduces to the third axiom. The general addition rule is sometimes listed among the axioms; it is a consequence of them, and the axiom is its special case for disjoint events.
For two dice, "sum 7" and "doubles" are mutually exclusive, so $P(\text{sum 7 or doubles}) = 1/6 + 1/6 = 1/3$. "First die even" and "sum 7" overlap in the three outcomes $(2,5)$, $(4,3)$ and $(6,1)$, so
$$P(\text{first even or sum } 7) = \frac{1}{2} + \frac{1}{6} - \frac{1}{12} = \frac{7}{12}.$$
Check your understanding. Suppose we knew only that $P(H) = 0.5449$ and $P(O) = 0.4110$, and nothing about how many adults have both. What is the smallest $P(H \cup O)$ could be, and the largest?
By monotonicity the union is at least as probable as its larger event, so at least 0.5449, which happens if every obese adult is hypertensive. Dropping the subtracted term can only raise the addition rule's right side, so the union is at most $0.5449 + 0.4110 = 0.9559$, which happens if no adult has both. The overlap decides where in that range the answer falls.
In NHANES the overlap is 0.2655, and
$$P(H \cup O) = 0.5449 + 0.4110 - 0.2655 = 0.6904,$$
which agrees with the direct count of 3,526 adults with at least one of the two.
Inclusion–exclusion for three events
The opening question involves three conditions. Applying the addition rule twice extends it to three events. Write $A \cup B \cup C$ as $(A \cup B) \cup C$:
$$P(A \cup B \cup C) = P(A \cup B) + P(C) - P\big((A \cup B) \cap C\big).$$
The last intersection distributes, $(A \cup B) \cap C = (A \cap C) \cup (B \cap C)$, and the addition rule applies to it as well, with $(A \cap C) \cap (B \cap C) = A \cap B \cap C$:
$$P\big((A \cap C) \cup (B \cap C)\big) = P(A \cap C) + P(B \cap C) - P(A \cap B \cap C).$$
Substituting both expansions,
$$P(A \cup B \cup C) = P(A) + P(B) + P(C) - P(A \cap B) - P(A \cap C) - P(B \cap C) + P(A \cap B \cap C).$$
The pattern shows in the figure.
- An adult in exactly one circle is counted once by the three single terms.
- An adult in exactly two circles is counted twice by the singles, and removed once by the one pairwise term that contains them.
- An adult in all three is counted three times by the singles, removed three times by the pairs, and restored once by the triple term.
Each adult in the union is counted exactly once. In counts for the NHANES population,
$$2{,}783 + 2{,}099 + 777 - 1{,}356 - 615 - 432 + 361 = 3{,}617,$$
and $3{,}617/5{,}107 = 0.7082$, the probability of at least one condition, now reached by rule.
The general rule for $n$ events alternates in the same way. Add the single probabilities, subtract those of all pairs, add those of all triples, and so on, ending with $(-1)^{n+1} P(A_1 \cap \cdots \cap A_n)$ (Grinstead and Snell, 1997, Theorem 3.8). It can be proved by induction on $n$, exactly as the three-event case was proved from the two-event case, or by checking that an outcome lying in exactly $k \ge 1$ of the events is counted $\binom{k}{1} - \binom{k}{2} + \cdots \pm \binom{k}{k} = 1 - (1 - 1)^k = 1$ time, by the binomial theorem.
Boole's inequality and the Bonferroni correction
The opening's naive sum, 1.108, was wrong as an answer. Is it good for anything? It is an upper bound, and in some settings a useful one.
Dropping the subtracted terms can only increase the right-hand side of the addition rule, so $P(A \cup B) \le P(A) + P(B)$. By induction, for any events,
$$P(A_1 \cup A_2 \cup \cdots \cup A_n) \le P(A_1) + P(A_2) + \cdots + P(A_n).$$
Proof. The case $n = 1$ is an equality. Suppose the bound holds for $n - 1$ events. Apply the two-event inequality to $A_1 \cup \cdots \cup A_{n-1}$ and $A_n$:
$$P(A_1 \cup \cdots \cup A_n) \le P(A_1 \cup \cdots \cup A_{n-1}) + P(A_n) \le \sum_{i=1}^{n-1} P(A_i) + P(A_n). \qquad \square$$
This is Boole's inequality, also called the union bound. For the three conditions it gives $P(H \cup O \cup D) \le 1.1081$, which is true and useless. Its value lies elsewhere: it needs nothing but the individual probabilities, and it is close to exact when the events rarely overlap.
That is the situation in Multiple Comparisons and the Bonferroni Correction. If $m$ tests are each run at level $\alpha/m$, the event "at least one false rejection" is a union of $m$ events, each of probability at most $\alpha/m$, so its probability is at most $m \times \alpha/m = \alpha$, whatever the dependence between the tests.
The rules so far hold for any events. The next property holds only for some pairs, and it is what makes probabilities multiply.
Independence
Two events are independent when
$$P(A \cap B) = P(A)\,P(B).$$
In words, the probability of both is the product of the separate probabilities. The next article shows that, when $P(B) > 0$, this is the same as saying that learning $B$ occurred leaves the probability of $A$ unchanged.
The product form is used here as the definition because it is symmetric in $A$ and $B$ and makes sense even when one of the events has probability 0. Grinstead and Snell (1997, §4.1) define independence through conditional probabilities instead, with a separate clause for events of probability 0, and prove that their definition is equivalent to the product rule (Theorem 4.1).
Check your understanding. Roll two fair dice. The sum involves the first die, so "first die even" and "sum 7" look related. Are they independent?
They are. Independence is a property of the probabilities, not of how the events look, and here
$$P(\text{first even} \cap \text{sum } 7) = \frac{3}{36} = \frac{1}{12} = \frac{1}{2} \times \frac{1}{6}.$$
The reason is that whatever the first die shows, exactly one face of the second die makes the sum 7. "Doubles" and "at least one six," by contrast, are not independent. $P(\text{doubles} \cap \text{a six}) = 1/36$, the single outcome $(6,6)$, while $P(\text{doubles})\,P(\text{a six}) = (1/6)(11/36) = 11/216$.
In a real population, exact independence is rare, and the interesting question is how far from independence two events are. For the NHANES conditions:
| Events | $P(A \cap B)$ | $P(A)P(B)$ | Ratio |
|---|---|---|---|
| Hypertension, obesity | 0.2655 | 0.2240 | 1.185 |
| Hypertension, diabetes | 0.1204 | 0.0829 | 1.452 |
| Obesity, diabetes | 0.0846 | 0.0625 | 1.353 |
Each pair occurs together more often than independence would predict: hypertension and diabetes about 45% more often. The ratio in the last column is one way to measure association. Relative Risk and the Odds Ratio develops the two measures epidemiology uses instead.
Complements of independent events
If $A$ and $B$ are independent, so are $A$ and $B^c$, and therefore also $A^c$ and $B^c$.
Proof. $A$ is the disjoint union of $A \cap B$ and $A \cap B^c$, so
$$P(A \cap B^c) = P(A) - P(A \cap B) = P(A) - P(A)P(B) = P(A)\,[1 - P(B)] = P(A)\,P(B^c).$$
Applying the same argument to the independent pair $B^c$ and $A$ gives the second statement. $\square$
This is what licenses a common calculation. If each of three independent tests has a probability 0.95 of not raising a false alarm, the probability that none does is $0.95^3 = 0.857$, and the probability of at least one false alarm is $1 - 0.857 = 0.143$. Boole's inequality would have bounded the same probability by $3 \times 0.05 = 0.15$. The exact value is a little smaller, because independent false alarms can occur together.
Several events
Three events $A$, $B$, $C$ are mutually independent when every pair is independent and, in addition,
$$P(A \cap B \cap C) = P(A)\,P(B)\,P(C).$$
The last condition does not follow from the pairs. Toss two fair coins and let $A$ be "first is heads," $B$ "second is heads" and $C$ "the two agree."
- Each event has probability $1/2$.
- Each pair intersects in a single outcome of probability $1/4 = (1/2)(1/2)$, so the events are pairwise independent.
- But $A \cap B \cap C$ is "both heads," of probability $1/4$, not $1/8$.
Knowing any two of the events determines the third. For $n$ events, mutual independence requires the product rule for every subcollection of two or more. It is the assumption behind the binomial distribution and behind most sampling models of the course, where the observations of a random sample are taken to be mutually independent.
Independence is not mutual exclusivity
The two properties are easily confused, because both describe events that "have nothing to do with each other." In fact they are close to opposites.
Check your understanding. In the NHANES population, 59.1% of the adults have never smoked and 17.8% currently smoke. Are "never smoked" and "currently smokes" independent events?
They are not, and the reason is general.
Proposition. If $A$ and $B$ are mutually exclusive and both have positive probability, they are not independent.
Proof. Mutual exclusivity gives $P(A \cap B) = P(\varnothing) = 0$, while $P(A)\,P(B) > 0$. The product rule fails. $\square$
Mutually exclusive events are, in fact, as dependent as two events can be: learning that one occurred makes the other impossible. Smoking status in the NHANES population is a partition of the adults into three mutually exclusive groups that together make up $S$: 59.1% have never smoked, 23.1% are former smokers and 17.8% currently smoke, shares that sum to 1. The probability that an adult is both a never smoker and a current smoker is 0, while the product of the two probabilities is $0.5910 \times 0.1776 = 0.1050$. Two events with positive probability can be independent only if they can occur together.
The dice give the same lesson: "sum 7" and "doubles" are mutually exclusive, each with probability $1/6$, so they are dependent, while "first even" and "sum 7," which overlap, are independent.
A note on the population
The probabilities in this article are exact proportions of a finite list of 5,107 examined adults, which is what makes it possible to check every rule by counting. They are not estimates for the United States. NHANES deliberately oversamples some groups, and national estimates require the survey's sampling weights, which this course sets aside, as the earlier units did. The rules themselves do not depend on this: they hold for any probability, whether it comes from counting a finite population, from a model or from long-run frequency.
Corrections to the course notes
- The unit's notes give the third axiom as the general addition rule, $P(A \cup B) = P(A) + P(B) - P(A \cap B)$. The axiom is the special case for mutually exclusive events, and the general rule is a theorem proved from it, as above. The longer outline in the notes states the axiom for mutually exclusive events, as here.
- The notes describe mutually exclusive events by $P(A \cap B) = 0$. The definition is $A \cap B = \varnothing$, which implies a zero probability. In the finite examples of this course the two agree, but in continuous models events with probability 0 need not be empty.
- The notes define independence by $P(A \mid B) = P(A)$, which requires $P(B) > 0$. The product rule above covers every case.
The course's formula sheets do not cover this unit.
Summary
| Rule | Statement |
|---|---|
| Axioms | $P(A) \ge 0$; $P(S) = 1$; $P(\bigcup A_i) = \sum P(A_i)$ for mutually exclusive $A_i$ |
| Complement | $P(A^c) = 1 - P(A)$; $P(\varnothing) = 0$; $P(A) \le 1$ |
| Monotonicity | $A \subseteq B \Rightarrow P(A) \le P(B)$ |
| Addition | $P(A \cup B) = P(A) + P(B) - P(A \cap B)$ |
| Inclusion–exclusion | $P(A \cup B \cup C) = \sum P(\text{single}) - \sum P(\text{pairs}) + P(A \cap B \cap C)$ |
| Boole's inequality | $P(\bigcup A_i) \le \sum P(A_i)$ |
| Independence | $P(A \cap B) = P(A)P(B)$; then also $A$, $B^c$ and $A^c$, $B^c$ |
| Mutually exclusive | $A \cap B = \varnothing$; with positive probabilities, never independent |
What we have learned
Three axioms are enough to prove every rule for combining events. Inclusion–exclusion corrects the naive sum for overlaps: 0.708 of the NHANES adults, not 1.108, have at least one of the three conditions. The naive sum survives as Boole's inequality, the bound behind the Bonferroni correction. Independence means that probabilities multiply; it is a fact about the numbers, not about how events look, and disjoint events with positive probability are never independent.
The NHANES conditions are far from independent: hypertension and diabetes occur together 45% more often than independence predicts. How much does learning one condition change the chance of another? That is conditional probability, the subject of Conditional Probability and the Law of Total Probability.
In Python
import itertools
from fractions import Fraction
S = list(itertools.product(range(1, 7), repeat=2)) # two dice
P = lambda ev: Fraction(sum(1 for w in S if ev(w)), len(S))
seven = lambda w: w[0] + w[1] == 7
even1 = lambda w: w[0] % 2 == 0
print(P(lambda w: 6 in w)) # 11/36
print(P(lambda w: even1(w) and seven(w)), P(even1) * P(seven)) # 1/12 1/12
# inclusion-exclusion from the NHANES counts
H, O, D, HO, HD, OD, HOD, N = 2783, 2099, 777, 1356, 615, 432, 361, 5107
print((H + O + D - HO - HD - OD + HOD) / N) # 0.7082
Every number in this article is computed by tools/experiments/applied-statistics/rules-of-probability.py in the site's repository, from the NHANES files loaded by probdata.py.
References
- Grinstead, C. M., and Snell, J. L. (1997). Introduction to Probability (2nd rev. ed.), §1.2 (Theorems 1.1 to 1.4), §3.2 (Theorem 3.8) and §4.1 (Definition 4.1, Theorem 4.1). American Mathematical Society. record · 10 pages
- Kolmogorov, A. N. (1956). Foundations of the Theory of Probability (2nd English ed., N. Morrison, Trans.). Chelsea. Originally published in German in 1933. record · 2 pages
- National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018: DEMO_J, BPX_J, BPQ_J, BMX_J, DIQ_J and SMQ_J data files. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 45 pages
- Pagano, M., Gauvreau, K., and Mattie, H. (2022). Principles of Biostatistics (3rd ed.). CRC Press. The course text. record · 35 pages