Conditional Probability and the Law of Total Probability

How the probability of an event changes when another event is known to have occurred. The definition and its reading as a restricted population; why a conditional probability obeys the axioms; P(A | B) against P(B | A); the multiplication and chain rules, with draws without replacement; independence restated; the law of total probability as a tree and Bayes' theorem for reversing a condition; crude rates and standardization as total probability; and confounding and Simpson's paradox. Worked on obesity, diabetes, age and smoking among NHANES adults.

Among the 2,099 obese adults in this unit's NHANES population, 432 have been diagnosed with diabetes: 20.6%. Among the 3,008 adults who are not obese, 345 have: 11.5%. Knowing that a randomly chosen adult is obese nearly doubles the probability of diabetes. Now turn the question around. Among the 777 adults with diabetes, 432 are obese: 55.6%. Same 432 people, very different number. How can learning one fact change the chance of another, and why does the direction of the question matter so much?

The first figure is the probability of diabetes given obesity; the second, of obesity given diabetes. Both are conditional probabilities: probabilities computed within the part of the population where some event is known to have occurred. They answer different questions, and mistaking one for the other is among the commonest errors in reading medical statistics.

First we define conditional probability and see why it obeys every rule of The Rules of Probability, which this article assumes. Then we develop the three results built on it:

  • the multiplication rule, which computes the probability that several things happen together, one step at a time;
  • the law of total probability, which computes the probability of an event by breaking the population into groups;
  • Bayes' theorem, which reverses a conditional probability, turning $P(B \mid A)$ into $P(A \mid B)$.

Finally we use total probability to explain why an overall comparison can exaggerate, or even contradict, every comparison within the groups. The examples continue with the same 5,107 adults: $H$ is hypertension, $O$ obesity (body mass index of 30 or more) and $D$ diagnosed diabetes.

The definition

Here is the idea in one sentence: learning that $B$ occurred shrinks the population to the members of $B$, and the probability of $A$ becomes the share of that smaller population that lies in $A$.

The formal version. For events $A$ and $B$ with $P(B) > 0$, the conditional probability of $A$ given $B$ is

$$P(A \mid B) = \frac{P(A \cap B)}{P(B)}.$$

The reasoning behind the formula is clearest with a finite population. Learning that $B$ occurred rules out every outcome outside $B$. The outcomes in $B$ remain, and nothing in the information favors one of them over another, so they keep their relative probabilities. The probability of $A$ becomes the share of $B$'s probability that lies in $A$. With equally likely outcomes, this is a ratio of counts:

$$P(A \mid B) = \frac{\text{number of outcomes in } A \cap B}{\text{number of outcomes in } B}.$$

So conditioning on $B$ amounts to treating $B$ as a new, smaller population. For the opening example,

$$P(D \mid O) = \frac{432}{2{,}099} = 0.2058, \qquad P(D \mid O^c) = \frac{345}{3{,}008} = 0.1147, \qquad P(O \mid D) = \frac{432}{777} = 0.5560.$$

The unconditional probabilities are $P(D) = 0.1521$ and $P(O) = 0.4110$. Grinstead and Snell (1997, §4.1) motivate the definition in the same way, by requiring that the outcomes left after the information arrives keep their relative probabilities.

A conditional probability is a probability

If conditioning creates a new population, every rule we already have should hold inside it. For fixed $B$, the function $A \mapsto P(A \mid B)$ satisfies the three axioms, so every rule of the previous article holds for it.

Proof. Nonnegativity: $P(A \cap B) \ge 0$ and $P(B) > 0$. Normalization: $P(S \mid B) = P(S \cap B)/P(B) = P(B)/P(B) = 1$. Additivity: if $A_1, A_2, \ldots$ are mutually exclusive, so are $A_1 \cap B, A_2 \cap B, \ldots$, and their union is $(A_1 \cup A_2 \cup \cdots) \cap B$. Dividing the additivity of $P$ by $P(B)$ gives $P(\bigcup A_i \mid B) = \sum P(A_i \mid B)$. $\square$

In particular $P(A^c \mid B) = 1 - P(A \mid B)$: given obesity, the probability of no diabetes is $1 - 0.2058 = 0.7942$. What does not hold is any rule that changes the condition. $P(A \mid B) + P(A \mid B^c)$ need not equal 1 ($0.2058 + 0.1147$ does not). And $P(A \mid B)$ need not equal $P(B \mid A)$.

The two directions

$P(D \mid O)$ and $P(O \mid D)$ share a numerator and differ in the denominator:

$$P(D \mid O) = \frac{P(O \cap D)}{P(O)}, \qquad P(O \mid D) = \frac{P(O \cap D)}{P(D)}.$$

So $P(O \mid D)/P(D \mid O) = P(O)/P(D)$. The two directions agree only when the two events are equally common. Here obesity is 2.7 times as common as diabetes, and the probabilities differ by the same factor.

Check your understanding. Of the adults with diabetes, 79.1% have hypertension ($P(H \mid D) = 615/777 = 0.7915$). Hypertension is far more common than diabetes. Is the share of hypertensive adults who have diabetes larger or smaller than 79.1%, and by roughly how much?

Much smaller. Only 22.1% of the adults with hypertension have diabetes ($P(D \mid H) = 615/2{,}783 = 0.2210$). The shared numerator is 615; the denominators are 777 and 2,783, so the two directions differ by the factor $2{,}783/777 = 3.6$, the ratio of $P(H)$ to $P(D)$. A statement such as "most diabetic adults are hypertensive" says nothing directly about how likely a hypertensive adult is to be diabetic.

In medicine the confusion has a standard form. The probability that a test is positive given disease, its sensitivity, is reported by the test's developers. The patient and the doctor need the probability of disease given a positive test. The two can differ by an order of magnitude when the disease is rare. The calculation that connects them is Bayes' theorem, below.

The multiplication rule

Read backwards, the definition builds a joint probability from a conditional one. This is the multiplication rule:

$$P(A \cap B) = P(B)\,P(A \mid B) = P(A)\,P(B \mid A).$$

It computes a joint probability in two stages: first the probability of one event, then the probability of the other in the population where the first has occurred. For obesity and diabetes, $P(O \cap D) = P(O)\,P(D \mid O) = 0.4110 \times 0.2058 = 0.0846$. This matches the direct count $432/5{,}107$.

Applied repeatedly, it becomes the chain rule:

$$P(A_1 \cap A_2 \cap \cdots \cap A_n) = P(A_1)\,P(A_2 \mid A_1)\,P(A_3 \mid A_1 \cap A_2) \cdots P(A_n \mid A_1 \cap \cdots \cap A_{n-1}).$$

It holds whenever $P(A_1 \cap \cdots \cap A_{n-1}) > 0$, which by monotonicity makes every conditioning event on the right positive.

Proof. By induction on $n$. For $n = 2$ it is the multiplication rule. If it holds for $n - 1$ events, apply the multiplication rule to the two events $A_1 \cap \cdots \cap A_{n-1}$ and $A_n$, and expand the first factor by the induction hypothesis. $\square$

The chain rule is the natural tool for sampling without replacement, where each draw changes the population for the next. Draw three adults one after another without replacement, and let $D_i$ be the event that the $i$th is diabetic. After a diabetic adult is drawn, 776 diabetic adults remain among 5,106:

$$P(D_1 \cap D_2 \cap D_3) = \frac{777}{5{,}107} \times \frac{776}{5{,}106} \times \frac{775}{5{,}105} = 0.003510.$$

Check your understanding. Drawing with replacement would make the three draws independent, so the probability would be $(777/5{,}107)^3$. Will that be noticeably different from 0.003510?

Barely: it is 0.003522. Removing one or two people hardly changes a population of 5,107. A simulation of 100,000 triples drawn without replacement, repeated with 30 seeds, gave a median of 0.003470 (interquartile range 0.003412 to 0.003587), in agreement with the exact value.

The complement rule gives the probability that at least one of the three is diabetic. By the chain rule, the probability that none is equals 0.60943, so at least one occurs with probability 0.39057. The difference between with and without replacement becomes large only when the sample is a sizable fraction of the population. That is the situation of the finite population correction in The Normal Approximation to the Binomial.

Independence, restated

The multiplication rule also gives independence its intuitive meaning. The previous article defined $A$ and $B$ to be independent when $P(A \cap B) = P(A)\,P(B)$. When $P(B) > 0$, dividing by $P(B)$ shows that this is equivalent to

$$P(A \mid B) = P(A):$$

learning that $B$ occurred does not change the probability of $A$. By the complement result of the previous article, it is then also true that $P(A \mid B^c) = P(A)$ when $P(B^c) > 0$. So independence means that $A$ has the same probability whether or not $B$ occurs.

The NHANES conditions are far from independent. The probability of diabetes is 0.2210 among adults with hypertension and 0.0697 among those without, against 0.1521 overall. For obesity it is 0.2058 and 0.1147. Comparing $P(A \mid B)$ with $P(A \mid B^c)$ in this way is the starting point of Relative Risk and the Odds Ratio, which turns the comparison into a single number.

The law of total probability

Often we know the probability of an outcome within each of several groups and want the overall probability. Diabetes, for instance, is far more common among older adults.

Here is the idea in one sentence: the overall probability is a weighted average of the group probabilities, with each group weighted by its share of the population.

The formal version. Divide the population into three age groups, $A_1$ (18 to 39), $A_2$ (40 to 59) and $A_3$ (60 and over). The groups are mutually exclusive and together make up the whole population, which makes them a partition of $S$. The law of total probability says that

$$P(D) = \sum_{i=1}^{k} P(A_i)\,P(D \mid A_i).$$

Proof. Because the $A_i$ partition $S$, the events $D \cap A_1, \ldots, D \cap A_k$ are mutually exclusive and their union is $D$: each outcome of $D$ lies in exactly one group. By additivity, $P(D) = \sum P(D \cap A_i)$. By the multiplication rule each term is $P(A_i)\,P(D \mid A_i)$. $\square$

The calculation is conveniently laid out as a tree, with one branch per group and then one per outcome. The probability of reaching a leaf is the product of the probabilities along its path, by the multiplication rule. The probability of $D$ is the sum over the leaves that end in $D$.

The law of total probability as a tree. The first branches split the 5,107 adults by age group, with the share of the population in each; the second give the probability of diagnosed diabetes within the group. Multiplying along a path gives the probability of that age group and diabetes together, and the three products sum to the overall probability, 0.1521. Notice that the oldest group's product is the largest by far: two thirds of the total comes from it.
The law of total probability as a tree. The first branches split the 5,107 adults by age group, with the share of the population in each; the second give the probability of diagnosed diabetes within the group. Multiplying along a path gives the probability of that age group and diabetes together, and the three products sum to the overall probability, 0.1521. Notice that the oldest group's product is the largest by far: two thirds of the total comes from it.
Age group $P(A_i)$ $P(D \mid A_i)$ $P(A_i)\,P(D \mid A_i)$
18–39 0.3290 (1,680 adults) 0.0202 (34 diabetic) 0.0067
40–59 0.3049 (1,557) 0.1342 (209) 0.0409
60 and over 0.3662 (1,870) 0.2856 (534) 0.1046
Total 1 0.1521

The sum is exactly $777/5{,}107 = 0.1521$. The rule is most useful in the other direction: when the overall probability is unknown and the group probabilities are known, or when the same group probabilities are to be combined with different weights. Two such uses follow.

Bayes' theorem

The tree gives $P(D \mid A_i)$, the probability of diabetes within an age group. Now ask the reverse question: which age group does a randomly chosen diabetic adult come from?

Check your understanding. Adults aged 60 and over are 37% of the population. What share of the diabetic adults would we expect to be 60 or over: about 37%, or much more?

Much more. The question asks for $P(A_i \mid D)$. The definition of conditional probability, with the multiplication rule in the numerator and total probability in the denominator, gives Bayes' theorem:

$$P(A_i \mid D) = \frac{P(A_i)\,P(D \mid A_i)}{\sum_{j=1}^{k} P(A_j)\,P(D \mid A_j)}.$$

The numerator is one leaf of the tree, and the denominator is the sum of the leaves ending in $D$. For the oldest group,

$$P(A_3 \mid D) = \frac{0.1046}{0.1521} = 0.6873,$$

which matches the count $534/777$. So although only 37% of the adults are 60 or over, 69% of the diabetic adults are. The youngest group, a third of the population, supplies 4.4% of them.

Grinstead and Snell (1997, §4.1) derive the theorem in this form. They call the partition's events hypotheses and $D$ the evidence. $P(A_i)$ is then the prior probability of a hypothesis, and $P(A_i \mid D)$ its posterior probability once the evidence is seen.

Bayes' theorem is the calculation that connects the accuracy of a diagnostic test to what a positive result means for a patient. With the partition "disease" and "no disease" and the evidence "positive test," it turns sensitivity and specificity into the probability of disease given a positive result. The next unit develops this in Predictive Values and Bayes' Theorem, after Sensitivity and Specificity. The same theorem ranks documents in Probability for Information Retrieval and classifies text in Naive Bayes.

Crude rates and standardization

The second use combines the same group probabilities with different weights. The law of total probability is the formula behind two results of the previous unit.

  • A crude death rate is the weighted average of the age-specific rates, with the population's own age distribution as weights. That is total probability with "death" for $D$, and it is the observation from which Rates in Public Health starts.
  • A directly standardized rate keeps the age-specific rates and replaces the weights with those of a standard population. It is total probability computed for a population that does not exist, to answer "what would the overall rate be if this population had the standard's ages?" Direct and Indirect Standardization develops the method.

The same device, applied to a comparison between two groups, exposes confounding.

Confounding and Simpson's paradox

Former smokers have more diabetes than adults who never smoked: 271 of 1,182 (22.9%) against 394 of 3,018 (13.1%), a ratio of 1.76. Is smoking history responsible for that much of a difference?

The two groups differ in another way. Former smokers are older, with a mean age of 58.3 years against 47.4, and 57% of them are 60 or over against 31% of never smokers. Age raises the probability of diabetes on its own, so part of the difference may be due to age alone.

Conditional probabilities obey the same rules, so the law of total probability applies within each smoking group. The probability of diabetes in a group is a weighted average of its age-specific probabilities, with that group's own age distribution as weights:

$$P(D \mid \text{former}) = \sum_{i} P(A_i \mid \text{former})\,P(D \mid \text{former} \cap A_i).$$

Check your understanding. Within each age group, will former smokers still have more diabetes than never smokers? If so, will the ratio be larger or smaller than the overall 1.76?

Age group Former: share Former: $P(D)$ Never: share Never: $P(D)$ Ratio
18–39 0.1734 0.0293 (6/205) 0.3794 0.0183 (21/1,145) 1.60
40–59 0.2538 0.1633 (49/300) 0.3078 0.1238 (115/929) 1.32
60 and over 0.5728 0.3191 (216/677) 0.3128 0.2733 (258/944) 1.17
All ages 0.2293 0.1306 1.76

Within every age group former smokers still have more diabetes, but by factors of 1.60, 1.32 and 1.17, all smaller than the overall 1.76. The overall comparison is inflated by its weights. It puts most of the former smokers' weight on the oldest group, where diabetes is common, and most of the never smokers' weight on the younger groups.

How much of the 1.76 is due to age? Give the former smokers the never smokers' age distribution, while keeping their own age-specific probabilities:

$$0.3794 \times 0.0293 + 0.3078 \times 0.1633 + 0.3128 \times 0.3191 = 0.1612.$$

That is a ratio of $0.1612/0.1306 = 1.23$ to the never smokers. The crude excess of 75.6% shrinks to 23.5%, so about 69% of the excess in the crude comparison is explained by age.

Left: the age distributions of never smokers and former smokers among the NHANES adults; former smokers are concentrated in the oldest group. Right: the probability of diagnosed diabetes within each age group. Notice that the gaps on the right are small compared with the overall comparison of 22.9% against 13.1%. Former smokers have more diabetes at every age, but the overall figure weights the oldest group, where diabetes is commonest, far more heavily for former smokers.
Left: the age distributions of never smokers and former smokers among the NHANES adults; former smokers are concentrated in the oldest group. Right: the probability of diagnosed diabetes within each age group. Notice that the gaps on the right are small compared with the overall comparison of 22.9% against 13.1%. Former smokers have more diabetes at every age, but the overall figure weights the oldest group, where diabetes is commonest, far more heavily for former smokers.

A confounder is a variable that is associated with the exposure (here, smoking history) and, separately, with the outcome (diabetes), and so can distort the comparison of the outcome between exposure groups. Breslow and Day (1980, §3.4) give the epidemiologist's treatment. Age is the classic confounder. That is why Rates and Standardization adjusts death rates for it, and why the Mantel–Haenszel methods of Relative Risk and the Odds Ratio adjust risk ratios for it.

When a comparison reverses

The distortion can be strong enough to reverse the direction of a comparison. Suppose one hospital treats mostly severe cases and another mostly mild ones. The first can have the better survival in both severity groups and still the worse survival overall. Its patients are weighted toward the severe group, where survival is low for everyone. A reversal of this kind, in which an association that holds within every group is reversed in the combined data, is known as Simpson's paradox, after Simpson (1951).

In the NHANES example the direction survives and only the size changes. Direct and Indirect Standardization shows a full reversal: U.S. death rates fell in every age group between 2007 and 2013 while the crude death rate rose.

Total probability also shows when a reversal is impossible. Here is the idea in one sentence: if the two exposure groups have the same weights, the overall comparison is a weighted average of the group comparisons and cannot change sign.

Proposition. Let $E$ be an exposure and $A_1, \ldots, A_k$ a partition such that $E \cap A_i$ and $E^c \cap A_i$ have positive probability for every $i$. Suppose the exposure groups have the same distribution over the groups of the partition, $P(A_i \mid E) = P(A_i \mid E^c)$ for every $i$, and $P(D \mid E \cap A_i) \ge P(D \mid E^c \cap A_i)$ for every $i$. Then $P(D \mid E) \ge P(D \mid E^c)$.

Proof. Write $w_i$ for the common value of $P(A_i \mid E)$ and $P(A_i \mid E^c)$. By total probability within each exposure group,

$$P(D \mid E) - P(D \mid E^c) = \sum_i w_i\,\big[P(D \mid E \cap A_i) - P(D \mid E^c \cap A_i)\big] \ge 0,$$

since every weight and every bracket is nonnegative. $\square$

This is the logic of randomization in a clinical trial. Assigning treatment by chance makes the treatment groups alike, in expectation, in the distribution of every other variable, measured or not, so none of them can confound the comparison. It is also the logic of standardization, which imposes a common distribution by calculation.

Two cautions

  • The data are cross-sectional, recording smoking and diabetes at the same visit, so they cannot show which came first. Some former smokers may have quit after their diagnosis. That would raise diabetes among former smokers for a reason that has nothing to do with smoking's effects.
  • Adjusting for age removes only the confounding by age. Former and never smokers differ in sex, weight and much else, and the ratio of 1.23 is not an estimate of the effect of having smoked. Adjusting for several variables at once is the work of regression models such as Logistic Regression.

Corrections to the course notes

The unit's notes present the law of total probability with $\sum P(A \mid B_i)$ in one place, without the weights $P(B_i)$. The law needs them, $P(A) = \sum P(A \mid B_i)\,P(B_i)$, as in their other statements of it. The notes' example $P(\text{Disease}) = P(\text{Disease} \mid \text{Symptom})P(\text{Symptom}) + P(\text{Disease} \mid \text{No Symptom})P(\text{No Symptom})$ is a correct use of the law with a two-event partition. Their description of Simpson's paradox, an association within every category that reverses in the combined data, agrees with the one here. The explanation is the difference in the categories' weights, as the proposition above shows. The course's formula sheets do not cover this unit.

Summary

Result Formula
Conditional probability $P(A \mid B) = P(A \cap B)/P(B)$, for $P(B) > 0$
Multiplication rule $P(A \cap B) = P(B)\,P(A \mid B) = P(A)\,P(B \mid A)$
Chain rule $P(A_1 \cap \cdots \cap A_n) = P(A_1)\,P(A_2 \mid A_1) \cdots P(A_n \mid A_1 \cap \cdots \cap A_{n-1})$
Independence $P(A \mid B) = P(A)$, equivalently $P(A \cap B) = P(A)P(B)$
Total probability $P(D) = \sum_i P(A_i)\,P(D \mid A_i)$ for a partition $A_1, \ldots, A_k$
Bayes' theorem $P(A_i \mid D) = P(A_i)\,P(D \mid A_i) \big/ \sum_j P(A_j)\,P(D \mid A_j)$

What we have learned

A conditional probability is an ordinary probability computed in a smaller population, and its direction matters: 79% of diabetic adults are hypertensive, but only 22% of hypertensive adults are diabetic. The multiplication rule builds joint probabilities one step at a time, the law of total probability writes an overall probability as a weighted average over groups, and Bayes' theorem reverses the condition. Total probability also explains confounding: former smokers' crude ratio of 1.76 falls to 1.23 when they are given the never smokers' ages.

Throughout, we compared groups by setting two conditional probabilities side by side, such as 0.2058 against 0.1147. Epidemiology summarizes such a comparison in one number, and there are two standard choices. Which to use, and what each kind of study can estimate, is the subject of Relative Risk and the Odds Ratio.

In Python

import pandas as pd

# counts of adults and of diabetic adults by age group (NHANES 2017-2018 adults of this unit)
t = pd.DataFrame({'n': [1680, 1557, 1870], 'diabetic': [34, 209, 534]},
                 index=['18-39', '40-59', '60+'])
p_age = t.n / t.n.sum()                 # P(A_i)
p_d_given_age = t.diabetic / t.n        # P(D | A_i)
p_d = (p_age * p_d_given_age).sum()     # total probability: 0.1521
print(p_age * p_d_given_age / p_d)      # Bayes: P(A_i | D) = 0.0438, 0.2690, 0.6873

Every number in this article is computed by tools/experiments/applied-statistics/conditional-probability-and-total-probability.py in the site's repository, from the NHANES files loaded by probdata.py.

References

  • Breslow, N. E., and Day, N. E. (1980). Statistical Methods in Cancer Research, Volume I: The Analysis of Case-Control Studies, §3.4 (Confounding). IARC Scientific Publications No. 32. International Agency for Research on Cancer. record · 3 pages
  • Grinstead, C. M., and Snell, J. L. (1997). Introduction to Probability (2nd rev. ed.), §4.1 (Discrete conditional probability, independent events, Bayes' formula). American Mathematical Society. record · 10 pages
  • National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018: DEMO_J, BPX_J, BPQ_J, BMX_J, DIQ_J and SMQ_J data files. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 45 pages
  • Pagano, M., Gauvreau, K., and Mattie, H. (2022). Principles of Biostatistics (3rd ed.). CRC Press. The course text. record · 35 pages
  • Simpson, E. H. (1951). The interpretation of interaction in contingency tables. Journal of the Royal Statistical Society, Series B, 13(2), 238–241. record