The 5,247 adults whose blood pressure was measured in the National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018 have 237 different systolic pressures between 72.67 and 238 mm Hg, each the mean of up to three readings (National Center for Health Statistics, 2020a). Pick one adult at random and call the pressure $X$. The course pages describe such a variable in one of two ways: by a table of probabilities when its values can be listed, or by a density curve when it is modeled as continuous. Is there one description that serves both, and also a variable that is neither? There is: the distribution function $F(x) = P(X \le x)$. For this population $F(130) = 0.6503$, and every probability about $X$ can be computed from $F$.
A random variable is a number determined by the outcome of a random experiment, and its distribution function gives, for each real number $x$, the probability that the variable is at most $x$. This article is the general treatment that the course page Random Variables, Expectation and Variance set aside when it restricted itself to discrete variables, and that The Normal Distribution set aside when it introduced densities. The other articles of Probability Theory build on the definitions and theorems given here.
First we prove the three properties that every distribution function has. Then we fit mass functions, densities and mixtures of the two into that one frame. Next we define the quantile function and prove the probability integral transform, the pair of theorems that turns any variable into a uniform one and back. Finally we find the distribution of a transformed variable, of a sum, and of a maximum, and say exactly where measure theory is being avoided and what that costs. The article assumes The Rules of Probability and one-variable calculus.
Random variables and events about them
Fix a random experiment with sample space $\Omega$, the set of its outcomes $\omega$, and a probability $P$ on its events that satisfies the three axioms of The Rules of Probability. The third axiom there is stated for a sequence of events: if $A_1, A_2, \ldots$ are mutually exclusive, then $P(A_1 \cup A_2 \cup \cdots) = P(A_1) + P(A_2) + \cdots$. That property is called countable additivity, and this article uses it for infinite sequences, which the course pages never needed.
A random variable $X$ is a function that assigns a real number $X(\omega)$ to each outcome. Drawing one of the 5,247 adults is the experiment, the adult is the outcome, and the adult's systolic pressure is $X$. An event about $X$ is a set of outcomes described by the value of $X$, such as $\{X \le 130\} = \{\omega : X(\omega) \le 130\}$. The distribution of $X$ is the assignment of a probability $P(X \in B)$ to each set $B$ of real numbers. Two variables on different experiments can have the same distribution, and everything in this article depends on $X$ only through its distribution.
Countable additivity has one consequence that the proofs below use repeatedly. Here is the idea: a growing sequence of events can be cut into disjoint layers, and additivity then adds the layers.
Lemma (continuity of probability). If $A_1 \subseteq A_2 \subseteq \cdots$ and $A = A_1 \cup A_2 \cup \cdots$, then $P(A_n) \to P(A)$. If $A_1 \supseteq A_2 \supseteq \cdots$ and $A = A_1 \cap A_2 \cap \cdots$, then $P(A_n) \to P(A)$.
Proof. For the increasing case, let $B_1 = A_1$ and $B_n = A_n \cap A_{n-1}^c$ for $n \ge 2$, the outcomes that enter at step $n$. The $B_n$ are mutually exclusive, $A_n = B_1 \cup \cdots \cup B_n$ and $A = B_1 \cup B_2 \cup \cdots$. By countable additivity, and then by additivity for finitely many events,
$$P(A) = \sum_{k=1}^{\infty} P(B_k) = \lim_{n \to \infty} \sum_{k=1}^{n} P(B_k) = \lim_{n \to \infty} P(A_n).$$
For the decreasing case, the complements increase, $A_1^c \subseteq A_2^c \subseteq \cdots$, and their union is $A^c$ (an outcome is outside the intersection exactly when it is outside some $A_n$). By the increasing case $P(A_n^c) \to P(A^c)$, and the complement rule gives $P(A_n) = 1 - P(A_n^c) \to 1 - P(A^c) = P(A)$. $\square$
The distribution function
The distribution function of $X$, also called the cumulative distribution function (CDF), is
$$F_X(x) = P(X \le x), \qquad x \in \mathbb{R}.$$
The subscript is dropped when only one variable is in play. For the blood pressures, $F(x)$ is the share of the 5,247 adults whose pressure is at most $x$: 2,269 of them are at or below 120, so $F(120) = 0.4324$, and 3,412 are at or below 130, so $F(130) = 0.6503$.
What must a function look like to be a distribution function? The next theorem gives three properties (its parts 1 to 3; part 4 is a consequence), and Theorem 3 below shows that they are the only requirements.
Theorem 1 (properties of a distribution function). Let $F$ be the distribution function of a random variable $X$.
- $F$ is nondecreasing: $F(a) \le F(b)$ when $a < b$.
- $F(x) \to 0$ as $x \to -\infty$ and $F(x) \to 1$ as $x \to +\infty$.
- $F$ is right-continuous: $F(x + h) \to F(x)$ as $h$ decreases to 0.
- The limit from the left, $F(x-) = \lim_{h \downarrow 0} F(x - h)$, exists and equals $P(X < x)$. So $P(X = x) = F(x) - F(x-)$, the size of the jump of $F$ at $x$.
The idea is that each statement is the continuity lemma applied to a sequence of events of the form $\{X \le x_n\}$.
Proof. (1) When $a < b$, the event $\{X \le a\}$ is contained in $\{X \le b\}$, and the monotonicity rule of The Rules of Probability gives $F(a) \le F(b)$.
(2) The events $\{X \le n\}$, $n = 1, 2, \ldots$, increase, and their union is all of $\Omega$, because the number $X(\omega)$ is at most $n$ once $n$ is large enough. By the lemma $F(n) \to P(\Omega) = 1$. Given $\varepsilon > 0$, choose $n$ with $F(n) > 1 - \varepsilon$; then $1 - \varepsilon < F(x) \le 1$ for every $x \ge n$ by part 1, which is the statement $F(x) \to 1$. The events $\{X \le -n\}$ decrease, and their intersection is empty, because no real number $X(\omega)$ is below every $-n$. By the lemma $F(-n) \to P(\varnothing) = 0$, and part 1 gives $0 \le F(x) < \varepsilon$ for all $x \le -n$ once $F(-n) < \varepsilon$.
(3) The events $\{X \le x + 1/n\}$ decrease, and their intersection is $\{X \le x\}$: a number is at most $x + 1/n$ for every $n$ exactly when it is at most $x$. So $F(x + 1/n) \to F(x)$. For $0 < h < 1/n$, part 1 gives $F(x) \le F(x + h) \le F(x + 1/n)$, so $F(x + h) \to F(x)$ as $h \downarrow 0$.
(4) The events $\{X \le x - 1/n\}$ increase, and their union is $\{X < x\}$: a number is at most $x - 1/n$ for some $n$ exactly when it is less than $x$. So $F(x - 1/n) \to P(X < x)$, and for $0 < h < 1/n$ part 1 gives $F(x - 1/n) \le F(x - h) \le P(X < x)$, so the left limit exists and equals $P(X < x)$. The events $\{X < x\}$ and $\{X = x\}$ are mutually exclusive with union $\{X \le x\}$, so $P(X = x) = F(x) - F(x-)$. $\square$
Probabilities of intervals follow by subtraction. The events $\{X \le a\}$ and $\{a < X \le b\}$ are mutually exclusive with union $\{X \le b\}$, which gives the first formula below. The second replaces $\{X \le a\}$ by $\{X < a\}$, of probability $F(a-)$ by part 4, and the third replaces $\{X \le b\}$ by $\{X < b\}$, of probability $F(b-)$:
$$P(a < X \le b) = F(b) - F(a), \qquad P(a \le X \le b) = F(b) - F(a-), \qquad P(a < X < b) = F(b-) - F(a).$$
For the blood pressures, $P(120 < X \le 130) = 0.6503 - 0.4324 = 0.2178$ from the unrounded values, and $P(X > 140) = 1 - F(140) = 1 - 0.7951 = 0.2049$.
Example (the jump at 120). Whether an endpoint is included matters exactly where $F$ jumps. Sixty adults have a pressure of exactly 120, so $P(X = 120) = 60/5{,}247 = 0.0114$ and $F(120-) = 0.4324 - 0.0114 = 0.4210$. The probability of a pressure below 120 is 0.4210 and of a pressure at or below 120 is 0.4324. The next larger value in the population is 120.67, so $F$ is flat from 120 up to 120.67: $F(120.5) = F(120) = 0.4324$.
A distribution function can jump at only countably many points, that is, its jumps can be listed in a sequence. The jumps are probabilities of mutually exclusive events $\{X = x\}$, so the jumps of size greater than $1/n$ number fewer than $n$, or their probabilities would add to more than 1. Listing the jumps larger than $1/2$, then those larger than $1/3$ not yet listed, and so on, lists them all.
Theorem 1 says what $F$ gives. Does it give everything, so that two variables with the same distribution function have the same distribution? For intervals, yes, by the formulas above, and so for finite unions of intervals. For every set that can be built from intervals by countably many unions, intersections and complements the answer is also yes, but that is a theorem of measure theory and is not proved here; Billingsley (1995) treats distribution functions in his Section 14.
Mass functions, densities and mixtures
The course pages meet distribution functions of two shapes. This section places both in the frame of Theorem 1 and adds a third.
Discrete variables. $X$ is discrete if there is a finite or countable list of values $x_1, x_2, \ldots$ with $\sum_k P(X = x_k) = 1$. Its probability mass function (pmf) is $p_X(x) = P(X = x)$. Then for every set $B$, by countable additivity,
$$P(X \in B) = \sum_{k :\, x_k \in B} p_X(x_k), \qquad\text{in particular}\qquad F(x) = \sum_{k :\, x_k \le x} p_X(x_k).$$
The pmf and the distribution function carry the same information: $F$ is the running total of $p_X$, and $p_X(x) = F(x) - F(x-)$ is the jump of $F$ at $x$ by Theorem 1. The blood pressure of a randomly chosen adult is discrete, with 237 values. Its largest jump is at 125.33, the pressure of 92 adults, a probability of 0.0175.
Variables with a density. $X$ is continuous, or has a density, if there is a function $f_X \ge 0$ with
$$F(x) = \int_{-\infty}^{x} f_X(t)\,dt \qquad \text{for every } x.$$
Such an $f_X$ is a probability density function (pdf) of $X$. Letting $x \to \infty$ and using part 2 of Theorem 1 shows that the total area under a density is 1. The integrals in this article are Riemann integrals, improper where the range is infinite, and the densities are continuous except at finitely many points. Three facts follow from the definition and the fundamental theorem of calculus.
- $F$ is continuous, since an integral is a continuous function of its upper limit. By part 4 of Theorem 1, $P(X = x) = F(x) - F(x-) = 0$ for every $x$: a variable with a density takes each single value with probability zero. This is the fact The Normal Distribution states for heights, and it is why endpoints never matter for such a variable.
- $P(a < X \le b) = F(b) - F(a) = \int_a^b f_X(t)\,dt$: probabilities are areas under the density.
- $F'(x) = f_X(x)$ at every point $x$ where $f_X$ is continuous. So a density is found by differentiating the distribution function.
The third fact has a converse that the section on transformations uses. Suppose $F$ is a continuous distribution function whose derivative exists and is continuous except at finitely many points $c_1 < \cdots < c_m$. Then $f = F'$ (given any values at the $c_j$) is a density of $F$. To see it, take $a < b$ in the same interval between consecutive exceptional points. The fundamental theorem of calculus gives $F(b) - F(a) = \int_a^b F'(t)\,dt$. Because $F$ is continuous, letting $a$ and $b$ tend to the ends of the interval extends the formula to the closed interval, with the integral improper if $F'$ is unbounded near an end. Adding the pieces gives $F(x) - F(y) = \int_y^x F'(t)\,dt$ for all $y < x$, and letting $y \to -\infty$ gives $F(x) = \int_{-\infty}^x F'(t)\,dt$.
A density is not unique: changing $f_X$ at finitely many points changes no integral. And a density is not a probability. Its values can exceed 1, as the course page shows with heights measured in meters.
Example (a variable that is neither). C-reactive protein (CRP) was measured in 5,146 of the NHANES adults. The assay cannot quantify a concentration below its lower limit of detection, 0.15 mg/L. For such a result the data file holds a fill value, the limit divided by $\sqrt{2}$, which is 0.11 mg/L (National Center for Health Statistics, 2020b). Twelve adults, 0.23% of the 5,146, have it. A model for the reported value must therefore put positive probability on the single number 0.11 and spread the rest continuously over the values at or above 0.15. Its distribution function has one jump, so the variable has no density, and it is not discrete either, because the jump accounts for only part of the probability. Problem 1 below works such a distribution in full.
A distribution function that is a weighted average of a discrete one and one with a density, $F = \alpha F_{\mathrm{d}} + (1 - \alpha)F_{\mathrm{c}}$ with $0 < \alpha < 1$, is called a mixed distribution. Theorem 1 covers it without any change, which is the reason for working with $F$. Mixed distributions do not exhaust the possibilities: there are continuous distribution functions that have no density at all. The Cantor function is the standard example, treated under "Singular Functions" in Section 31 of Billingsley (1995). None appears in this collection.
The quantile function
The distribution function answers "what share of the population is at or below 130?" The inverse question asks below what pressure half the adults lie, and for this population the answer will be 123.33 mm Hg. Because $F$ may jump over the level one half and may be flat at it, the inverse has to be defined with care.
For $0 < u < 1$ the quantile function of $X$ is
$$Q_X(u) = \inf\{x : F(x) \ge u\},$$
the smallest value at which the distribution function has reached the level $u$. The set is not empty, because $F(x) \to 1 > u$, and it is bounded below, because $F(x) \to 0 < u$, so the infimum is a real number. $Q(1/2)$ is the median, and $Q(1/4)$ and $Q(3/4)$ are the quartiles.
Theorem 2 (the quantile function inverts the distribution function). For $0 < u < 1$ and every real $x$,
$$Q(u) \le x \quad\Longleftrightarrow\quad u \le F(x).$$
Also, $F(Q(u)-) \le u \le F(Q(u))$.
The idea is that the infimum in the definition is attained, because $F$ is right-continuous, so $Q(u)$ is itself a point where $F$ has reached $u$.
Proof. First, $F(Q(u)) \ge u$. By the definition of an infimum there are points $x_n$ decreasing to $Q(u)$ with $F(x_n) \ge u$, and right-continuity gives $F(Q(u)) = \lim F(x_n) \ge u$.
Now the equivalence. If $u \le F(x)$, then $x$ belongs to the set $\{x : F(x) \ge u\}$, so the infimum of the set is at most $x$: $Q(u) \le x$. Conversely, if $Q(u) \le x$, then $F(x) \ge F(Q(u)) \ge u$, by monotonicity and the first step.
Finally, every $x < Q(u)$ lies outside the set, so $F(x) < u$. Letting $x$ increase to $Q(u)$ gives $F(Q(u)-) \le u$. $\square$
Four consequences are worth stating, each read off from the theorem.
- $Q$ is nondecreasing. If $u \le v$, then $u \le v \le F(Q(v))$, so $Q(u) \le Q(v)$ by the equivalence with $x = Q(v)$.
- $P(X < Q(u)) \le u \le P(X \le Q(u))$, by part 4 of Theorem 1. At most a share $u$ of the distribution lies strictly below the quantile, and at least $u$ lies at or below it.
- If $F$ is continuous at $Q(u)$, then $F(Q(u)) = u$, since the two ends of the last inequality coincide.
- $Q(F(x)) \le x$ whenever $0 < F(x) < 1$, by the equivalence with $u = F(x)$. Equality can fail where $F$ is flat to the left of $x$.
If $F$ is continuous and strictly increasing on the interval where $0 < F < 1$, then $F$ has an ordinary inverse there and $Q = F^{-1}$: the equation $F(x) = u$ has exactly one solution, and it is the smallest $x$ with $F(x) \ge u$. That is the case for the normal distribution, where $Q(u) = \mu + \sigma\,\Phi^{-1}(u)$. In general the two functions mirror each other. A jump of $F$ is a flat stretch of $Q$: if $F(x_0-) < F(x_0)$, then for every $u$ with $F(x_0-) < u \le F(x_0)$ the equivalence gives $Q(u) \le x_0$, and $Q(u) \le x$ fails for each $x < x_0$ because $F(x) \le F(x_0-) < u$, so $Q(u) = x_0$. A flat stretch of $F$ is a jump of $Q$: if $F$ equals $u_0$ on an interval $[a, b)$ with $0 < u_0 < 1$, then $Q(u_0) \le a$, while for $u > u_0$ no $x < b$ has $F(x) \ge u$, so $Q(u) \ge b$.
The blood pressures. The median is $Q(1/2) = 123.33$ mm Hg. Seventy-nine adults have exactly that pressure. Just below it the distribution function is $F(123.33-) = 0.4946$, and at it $F(123.33) = 0.5096$. No value has $F$ equal to one half, and the definition settles the matter: 123.33 is the smallest pressure at which at least half the adults have been counted. The quartiles are $Q(1/4) = 112.00$ and $Q(3/4) = 136.67$, with $F$ equal to 0.2546 and 0.7530 there, and $Q(0.95) = 164.67$. These quartiles are the values 112.0 and 136.67 that Measures of Center and Spread finds by every interpolating rule in its table. For a flat stretch, take $x = 120.5$, between the values 120 and 120.67: $Q(F(120.5)) = 120$, which is less than 120.5.
The script checks the equivalence of Theorem 2 on this population for all 237 values $x$ and 1,235 levels $u$ (the thousandths and every value that $F$ takes), 292,695 pairs, with exact fractions.
The probability integral transform
Theorem 2 has two consequences that go by one name. The first makes a variable with any prescribed distribution out of a uniform one. The second turns a continuous variable into a uniform one.
A random variable $U$ is uniform on $(0, 1)$ if its distribution function is $F_U(u) = u$ for $0 \le u \le 1$ (and 0 below, 1 above). It has the density 1 on $(0, 1)$, and $P(U \le u) = u$.
Theorem 3 (the inverse transform). Let $F$ be any function with properties 1 to 3 of Theorem 1, let $Q(u) = \inf\{x : F(x) \ge u\}$, and let $U$ be uniform on $(0, 1)$. Then $X = Q(U)$ has distribution function $F$.
Proof. The proof of Theorem 2 used only properties 1 to 3 of $F$, so its equivalence holds for this $Q$. Since $0 < U < 1$ with probability 1, for every real $x$
$$P(Q(U) \le x) = P(U \le F(x)) = F(x). \qquad \square$$
Theorem 3 is the converse promised after Theorem 1: every function with the three properties is the distribution function of some random variable, provided a uniform variable exists. That proviso is not empty. A uniform variable is a probability on the subsets of the unit interval under which each interval has probability equal to its length, and constructing that probability is a task for measure theory. Billingsley (1995) takes up the existence of probability measures in his Sections 2 and 3, "Probability Measures" and "Existence and Extension". This article takes the existence of a uniform variable as given.
The theorem is also a recipe, the inversion method of simulation: to draw from $F$, draw $U$ from a uniform random number generator and return $Q(U)$. Devroye (1986, Theorem 2.1, p. 28) states it for a continuous $F$. Theorem 3 shows that with the infimum in the definition of $Q$ it holds for every $F$, including the discrete blood pressure population, where $Q(U)$ is the pressure of the adult in position $\lceil 5{,}247\,U \rceil$ of the sorted list. Grinstead and Snell (1997, Corollary 5.2) give the strictly increasing case.
Theorem 4 (the probability integral transform). Let $X$ have distribution function $F$, and let $0 < u < 1$.
- $P(F(X) \le u) \le u$, whatever $F$ is.
- If $F$ is continuous, then $P(F(X) \le u) = u$: the variable $F(X)$ is uniform on $(0, 1)$.
- If $F$ has a jump, then $F(X)$ is not uniform.
Here is the idea. The set of $x$ with $F(x) \le u$ is everything to the left of some point $x^*$. So the event $\{F(X) \le u\}$ is the event that $X$ falls to the left of $x^*$, and its probability is the height of $F$ there. If $F$ is continuous it passes through the level $u$ at $x^*$. If $F$ jumps over the level, the probability falls short of $u$.
Proof. Let $S = \{x : F(x) \le u\}$ and $x^* = \sup S$. The set $S$ is not empty because $F(x) \to 0 < u$, and it is bounded above because $F(x) \to 1 > u$, so $x^*$ is a real number. Since $F$ is nondecreasing, every $x < x^*$ is in $S$ and no $x > x^*$ is. So $S$ is $(-\infty, x^*]$ or $(-\infty, x^*)$, according to whether $x^*$ itself belongs to it.
(1) If $x^* \in S$, then $\{F(X) \le u\} = \{X \le x^*\}$, with probability $F(x^*) \le u$. If $x^* \notin S$, then $\{F(X) \le u\} = \{X < x^*\}$, with probability $F(x^*-)$ by part 4 of Theorem 1. That is a limit of values $F(x)$ with $x < x^*$, each at most $u$, so it is at most $u$.
(2) Every $x > x^*$ has $F(x) > u$, so right-continuity gives $F(x^*) \ge u$. Every $x < x^*$ has $F(x) \le u$, so $F(x^*-) \le u$. If $F$ is continuous, $F(x^*-) = F(x^*)$, and both equal $u$. Then $x^* \in S$, and by the first case of part 1, $P(F(X) \le u) = F(x^*) = u$. For the endpoints: $P(F(X) \le 1) = 1$ trivially, and $P(F(X) \le 0) \le P(F(X) \le u) = u$ for every $u > 0$, so it is 0. So $F(X)$ has the uniform distribution function.
(3) Suppose $F$ jumps at $x_0$, so that $p = P(X = x_0) > 0$. When $X = x_0$ the variable $F(X)$ equals the number $F(x_0)$, so $P(F(X) = F(x_0)) \ge p > 0$. A uniform variable takes each single value with probability zero. $\square$
The name is older than these proofs: E. S. Pearson (1938) has "the probability integral transformation" in the title of a paper on testing goodness of fit, the probability integral being the distribution function. Angus (1994) gives a simple proof of the transform in full generality and contrasts it with the theorem behind the inversion method, which is the distinction between Theorems 4 and 3 here: Theorem 3 holds for every $F$, Theorem 4 gives a uniform variable only for a continuous one. Rosenblatt (1952) extended the transform to several variables with a joint density, by applying to each variable its conditional distribution function given the earlier ones; the results are independent uniform variables.
The transform on the blood pressures. Applied to the population's own distribution function, the transform is not uniform, because that $F$ is a step function. Part 1 still holds: $P(F(X) \le 0.1) = 0.0915$, $P(F(X) \le 0.5) = 0.4946$ and $P(F(X) \le 0.9) = 0.8978$, each a little below its level. Over all levels $u$, the shortfall $u - P(F(X) \le u)$ never exceeds the largest jump of $F$, which is 0.0175, and comes arbitrarily close to it just below the value that $F$ takes at 125.33.
Applied with a model's distribution function, the transform tests the model. If the pressures were normal with their own mean 126.36 and standard deviation 20.12, the transformed values $\Phi((X - 126.36)/20.12)$ would be uniform, apart from the small effect of the steps. The shares of the 5,247 adults in the ten tenths of $(0, 1)$ are
| Tenth | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
|---|---|---|---|---|---|---|---|---|---|---|
| Share | 0.053 | 0.148 | 0.128 | 0.119 | 0.124 | 0.100 | 0.081 | 0.067 | 0.074 | 0.106 |
in place of 0.1 each. The model expects nearly twice as many adults in its lowest tenth as there are, and too few in the next four: the pressures are skewed to the right, as The Normal Distribution finds with a normal probability plot. That plot is this same idea drawn with quantiles in place of probabilities.
Three uses of the two theorems recur across the site.
- Simulation. The inversion method draws from any distribution whose quantile function can be computed. Problem 3 makes a Cauchy variable this way.
- The p-value. A one-sided p-value is a tail probability of the test statistic $T$ computed under the null hypothesis, with $F$ the distribution function of $T$ under that hypothesis. For the lower tail it is $F(T)$, the probability of a value at most the one observed. For the upper tail it is $1 - F(T-)$, the probability of a value at least the one observed, which is $1 - F(T)$ when $F$ is continuous. When $F$ is continuous, part 2 says that $F(T)$ is uniform, and $1 - F(T)$ is uniform because $1 - U$ is uniform when $U$ is. Type I and Type II Errors proves this for the t-test and shows it by simulation. For a discrete statistic, part 1 gives $P(p \le \alpha) \le \alpha$ for the lower-tail p-value directly: the test is conservative. The upper-tail p-value of $T$ is the lower-tail p-value of $-T$, because $-T$ has the distribution function $F_{-T}(s) = P(T \ge -s) = 1 - F((-s)-)$, so that $F_{-T}(-T) = 1 - F(T-)$. Part 1 applied to $-T$ makes it conservative too.
- Checking a forecast. If a model issues a predictive distribution function $F_i$ for each outcome $x_i$, the values $F_i(x_i)$ should look uniform when the model is right, as the table above does not. That is a check of calibration, a subject of Statistics of Model Evaluation.
What the experiment shows. The script drew 10,000 uniform numbers for each of 30 seeds (NumPy's default_rng with seeds 0 to 29) and measured the largest vertical distance between the empirical distribution function of the transformed draws and the target distribution function.
| Transform | Target | Median distance | Interquartile range |
|---|---|---|---|
| $Q(U)$, blood pressure population | the population's $F$ | 0.0077 | 0.0059 to 0.0091 |
| $-\ln(1 - U)$ | exponential, $1 - e^{-x}$ | 0.0080 | 0.0065 to 0.0097 |
| $1 - e^{-X}$, $X$ from NumPy's exponential generator | uniform | 0.0086 | 0.0065 to 0.0101 |
In these runs the three distances are of the same size, a little under $1/\sqrt{10{,}000} = 0.01$. The share of the draws $Q(U)$ at or below 130 had median 0.6499 (interquartile range 0.6470 to 0.6538), against $F(130) = 0.6503$. The experiment illustrates Theorems 3 and 4 on one discrete and one continuous distribution; it does not prove them.
The distribution of a transformed variable
A pressure in mm Hg may be wanted in kilopascals, a concentration on a logarithmic scale, a deviation as its square. If $Y = g(X)$, what is the distribution of $Y$? In every case the method is the same: write the event $\{g(X) \le y\}$ as an event about $X$.
For a discrete $X$ the answer is a sum over the values that $g$ sends to $y$,
$$p_Y(y) = P(g(X) = y) = \sum_{x :\, g(x) = y} p_X(x),$$
by additivity, since $\{g(X) = y\}$ is the union of the mutually exclusive events $\{X = x\}$ over those $x$. For a variable with a density, the answer comes from one computation, done for a stretch where $g$ is monotone.
Lemma (one monotone piece). Let $X$ have density $f_X$. Let $J$ be an open interval on which $g$ is continuously differentiable with $g' > 0$ throughout or $g' < 0$ throughout, and let $h$ be the inverse of $g$ on the open interval $g(J)$. Then for every real $y$,
$$P\big(X \in J,\ g(X) \le y\big) = \int_{-\infty}^{y} f_X(h(t))\,\lvert h'(t)\rvert\,\mathbf{1}\{t \in g(J)\}\,dt,$$
where $\mathbf{1}\{\cdot\}$ is 1 when the statement in braces holds and 0 otherwise.
The idea is the substitution $x = h(t)$ in the integral of $f_X$ over the part of $J$ that $g$ sends below $y$.
Proof. Write $J = (a, b)$, where $a$ may be $-\infty$ and $b$ may be $+\infty$. The inverse $h$ exists and is continuously differentiable on $g(J)$, with $h'(t) = 1/g'(h(t))$, because $g'$ is continuous and never zero.
Suppose $g' > 0$ on $J$. Then $g(J) = (c, d)$ with $c = \lim_{x \downarrow a} g(x)$ and $d = \lim_{x \uparrow b} g(x)$, and $h$ is increasing with $h' > 0$. For $c < y < d$, an $x$ in $J$ has $g(x) \le y$ exactly when $x \le h(y)$, so
$$P\big(X \in J,\ g(X) \le y\big) = \int_{a}^{h(y)} f_X(x)\,dx = \int_{c}^{y} f_X(h(t))\,h'(t)\,dt,$$
by the substitution $x = h(t)$, under which $t$ runs from $c$ to $y$ as $x$ runs from $a$ to $h(y)$. That is the claimed integral. For $y \le c$ no $x$ in $J$ has $g(x) \le y$, and both sides are 0. For $y \ge d$ every $x$ in $J$ does, and both sides equal $P(X \in J) = \int_a^b f_X(x)\,dx = \int_c^d f_X(h(t))\,h'(t)\,dt$.
Suppose $g' < 0$ on $J$. Then $g(J) = (c, d)$ with $c = \lim_{x \uparrow b} g(x)$ and $d = \lim_{x \downarrow a} g(x)$, and $h$ is decreasing with $h' < 0$. For $c < y < d$, an $x$ in $J$ has $g(x) \le y$ exactly when $x \ge h(y)$, so
$$P\big(X \in J,\ g(X) \le y\big) = \int_{h(y)}^{b} f_X(x)\,dx = \int_{y}^{c} f_X(h(t))\,h'(t)\,dt = \int_{c}^{y} f_X(h(t))\,\lvert h'(t)\rvert\,dt,$$
by the same substitution, under which $t$ runs from $y$ down to $c$ as $x$ runs from $h(y)$ up to $b$; reversing the limits changes the sign, and $-h' = \lvert h'\rvert$. For $y \le c$ both sides are 0 and for $y \ge d$ both equal $P(X \in J)$, as in the first case. $\square$
Theorem 5 (change of variables). Let $X$ have density $f_X$, and suppose that with probability 1 the variable $X$ falls in one of finitely many disjoint open intervals $J_1, \ldots, J_m$, on each of which $g$ is continuously differentiable with $g'$ of one sign and never zero. Let $h_j$ be the inverse of $g$ on $g(J_j)$. Then $Y = g(X)$ has the density
$$f_Y(y) = \sum_{j=1}^{m} f_X(h_j(y))\,\lvert h_j'(y)\rvert\,\mathbf{1}\{y \in g(J_j)\}.$$
When $g$ is monotone on a single interval ($m = 1$), this is $f_Y(y) = f_X(h(y))\,\lvert h'(y)\rvert$ for $y$ in the range of $g$ and 0 elsewhere.
Proof. The events $\{X \in J_j\}$ are mutually exclusive and their probabilities add to 1, so $P(Y \le y) = \sum_j P(X \in J_j,\ g(X) \le y)$. By the lemma each term is the integral from $-\infty$ to $y$ of the $j$th term of the sum that defines $f_Y$. A finite sum of integrals is the integral of the sum, so $F_Y(y) = \int_{-\infty}^y f_Y(t)\,dt$, which is the definition of a density. $\square$
Grinstead and Snell (1997, Theorem 5.1 and Corollary 5.1) give the case of one strictly monotone piece. Casella and Berger (2002) treat the subject in their Section 2.1, "Distributions of Functions of a Random Variable". The factor $\lvert h'(y)\rvert$ is what distinguishes a density from a mass function: it accounts for the stretching of the scale. Three cases cover most uses.
Example (location and scale). For $Y = a + bX$ with $b \ne 0$, the inverse is $h(y) = (y - a)/b$ and $h'(y) = 1/b$, so
$$f_Y(y) = \frac{1}{\lvert b\rvert}\, f_X\Big(\frac{y - a}{b}\Big).$$
With $f_X = \phi$, the standard normal density, this is the normal density with mean $a$ and standard deviation $\lvert b\rvert$, and read backwards it is the standardization of The Normal Distribution.
Example (the lognormal density). Let $X$ be normal with mean $\mu$ and standard deviation $\sigma$, and $Y = e^X$. Here $g(x) = e^x$ is increasing on the whole line with range $(0, \infty)$, $h(y) = \ln y$ and $h'(y) = 1/y$. So
$$f_Y(y) = \frac{1}{y\,\sigma\sqrt{2\pi}}\exp\Big(-\frac{(\ln y - \mu)^2}{2\sigma^2}\Big), \qquad y > 0.$$
This is the density of a variable whose logarithm is normal, the model behind the log transformation of the course page. The factor $1/y$ is easy to forget: the density of $Y$ is not the normal density evaluated at $\ln y$.
Example (a square, two pieces). Let $Y = X^2$. With probability 1, $X$ is in $J_1 = (-\infty, 0)$ or $J_2 = (0, \infty)$. On $J_2$ the inverse is $h_2(y) = \sqrt{y}$, on $J_1$ it is $h_1(y) = -\sqrt{y}$, both for $y > 0$, and $\lvert h_j'(y)\rvert = 1/(2\sqrt{y})$. So
$$f_Y(y) = \frac{f_X(\sqrt{y}) + f_X(-\sqrt{y})}{2\sqrt{y}}, \qquad y > 0.$$
For a standard normal $X$ this gives $f_Y(y) = e^{-y/2}/\sqrt{2\pi y}$, the chi-squared density with one degree of freedom, which The Chi-Squared and t Distributions reaches by differentiating $P(-\sqrt{y} \le X \le \sqrt{y})$. The density is unbounded near 0 and still integrates to 1; its integral up to 3.841 is 0.950, the familiar two-sided normal probability for $1.96^2$.
Several variables: joint distributions, independence, sums and extremes
Statistics concerns samples, so one variable is rarely enough. This section collects the definitions for two or more variables on the same experiment, and proves the properties of independent variables that the course pages cite.
The joint distribution function of $X$ and $Y$ is $F_{X,Y}(x, y) = P(X \le x,\ Y \le y)$. For discrete variables the joint pmf is $p_{X,Y}(x, y) = P(X = x,\ Y = y)$, and the pmf of $X$ alone, called its marginal pmf, is $p_X(x) = \sum_y p_{X,Y}(x, y)$ by additivity. The variables have a joint density $f_{X,Y} \ge 0$ if
$$P(a < X \le b,\ c < Y \le d) = \int_a^b\!\!\int_c^d f_{X,Y}(x, y)\,dy\,dx$$
for every rectangle. We take two facts about joint densities as given. The probability that $(X, Y)$ falls in a region bounded by finitely many smooth curves is the double integral of $f_{X,Y}$ over the region. And a double integral of a nonnegative function may be computed as an iterated integral in either order. The second is Fubini's theorem (Billingsley, 1995, Section 18, "Product Measure and Fubini's Theorem"). The first extends the definition from rectangles to such regions, and is also cited. From the first, with the region $\{(s, t) : s \le x\}$, the marginal density of $X$ is
$$f_X(x) = \int_{-\infty}^{\infty} f_{X,Y}(x, y)\,dy, \qquad\text{since}\qquad P(X \le x) = \int_{-\infty}^{x}\Big[\int_{-\infty}^{\infty} f_{X,Y}(s, y)\,dy\Big]ds.$$
Where $f_X(x) > 0$, the conditional density of $Y$ given $X = x$ is $f_{Y \mid X}(y \mid x) = f_{X,Y}(x, y)/f_X(x)$. It is the subject of Conditional Expectation and the Laws of Total Expectation and Variance.
Random variables $X$ and $Y$ are independent if
$$P(X \in A,\ Y \in B) = P(X \in A)\,P(Y \in B)$$
for all sets $A$ and $B$ of real numbers, that is, if every event about $X$ is independent of every event about $Y$. Variables $X_1, \ldots, X_n$ are (mutually) independent if $P(X_1 \in A_1, \ldots, X_n \in A_n) = \prod_i P(X_i \in A_i)$ for all sets $A_1, \ldots, A_n$. Taking some of the sets to be the whole line shows that every subcollection of independent variables is independent. Variables that are independent and have the same distribution are called independent and identically distributed (i.i.d.).
The definition extends to vectors. Random vectors $(X_1, \ldots, X_k)$ and $(Y_1, \ldots, Y_m)$ are independent if
$$P\big((X_1, \ldots, X_k) \in A,\ (Y_1, \ldots, Y_m) \in B\big) = P\big((X_1, \ldots, X_k) \in A\big)\,P\big((Y_1, \ldots, Y_m) \in B\big)$$
for all sets $A$ of $k$-vectors and $B$ of $m$-vectors, and several vectors are mutually independent if the product formula holds for all of them at once. "Independent pairs" $(X_1, Y_1), \ldots, (X_n, Y_n)$ are mutually independent vectors in this sense.
Theorem 6 (criteria for independence).
- Discrete variables $X$ and $Y$ are independent if and only if $p_{X,Y}(x, y) = p_X(x)\,p_Y(y)$ for all $x$ and $y$.
- If $X$ and $Y$ have a joint density of the form $f_{X,Y}(x, y) = f_X(x)\,f_Y(y)$, where $f_X$ and $f_Y$ are densities (they are then the marginal densities, as integrating out the other variable shows), then the product formula holds for all intervals $A$ and $B$, and $F_{X,Y}(x, y) = F_X(x)\,F_Y(y)$. Conversely, if $X$ and $Y$ are independent with densities $f_X$ and $f_Y$, then $f_X(x)\,f_Y(y)$ is a joint density of the pair.
- If $X$ and $Y$ are independent, then $g(X)$ and $h(Y)$ are independent, for any functions $g$ and $h$. Likewise, if the vectors $(X_1, \ldots, X_k)$ and $(Y_1, \ldots, Y_m)$ are independent, then $g(X_1, \ldots, X_k)$ and $h(Y_1, \ldots, Y_m)$ are independent, for any functions $g$ of $k$ variables and $h$ of $m$ variables.
Proof. (1) If the variables are independent, take $A = \{x\}$ and $B = \{y\}$. Conversely, if the pmf factors, then for any sets $A$ and $B$
$$P(X \in A,\ Y \in B) = \sum_{x \in A}\sum_{y \in B} p_X(x)\,p_Y(y) = \Big(\sum_{x \in A} p_X(x)\Big)\Big(\sum_{y \in B} p_Y(y)\Big) = P(X \in A)\,P(Y \in B),$$
where the sums run over the listed values in $A$ and $B$; a double series of nonnegative terms can be summed in any order.
(2) For intervals $A = (a, b]$ and $B = (c, d]$, the integrand factors and so does the iterated integral:
$$P(X \in A,\ Y \in B) = \int_a^b\!\!\int_c^d f_X(x)\,f_Y(y)\,dy\,dx = \int_a^b f_X(x)\,dx \int_c^d f_Y(y)\,dy = P(X \in A)\,P(Y \in B).$$
Since single points have probability zero for variables with densities, the same holds for intervals with or without their endpoints, and letting $a, c \to -\infty$ gives $F_{X,Y} = F_X F_Y$ by the continuity lemma. Conversely, if $X$ and $Y$ are independent with densities, then the left side equals the right side by independence, and the right side equals the double integral of $f_X(x)\,f_Y(y)$ over the rectangle. So that product satisfies the definition of a joint density.
(3) The event $\{g(X) \in A\}$ is the event $\{X \in A'\}$ with $A' = \{x : g(x) \in A\}$, and $\{h(Y) \in B\}$ is $\{Y \in B'\}$ with $B' = \{y : h(y) \in B\}$. So
$$P(g(X) \in A,\ h(Y) \in B) = P(X \in A',\ Y \in B') = P(X \in A')\,P(Y \in B') = P(g(X) \in A)\,P(h(Y) \in B).$$
For vectors, $\{g(X_1, \ldots, X_k) \in A\}$ is the event that $(X_1, \ldots, X_k)$ lies in the set $A'$ of $k$-vectors that $g$ sends into $A$, and $\{h(Y_1, \ldots, Y_m) \in B\}$ is the event that $(Y_1, \ldots, Y_m)$ lies in the set $B'$ of $m$-vectors that $h$ sends into $B$. The product formula for the sets $A'$ and $B'$, which the definition for vectors supplies, is the product formula for $g(X_1, \ldots, X_k)$ and $h(Y_1, \ldots, Y_m)$. $\square$
Part 3 is immediate because the definition asks for the product formula for all sets, and part 2 delivers it only for intervals. The step from intervals to all sets built from them by countable operations is a theorem of measure theory, cited and not proved here (Billingsley, 1995, Section 20, under "Independence"). The same remark applies to $n$ variables: a joint density that is a product of $n$ one-variable densities gives the product formula for intervals by the computation of part 2 with $n$ integrals, and for functions of disjoint groups of the variables by the fact stated next.
Fact (grouping), cited. If $X_1, \ldots, X_n$ are mutually independent, then vectors formed from disjoint groups of them, such as $(X_1, \ldots, X_k)$ and $(X_{k+1}, \ldots, X_n)$, are independent. For sets $A$ and $B$ that are products of sets of real numbers this is the definition of mutual independence. The step to all sets of vectors is the same theorem of measure theory as the step from intervals to all sets, and is not proved here. With part 3 of Theorem 6 it follows that functions of disjoint groups of independent variables are independent, and in the same way that the components of mutually independent vectors with different indices are independent of each other.
These are the properties that The Chi-Squared and t Distributions lists as cited. Casella and Berger (2002) treat them in Sections 4.1, "Joint and Marginal Distributions", and 4.2, "Conditional Distributions and Independence".
Theorem 7 (the distribution of a sum). Let $X$ and $Y$ be independent, and $Z = X + Y$.
- If both are discrete, $p_Z(z) = \sum_x p_X(x)\,p_Y(z - x)$.
- If both have densities, $Z$ has the density $f_Z(z) = \int_{-\infty}^{\infty} f_X(x)\,f_Y(z - x)\,dx$.
The operation on the right is the convolution of the two mass functions or densities. The idea: $Z = z$ happens when $X$ takes some value $x$ and $Y$ makes up the difference.
Proof. (1) The event $\{Z = z\}$ is the union of the mutually exclusive events $\{X = x,\ Y = z - x\}$ over the values $x$ of $X$, and each has probability $p_X(x)\,p_Y(z - x)$ by independence.
(2) By Theorem 6 the pair has joint density $f_X(x)\,f_Y(y)$. The event $\{Z \le z\}$ is the event that $(X, Y)$ falls in the half-plane $\{x + y \le z\}$. Computing the double integral as an iterated integral, substituting $y = t - x$ in the inner integral, and then exchanging the order of integration,
$$P(Z \le z) = \int_{-\infty}^{\infty} f_X(x)\Big[\int_{-\infty}^{z - x} f_Y(y)\,dy\Big]dx = \int_{-\infty}^{\infty} f_X(x)\Big[\int_{-\infty}^{z} f_Y(t - x)\,dt\Big]dx = \int_{-\infty}^{z}\Big[\int_{-\infty}^{\infty} f_X(x)\,f_Y(t - x)\,dx\Big]dt.$$
The inner integral on the right is $f_Z(t)$, so $F_Z(z) = \int_{-\infty}^z f_Z(t)\,dt$. $\square$
The proof of part 2 rests on the two facts about double integrals taken as given above. Grinstead and Snell (1997) state the result as their Theorem 7.1 and leave its proof as an exercise. It is the convolution formula that the course's derivation of the chi-squared density cites.
Theorem 8 (maximum and minimum). Let $X_1, \ldots, X_n$ be independent with distribution functions $F_1, \ldots, F_n$. Then
$$P(\max_i X_i \le x) = \prod_{i=1}^n F_i(x), \qquad P(\min_i X_i > x) = \prod_{i=1}^n \big(1 - F_i(x)\big).$$
Proof. The largest of the variables is at most $x$ exactly when every one of them is, and the smallest exceeds $x$ exactly when every one does. Independence, with the sets $A_i = (-\infty, x]$ in the first case and $A_i = (x, \infty)$ in the second, turns each probability into a product. $\square$
For i.i.d. variables the maximum and the minimum therefore have the distribution functions $F(x)^n$ and $1 - (1 - F(x))^n$. Problem 4 applies them.
What is proved here, what is cited, and what measure theory would add
This unit avoids measure theory. The avoidance is deliberate, and its cost can be stated exactly.
Proved on this page: the continuity lemma; Theorems 1 to 8 as stated, with Theorem 6 part 2 for intervals and Theorem 7 part 2 given the two facts about double integrals. Theorem 8 for $n$ variables uses only the definition of mutual independence.
Taken as given or cited:
- Which sets have probabilities. The general theory does not assign a probability to every set of outcomes of an uncountable sample space. It works with a collection of events closed under countable unions and complements, and calls a function $X$ a random variable only if every $\{X \le x\}$ is in the collection. This article writes $P(X \in B)$ for "all sets $B$" and means all sets $B$ for which $\{X \in B\}$ is in that collection. Nothing here depends on the difference, but a careful proof of part 3 of Theorem 6 must check that $g(X)$ is again a random variable.
- Existence. That a uniform variable exists, and that a sequence of independent variables with prescribed distributions exists, as every statement about i.i.d. $X_1, X_2, \ldots$ in this unit assumes. The second follows from Kolmogorov's existence theorem (Billingsley, 1995, Section 36).
- Uniqueness and extension. That the distribution function determines $P(X \in B)$ for every such set $B$, and a joint distribution function the joint distribution of several variables. That the product formula for intervals implies it for all such sets. And the grouping fact: that mutually independent variables form independent vectors when collected in disjoint groups.
- Integration. That a joint density gives the probability of a region bounded by smooth curves, not only of a rectangle, and Fubini's theorem for double integrals. Both are used for marginal densities and for the convolution formula.
What is lost. Without the general (Lebesgue) integral, a variable is handled through a mass function, a density, or an explicit mixture, and a theorem about "any random variable" has to be proved once for each kind or else through the distribution function, as Theorems 1 to 4 are. The next article, Expectation, Variance and Covariance, meets the same seam in the definition of the expected value, and Conditional Expectation and the Laws of Total Expectation and Variance meets it when conditioning on an event of probability zero. For discrete variables, variables with piecewise continuous densities and mixtures of the two, which covers every one-variable distribution used on this site, the results of this article hold as proved. What is given up is generality, and the proofs of the four items above. Joint distributions are another matter. A pair can have neither a joint mass function nor a joint density although each variable has one, as a variable and its square do, or an observation and the largest of its sample. Such pairs need a direct argument or the general theory, and Conditional Expectation and the Laws of Total Expectation and Variance treats the second example.
Worked problems
These problems are posed here, on this unit's NHANES data and on small exact cases; none is from a textbook. The script tools/experiments/statistics/random-variables-and-distribution-functions.py in the site's repository checks every number, the exact ones by formula and by simulation. Decimals are rounded to four places unless fewer are shown.
Problem 1 (a measurement with a detection limit). An assay measures a concentration $Y$ that has the exponential density $e^{-y}$ for $y > 0$, in units in which the mean is 1. The assay cannot quantify values below $\tfrac12$ and reports $R = \max(Y, \tfrac12)$. (a) Find the distribution function of $R$ and classify it. (b) Find $P(\tfrac12 < R \le 1)$, $P(R < 1)$ and $P(R = \tfrac12)$. (c) Find the quantile function of $R$, its median and $Q(0.2)$. (d) Find the distribution of $F_R(R)$. Is it uniform?
Solution. (a) The distribution function of $Y$ is $F_Y(y) = \int_0^y e^{-t}\,dt = 1 - e^{-y}$ for $y \ge 0$. For $x < \tfrac12$ the event $\{R \le x\}$ is impossible, since $R \ge \tfrac12$ always. For $x \ge \tfrac12$, $\max(Y, \tfrac12) \le x$ exactly when $Y \le x$. So
$$F_R(x) = 0 \ \text{ for } x < \tfrac12, \qquad F_R(x) = 1 - e^{-x} \ \text{ for } x \ge \tfrac12.$$
It satisfies the three properties of Theorem 1: it is nondecreasing, runs from 0 to 1, and is right-continuous at $\tfrac12$, where its value is the upper one. It has one jump, at $\tfrac12$, of size $F_R(\tfrac12) - F_R(\tfrac12-) = 1 - e^{-1/2} = 0.3935$. So $R$ has no density, because a variable with a density has a continuous distribution function. It is not discrete either, because the only value with positive probability carries 0.3935, not 1. It is mixed: with probability 0.3935 it is the constant $\tfrac12$, and otherwise it has the density $e^{-x}$ on $x > \tfrac12$, which integrates to the remaining $e^{-1/2} = 0.6065$.
(b) By the interval formulas, $P(\tfrac12 < R \le 1) = F_R(1) - F_R(\tfrac12) = (1 - e^{-1}) - (1 - e^{-1/2}) = 0.6321 - 0.3935 = 0.2387$ from unrounded values. $P(R < 1) = F_R(1-) = 0.6321$, because $F_R$ is continuous at 1. And $P(R = \tfrac12) = 0.3935$, the jump. For this variable $P(R \le \tfrac12)$ and $P(R < \tfrac12)$ differ, 0.3935 against 0.
(c) $Q(u)$ is the smallest $x$ with $F_R(x) \ge u$. For $u \le 0.3935$ the level is already reached at $x = \tfrac12$ and at no smaller point, so $Q(u) = \tfrac12$: the jump of $F_R$ is a flat stretch of $Q$. For $u > 0.3935$ we solve $1 - e^{-x} = u$, which gives $Q(u) = -\ln(1 - u)$. So $Q(0.2) = \tfrac12$, and the median is $Q(\tfrac12) = \ln 2 = 0.6931$, the same as the median of $Y$, since the limit only moves values that were below it.
(d) When $R = \tfrac12$, $F_R(R) = 0.3935$. Otherwise $F_R(R) = 1 - e^{-R}$ lies above 0.3935. So for $u < 0.3935$, $P(F_R(R) \le u) = 0$. For $u \ge 0.3935$, the event $\{F_R(R) \le u\}$ is $\{R \le -\ln(1 - u)\}$, with probability $F_R(-\ln(1 - u)) = u$. The variable $F_R(R)$ is not uniform: it has an atom of 0.3935 at the point 0.3935 and no probability below it. This is parts 1 and 3 of Theorem 4: $P(F_R(R) \le u) \le u$ always, with equality exactly for $u \ge 0.3935$. The inverse direction still works: by Theorem 3, $Q(U)$ has the distribution of $R$ for a uniform $U$, and indeed $Q(U) = \max(-\ln(1 - U), \tfrac12)$.
Problem 2 (the quantile function of a count). Random Variables, Expectation and Variance counts, for each of 4,730 adults with three blood pressure readings, the number $K$ of readings at or above 130 mm Hg: 2,778 adults have $K = 0$, 269 have $K = 1$, 268 have $K = 2$ and 1,415 have $K = 3$. (a) Find the quantile function of $K$. (b) Describe how to simulate $K$ from a uniform random number. (c) Find the distribution of $F(K)$ and compare $P(F(K) \le u)$ with $u$ at $u = 0.5$, 0.6 and 0.7.
Solution. (a) The distribution function takes the values $F(0) = 0.5873$, $F(1) = 0.6442$, $F(2) = 0.7008$ and $F(3) = 1$, the running totals of the shares 0.5873, 0.0569, 0.0567 and 0.2992. By the definition, $Q(u)$ is the smallest $k$ with $F(k) \ge u$:
$$Q(u) = 0 \ \text{ for } 0 < u \le 0.5873, \quad 1 \ \text{ for } 0.5873 < u \le 0.6442, \quad 2 \ \text{ for } 0.6442 < u \le 0.7008, \quad 3 \ \text{ for } 0.7008 < u < 1.$$
The median is $Q(0.5) = 0$: more than half the adults have no reading at or above 130. The third quartile is $Q(0.75) = 3$.
(b) By Theorem 3, draw $U$ uniform on $(0, 1)$ and return $Q(U)$: 0 if $U \le 0.5873$, 1 if $U$ is in the next interval, of length 0.0569, and so on. Each value is returned with probability equal to the length of its interval, which is its pmf. The script does this 200,000 times and recovers the four shares to within 0.005.
(c) $F(K)$ takes the value $F(k)$ when $K = k$. So it equals 0.5873, 0.6442, 0.7008 and 1 with probabilities 0.5873, 0.0569, 0.0567 and 0.2992. Then $P(F(K) \le 0.5) = 0$, $P(F(K) \le 0.6) = 0.5873$ and $P(F(K) \le 0.7) = 0.5873 + 0.0569 = 0.6442$. Each is at most its level $u$, as part 1 of Theorem 4 requires, and among levels $0 < u < 1$ equality holds only at the three values of $F$ below 1. A discrete variable transformed by its own distribution function is far from uniform when its jumps are large.
Problem 3 (the Cauchy distribution from a uniform variable). Let $U$ be uniform on $(0, 1)$ and $X = \tan\!\big(\pi(U - \tfrac12)\big)$. (a) Find the distribution function and the density of $X$. (b) Find its quartiles and $Q(0.975)$. (c) Show that $1/X$ has the same distribution as $X$. (d) Find the density of $W = X^2$ and $P(W \le 3)$.
Solution. (a) The function $g(u) = \tan(\pi(u - \tfrac12))$ is continuous and strictly increasing on $(0, 1)$, with range the whole line and inverse $h(x) = \tfrac12 + \arctan(x)/\pi$. So $X \le x$ exactly when $U \le h(x)$, and
$$F_X(x) = P(U \le h(x)) = \frac12 + \frac{\arctan x}{\pi}.$$
This is continuous with a continuous derivative, so by the converse of the third fact about densities, $X$ has the density
$$f_X(x) = F_X'(x) = \frac{1}{\pi(1 + x^2)},$$
the Cauchy density. Theorem 5 gives the same: $f_U(h(x))\,\lvert h'(x)\rvert = 1 \cdot \frac{1}{\pi(1 + x^2)}$. The construction is the inversion method of Theorem 3, with $Q = g$.
(b) $F_X$ is continuous and strictly increasing, so $Q(u) = F_X^{-1}(u) = \tan(\pi(u - \tfrac12))$. The quartiles are $Q(\tfrac14) = \tan(-\pi/4) = -1$ and $Q(\tfrac34) = 1$, and the median is 0. Half the probability lies between $-1$ and 1, and $F_X(1) = 0.75$. Further out, $Q(0.975) = \tan(0.475\pi) = 12.706$, against 1.96 for the standard normal: the Cauchy tails are far heavier.
(c) Let $Y = 1/X$. With probability 1, $X$ is in $J_1 = (-\infty, 0)$ or $J_2 = (0, \infty)$. On each, $g(x) = 1/x$ is decreasing with $g'(x) = -1/x^2 \ne 0$, it maps the interval onto itself, and its inverse is $h(y) = 1/y$ with $\lvert h'(y)\rvert = 1/y^2$. For $y > 0$ only the piece $J_2$ contributes, and for $y < 0$ only $J_1$. In both cases Theorem 5 gives
$$f_Y(y) = f_X\Big(\frac1y\Big)\frac{1}{y^2} = \frac{1}{\pi\,(1 + 1/y^2)\,y^2} = \frac{1}{\pi(y^2 + 1)},$$
the Cauchy density again.
(d) By the example of the square, for $w > 0$,
$$f_W(w) = \frac{f_X(\sqrt{w}) + f_X(-\sqrt{w})}{2\sqrt{w}} = \frac{1}{\pi\sqrt{w}\,(1 + w)}.$$
The probability is easier from the distribution function: $P(W \le 3) = P(-\sqrt3 \le X \le \sqrt3) = F_X(\sqrt3) - F_X(-\sqrt3) = 2\arctan(\sqrt3)/\pi = 2(\pi/3)/\pi = \tfrac23$, or 0.6667. The script confirms that the density integrates to this value.
Problem 4 (sums and extremes of exponential waiting times). Let $X_1, \ldots, X_5$ be independent, each with the exponential density $e^{-x}$ on $x > 0$. (a) Find the density of $S = X_1 + X_2$ and $P(S \le 2)$. (b) Find the distribution of the smallest of the five and the probability that it is at most 0.2. (c) Find the distribution function of the largest of the five, the probability that it is at most 2, and its median.
Solution. (a) By Theorem 7, $f_S(z) = \int f_{X_1}(x)\,f_{X_2}(z - x)\,dx$. The first factor is zero unless $x > 0$ and the second unless $z - x > 0$, so for $z > 0$ the integral runs over $0 < x < z$:
$$f_S(z) = \int_0^z e^{-x}\,e^{-(z - x)}\,dx = \int_0^z e^{-z}\,dx = z\,e^{-z}, \qquad z > 0,$$
and $f_S(z) = 0$ for $z \le 0$. Integrating by parts, $P(S \le 2) = \int_0^2 z e^{-z}\,dz = \big[-z e^{-z} - e^{-z}\big]_0^2 = 1 - 3e^{-2} = 0.5940$. The sum of two waiting times has a density that starts at zero, unlike either term: a short total needs both terms short.
(b) For one variable, $1 - F(x) = e^{-x}$. By Theorem 8, $P(\min_i X_i > x) = (e^{-x})^5 = e^{-5x}$, so the minimum has distribution function $1 - e^{-5x}$: it is exponential again, with mean $\tfrac15$. Then $P(\min_i X_i \le 0.2) = 1 - e^{-1} = 0.6321$.
(c) By Theorem 8, $P(\max_i X_i \le x) = (1 - e^{-x})^5$ for $x > 0$. At $x = 2$ this is $(1 - e^{-2})^5 = 0.4833$. The distribution function is continuous and strictly increasing, so the median solves $(1 - e^{-x})^5 = \tfrac12$: $x = -\ln\!\big(1 - 2^{-1/5}\big) = 2.0445$. The median of a single waiting time is $\ln 2 = 0.6931$; the largest of five has a median about three times that.
Summary
| Object | Definition or result |
|---|---|
| Distribution function | $F(x) = P(X \le x)$: nondecreasing, from 0 to 1, right-continuous; $P(X = x) = F(x) - F(x-)$ |
| Intervals | $P(a < X \le b) = F(b) - F(a)$ |
| Discrete | $F(x) = \sum_{x_k \le x} p(x_k)$ |
| Density | $F(x) = \int_{-\infty}^x f(t)\,dt$; $f = F'$ where $f$ is continuous; $P(X = x) = 0$ |
| Quantile function | $Q(u) = \inf\{x : F(x) \ge u\}$; $Q(u) \le x \iff u \le F(x)$ |
| Inverse transform | $Q(U)$ has distribution function $F$, for every $F$ |
| Integral transform | $P(F(X) \le u) \le u$; $F(X)$ is uniform when $F$ is continuous |
| Change of variables | $f_Y(y) = \sum_j f_X(h_j(y))\,\lvert h_j'(y)\rvert$ over the monotone pieces |
| Independence | $P(X \in A, Y \in B) = P(X \in A)P(Y \in B)$; the pmf or the joint density factors |
| Sum | $f_{X+Y}(z) = \int f_X(x)\,f_Y(z - x)\,dx$ for independent $X$, $Y$ |
| Extremes | $P(\max \le x) = \prod F_i(x)$; $P(\min > x) = \prod (1 - F_i(x))$ |
What we have learned
One function describes every random variable. The distribution function is nondecreasing, runs from 0 to 1 and is right-continuous, and any function with those properties is the distribution function of $Q(U)$ for a uniform $U$. Mass functions and densities are two ways of writing it, and a measurement with a detection limit needs both at once. The quantile function inverts it in the only sense available when it jumps or is flat, $Q(u) \le x$ exactly when $u \le F(x)$, and that equivalence gives the probability integral transform in both directions. For the blood pressures the transform under a normal model puts 5.3% of the adults in the lowest tenth where 10% belong, which is the population's skewness seen through $F$. Densities of transformed variables, sums and extremes all come from writing an event about the new variable as an event about the old ones.
The distribution function says where the probability lies. The next article, Expectation, Variance and Covariance, reduces a distribution to a few numbers, the expected value, the variance and, for two variables, the covariance and correlation, and proves the rules for computing them for discrete variables and for variables with densities.
The numbers in this article are computed and checked by tools/experiments/statistics/random-variables-and-distribution-functions.py in the site's repository, from the NHANES files loaded by tools/experiments/applied-statistics/statdata.py. The figure is drawn by tools/figures/random-variables-and-distribution-functions.py.
References
- Angus, J. E. (1994). The probability integral transform and related results. SIAM Review, 36(4), 652–654. https://doi.org/10.1137/1036146 record
- Billingsley, P. (1995). Probability and Measure (3rd ed.). Wiley. Sections 2 and 3 ("Probability Measures", "Existence and Extension"), 14 ("Distribution Functions"), 18 ("Product Measure and Fubini's Theorem"), 20 ("Random Variables and Distributions"), 31 ("Derivatives on the Line") and 36 ("Kolmogorov's Existence Theorem"). Section numbers and titles are those of the contents of the 2012 Anniversary Edition, which reissues the third edition. record · 8 pages
- Casella, G., and Berger, R. L. (2002). Statistical Inference (2nd ed.). Duxbury. Sections 1.5 ("Distribution Functions"), 1.6 ("Density and Mass Functions"), 2.1 ("Distributions of Functions of a Random Variable"), 4.1 ("Joint and Marginal Distributions") and 4.2 ("Conditional Distributions and Independence"). record · 11 pages
- Devroye, L. (1986). Non-Uniform Random Variate Generation. Springer. Chapter II, Section 2 ("The inversion method"), Theorem 2.1. record
- Grinstead, C. M., and Snell, J. L. (1997). Introduction to Probability (2nd rev. ed.). American Mathematical Society. Section 2.2 (densities and distribution functions), Section 5.2 (Theorem 5.1, Corollaries 5.1 and 5.2) and Section 7.2 (Theorem 7.1). record · 20 pages
- National Center for Health Statistics (2020a). National Health and Nutrition Examination Survey 2017–2018: Demographic Variables and Sample Weights (DEMO_J) and Blood Pressure (BPX_J) data files. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 67 pages
- National Center for Health Statistics (2020b). National Health and Nutrition Examination Survey 2017–2018 Data Documentation: High-Sensitivity C-Reactive Protein (HSCRP_J). Centers for Disease Control and Prevention. https://wwwn.cdc.gov/Nchs/Data/Nhanes/Public/2017/DataFiles/HSCRP_J.htm record · 3 pages
- Pearson, E. S. (1938). The probability integral transformation for testing goodness of fit and combining independent tests of significance. Biometrika, 30(1–2), 134–148. record
- Rosenblatt, M. (1952). Remarks on a multivariate transformation. The Annals of Mathematical Statistics, 23(3), 470–472. https://doi.org/10.1214/aoms/1177729394 record