Probability Inequalities: Markov, Chebyshev, Jensen and Cauchy–Schwarz

Four inequalities that bound a probability or an expected value from a mean, a variance or a convex function, each proved for random variables with the distributions that attain it. Markov's inequality and its general form, Chebyshev's inequality with its equality case and its one-sided version, the Chernoff bound, Jensen's inequality from a supporting line, and the Cauchy–Schwarz inequality with its equality case, all checked on the NHANES blood pressures. Worked problems on the pressure that each bound certifies as rare, the distributions that attain the bounds, three kinds of mean and the bias of the sample standard deviation, age against pressure, and a binomial tail.

Draw one adult at random from the 5,247 adults with a blood pressure reading in the National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018 (National Center for Health Statistics, 2020). Their systolic pressures have mean 126.36 mm Hg and variance 405.01. How likely is a pressure of 180 mm Hg or more, if the mean and the variance are all that is known? At most 0.7020 from the mean alone, and at most 0.1234 from the mean and the variance. The true share is 0.0181, and no better guarantee than those two can be given from that information: for each bound there is a distribution with the same mean, or the same mean and variance, that attains it.

A probability inequality bounds a probability or an expected value using a small part of what there is to know about a distribution: a mean, a variance, a moment generating function, the convexity of a function. Such a bound is true for every distribution that shares the stated quantities. That is its use in proofs, where the distribution is unknown or arbitrary, and it is also the reason the bound is usually far from the truth for any one distribution. This article proves the four inequalities that the rest of the site relies on, in the generality the later articles need, and says for each one when it holds with equality.

First we fix the three properties of expected values that every proof uses. Then we prove Markov's inequality and derive from it Chebyshev's inequality, its one-sided form, the bounds from higher moments and the Chernoff bound. Next we prove Jensen's inequality for convex functions, and then the Cauchy–Schwarz inequality. A section lists where each is used on the site, and five worked problems close the article. The article assumes the rules of expectation of Expectation, Variance and Covariance. It is the home of these four inequalities for the Probability Theory unit, and later articles cite them from here.

What the proofs use

Every proof below rests on three properties of the expected value $E[\cdot]$. Expectation, Variance and Covariance defines the expected value for every random variable and proves the three properties in that generality:

  • Linearity. $E[aX + bY] = aE[X] + bE[Y]$ for constants $a$, $b$ and variables with finite means.
  • Monotonicity. If $X \ge 0$ with probability 1 then $E[X] \ge 0$. With linearity, $X \le Y$ with probability 1 implies $E[X] \le E[Y]$.
  • Indicators. The indicator $\mathbf{1}_A$ of an event $A$ is the random variable equal to 1 when $A$ occurs and 0 when it does not, and $E[\mathbf{1}_A] = P(A)$.

Nothing else about the distribution enters, so the results hold for every random variable, discrete, continuous or neither. That is also why no measure theory is needed here; Random Variables and Distribution Functions says what it would add.

The equality cases need one more fact, which is itself a first use of the method.

Lemma (a nonnegative variable with mean zero). If $Y \ge 0$ with probability 1 and $E[Y] = 0$, then $P(Y = 0) = 1$.

Proof. Fix a whole number $k \ge 1$. Since $Y \ge 0$, the variable $Y - \frac{1}{k}\mathbf{1}\{Y \ge 1/k\}$ is never negative: when $Y \ge 1/k$ it equals $Y - 1/k \ge 0$, and otherwise it equals $Y \ge 0$. By monotonicity and linearity, $0 \le E[Y] - \frac{1}{k}P(Y \ge 1/k)$, so $P(Y \ge 1/k) \le kE[Y] = 0$. The event $\{Y > 0\}$ is the union of the disjoint events $\{Y \ge 1\}$ and $\{1/(k+1) \le Y < 1/k\}$ for $k = 1, 2, \ldots$. Each is contained in an event $\{Y \ge 1/(k+1)\}$ of probability 0. Probability is countably additive, so $P(Y > 0)$ is a sum of zeros. $\square$

Markov's inequality

The idea in one sentence: a nonnegative variable cannot be large often without its mean being large, because every outcome with $X \ge a$ contributes at least $a$ to the average.

Theorem (Markov's inequality). Let $X$ be a random variable with $X \ge 0$ with probability 1 and a finite mean. For every $a > 0$,

$$P(X \ge a) \le \frac{E[X]}{a}.$$

Proof. The variable $X - a\,\mathbf{1}\{X \ge a\}$ is never negative. When $X \ge a$ it equals $X - a \ge 0$, and when $X < a$ it equals $X \ge 0$. By monotonicity its expected value is not negative, and by linearity and the indicator property that expected value is $E[X] - a\,P(X \ge a)$. Dividing by $a$ gives the inequality. $\square$

For the blood pressures, $P(X \ge 180) \le 126.36/180 = 0.7020$. The bound uses the mean and the fact that a pressure is not negative, nothing more.

Example (the bound is weak, and it cannot be improved). A bound of 0.7020 against a true share of 0.0181 looks useless, and yet no smaller number is a valid bound from the same information. Let $X$ be 180 with probability $126.36/180 = 0.7020$ and 0 otherwise. This variable is nonnegative with mean $180 \times 126.36/180 = 126.36$, and $P(X \ge 180) = 0.7020$ exactly. Any guarantee that uses only "nonnegative with mean 126.36" must allow for this variable.

Proposition (equality in Markov's inequality). Under the hypotheses of the theorem, $P(X \ge a) = E[X]/a$ if and only if $P(X = 0 \text{ or } X = a) = 1$.

Proof. Equality says that the nonnegative variable $X - a\,\mathbf{1}\{X \ge a\}$ has expected value 0. By the lemma that happens exactly when $X = a\,\mathbf{1}\{X \ge a\}$ with probability 1. The right side takes only the values 0 and $a$. So equality forces $X$ to be 0 or $a$ with probability 1. Conversely, if $X$ takes only the values 0 and $a$, then $E[X] = a\,P(X = a) = a\,P(X \ge a)$. $\square$

So Markov's inequality is attained only by a variable that puts all its probability at 0 and at the threshold. The blood pressures are nothing like that, which is where the slack comes from.

The general form

The same proof bounds $P(X \ge a)$ by the expected value of any increasing function, and the choice of function is where all the later inequalities come from.

Theorem (Markov's inequality, general form). Let $X$ be any random variable and $g$ a nonnegative, nondecreasing function on an interval that contains the values of $X$ and the number $a$, with $g(a) > 0$ and $E[g(X)]$ finite. Then

$$P(X \ge a) \le \frac{E[g(X)]}{g(a)}.$$

Proof. When $X \ge a$, $g(X) \ge g(a)$ because $g$ is nondecreasing. So the variable $g(X) - g(a)\,\mathbf{1}\{X \ge a\}$ is at least $g(X) - g(X) = 0$ when $X \ge a$, and equals $g(X) \ge 0$ otherwise. Its expected value $E[g(X)] - g(a)\,P(X \ge a)$ is therefore not negative. $\square$

Durrett (2019, Theorem 1.6.4) states the result in this general form and calls it Chebyshev's inequality, with the remark that some authors call it Markov's inequality and keep Chebyshev's name for the case of the square. This article follows those authors. The choices of $g$ used below are:

Applied to Function $g$ Result
$X \ge 0$ $g(x) = x$ Markov's inequality
$\lvert X - \mu\rvert$ $g(x) = x^2$ Chebyshev's inequality
$\lvert X - \mu\rvert$ $g(x) = x^r$, $r > 0$ the moment bound of order $r$
$X$ $g(x) = e^{tx}$, $t > 0$ the Chernoff bound

Chebyshev's inequality

Markov's inequality knows only the mean. The variance says how far from the mean the values tend to lie, and it gives a bound on both tails at once.

Theorem (Chebyshev's inequality). Let $X$ have mean $\mu$ and finite variance $\sigma^2$. For every $a > 0$,

$$P(\lvert X - \mu\rvert \ge a) \le \frac{\sigma^2}{a^2}, \qquad\text{and with } a = k\sigma,\ \sigma > 0: \quad P(\lvert X - \mu\rvert \ge k\sigma) \le \frac{1}{k^2}.$$

Proof. Apply the general form of Markov's inequality to the nonnegative variable $\lvert X - \mu\rvert$ with $g(x) = x^2$, which is nondecreasing on $x \ge 0$: $P(\lvert X - \mu\rvert \ge a) \le E[(X - \mu)^2]/a^2 = \sigma^2/a^2$. $\square$

Two course pages hold the special cases. Standard Scores, Chebyshev's Inequality and Grouped Data proves the inequality for a list of $N$ values, as a statement about the share of values at least $k$ standard deviations from their mean. That is the theorem above for the random variable "a value drawn at random from the list", whose mean and variance are the list's mean and its variance with divisor $N$. Moment Generating Functions and Slutsky's Theorem proves it for a random variable by the indicator argument and uses it to prove the weak law of large numbers, which The Laws of Large Numbers takes further.

For a pressure of 180 mm Hg or more the distance from the mean is $a = 180 - 126.36 = 53.64$, and the event $X \ge 180$ is part of the event $\lvert X - \mu\rvert \ge 53.64$. So $P(X \ge 180) \le 405.01/53.64^2 = 0.1408$.

When Chebyshev's inequality is attained

Proposition (equality in Chebyshev's inequality). Let $X$ have mean $\mu$ and variance $\sigma^2 > 0$, and let $a \ge \sigma$. Then $P(\lvert X - \mu\rvert \ge a) = \sigma^2/a^2$ if and only if

$$P(X = \mu - a) = P(X = \mu + a) = \frac{\sigma^2}{2a^2}, \qquad P(X = \mu) = 1 - \frac{\sigma^2}{a^2}.$$

Proof. Chebyshev's inequality is Markov's inequality for the nonnegative variable $W = (X - \mu)^2$ at the threshold $a^2$. By the equality case of Markov's inequality, equality holds if and only if $W$ is 0 or $a^2$ with probability 1, that is, if and only if $X$ takes only the values $\mu - a$, $\mu$ and $\mu + a$. Suppose it does, with probabilities $p_-$, $p_0$ and $p_+$. The mean is $\mu + a(p_+ - p_-)$, and it equals $\mu$ only if $p_+ = p_-$. The variance is then $a^2(p_+ + p_-) = 2a^2p_+$, and it equals $\sigma^2$ only if $p_+ = \sigma^2/(2a^2)$. So the distribution is the one displayed, and $p_0 = 1 - \sigma^2/a^2$, which is a probability because $a \ge \sigma$. Conversely the displayed distribution has mean $\mu$ and variance $\sigma^2$ by the same two computations, and $P(\lvert X - \mu\rvert \ge a) = p_- + p_+ = \sigma^2/a^2$. $\square$

For $a < \sigma$ the bound exceeds 1 and says nothing. For every $a \ge \sigma$ it is attained by exactly one distribution, with three values. Grinstead and Snell (1997, §8.1) give the case $a = \sigma$, a variable equal to $\pm a$ with probability $\frac12$ each.

Example (where the slack is). For the blood pressures the share at least two standard deviations from the mean is 0.0463, against the bound of 0.25. The proof shows what was given away. It replaces $(X - \mu)^2$ by 0 where $\lvert X - \mu\rvert < a$ and by $a^2$ where $\lvert X - \mu\rvert \ge a$. Keeping the second replacement and not the first gives the sharper statement

$$P(\lvert X - \mu\rvert \ge a) \le \frac{E\big[(X - \mu)^2\,\mathbf{1}\{\lvert X - \mu\rvert \ge a\}\big]}{a^2},$$

proved by the same indicator argument: the variable $(X - \mu)^2\,\mathbf{1}\{\lvert X - \mu\rvert \ge a\} - a^2\,\mathbf{1}\{\lvert X - \mu\rvert \ge a\}$ is never negative. For the blood pressures at $a = 2\sigma$ the right side is 0.0895. So of the gap between 0.25 and 0.0463, the larger part comes from the variance contributed by the values within two standard deviations, which Chebyshev's inequality counts as if it lay outside. The rest comes from values beyond $2\sigma$ that lie well beyond it.

One tail: the one-sided inequality

Chebyshev's inequality bounds both tails together. When only one tail is wanted, a shift before squaring gives a better bound, and one that is never above 1.

Theorem (one-sided Chebyshev inequality). Let $X$ have mean $\mu$ and finite variance $\sigma^2$. For every $a > 0$,

$$P(X - \mu \ge a) \le \frac{\sigma^2}{\sigma^2 + a^2}, \qquad P(X - \mu \le -a) \le \frac{\sigma^2}{\sigma^2 + a^2}.$$

The idea: add a constant $t$ to $X - \mu$ before squaring, bound the probability for each $t$, and choose the best $t$.

Proof. Fix $t \ge 0$. When $X - \mu \ge a$, the number $X - \mu + t$ is at least $a + t > 0$, so $(X - \mu + t)^2 \ge (a + t)^2$. The indicator argument, with the nonnegative variable $(X - \mu + t)^2 - (a + t)^2\,\mathbf{1}\{X - \mu \ge a\}$, gives

$$P(X - \mu \ge a) \le \frac{E\big[(X - \mu + t)^2\big]}{(a + t)^2} = \frac{\sigma^2 + t^2}{(a + t)^2},$$

because $E[(X - \mu + t)^2] = E[(X - \mu)^2] + 2tE[X - \mu] + t^2 = \sigma^2 + t^2$. This holds for every $t \ge 0$. Take $t = \sigma^2/a$. Then $\sigma^2 + t^2 = \sigma^2(a^2 + \sigma^2)/a^2$ and $(a + t)^2 = (a^2 + \sigma^2)^2/a^2$, and the quotient is $\sigma^2/(a^2 + \sigma^2)$. (This $t$ is the best choice: the derivative of $(\sigma^2 + t^2)/(a + t)^2$ with respect to $t$ is $2(at - \sigma^2)/(a + t)^3$, negative for $t < \sigma^2/a$ and positive after.) The second inequality is the first applied to $-X$, which has mean $-\mu$ and the same variance. $\square$

The bound is attained. Let $X - \mu$ be $a$ with probability $\sigma^2/(\sigma^2 + a^2)$ and $-\sigma^2/a$ with probability $a^2/(\sigma^2 + a^2)$. Its mean is $\big(a\sigma^2 - \sigma^2 a\big)/(\sigma^2 + a^2) = 0$ and its variance is $\big(a^2\sigma^2 + \sigma^4\big)/(\sigma^2 + a^2) = \sigma^2$. Durrett (2019, Exercise 1.6.4) sets the inequality and its attainment as an exercise.

It is the only distribution that attains the bound. Put $t = \sigma^2/a$. Equality in the theorem means that the nonnegative variable $(X - \mu + t)^2 - (a + t)^2\,\mathbf{1}\{X - \mu \ge a\}$ of the proof has mean 0, so by the lemma it is 0 with probability 1. Where $X - \mu < a$ this says $(X - \mu + t)^2 = 0$, that is, $X - \mu = -t = -\sigma^2/a$. Where $X - \mu \ge a$ it says $(X - \mu + t)^2 = (a + t)^2$, and since $X - \mu + t \ge a + t > 0$ there, $X - \mu = a$. So $X - \mu$ takes only the values $a$ and $-\sigma^2/a$, and the requirement that its mean be 0 fixes the two probabilities at the values above.

For a pressure of 180 or more, $P(X \ge 180) \le 405.01/(405.01 + 53.64^2) = 0.1234$, the second number of the opening.

Example (the median is within one standard deviation of the mean). For every distribution with a finite variance, each median $m$ satisfies $\lvert m - \mu\rvert \le \sigma$. A median is a number $m$ with $P(X \ge m) \ge \frac12$ and $P(X \le m) \ge \frac12$. Suppose $m > \mu + \sigma$, and put $a = m - \mu > \sigma$. The one-sided inequality gives $P(X \ge m) \le \sigma^2/(\sigma^2 + a^2) < \frac12$, a contradiction. The case $m < \mu - \sigma$ contradicts the second inequality in the same way, with $a = \mu - m > \sigma$ and the event $X \le m$. For the blood pressures the median is 123.33 and the mean 126.36, a gap of 3.03 mm Hg against $\sigma = 20.12$.

Higher moments

If $E[\lvert X - \mu\rvert^r]$ is finite for some $r > 0$, the general form with $g(x) = x^r$ on $x \ge 0$ gives

$$P(\lvert X - \mu\rvert \ge a) \le \frac{E\big[\lvert X - \mu\rvert^r\big]}{a^r}.$$

A higher power makes the bound fall faster as $a$ grows, at the cost of a larger numerator. For the blood pressures the fourth central moment is $4.680\,\sigma^4$, so the fourth-moment bound at $k$ standard deviations is $4.680/k^4$. It is better than Chebyshev's $1/k^2$ exactly when $k^2 > 4.680$, that is, beyond $k = 2.163$ standard deviations. The proof of the strong law of large numbers in The Laws of Large Numbers uses the fourth-moment bound for that reason: applied to a sample mean it falls like $1/n^2$, where Chebyshev's falls like $1/n$.

The Chernoff bound

The powers $x^r$ grow polynomially. An exponential grows faster than any of them, and using it turns a bound on a sum of independent variables into a product.

Theorem (Chernoff bound). Let $X$ be a random variable whose moment generating function $M_X(t) = E[e^{tX}]$ is finite at some $t > 0$. Then for every real $a$,

$$P(X \ge a) \le e^{-ta}\,M_X(t).$$

Proof. The function $g(x) = e^{tx}$ is positive and increasing because $t > 0$. The general form of Markov's inequality gives $P(X \ge a) \le E[e^{tX}]/e^{ta}$. $\square$

The bound holds for each $t > 0$ at which $M_X$ is finite, so the smallest value over those $t$ is also a bound. The name is after Chernoff (1952). His Theorem 1 (p. 495) is the bound for a sum $S_n$ of $n$ independent observations that the example below derives: $P(S_n \ge na) \le [m(a)]^n$ for $a \ge E[X]$, where $m(a)$ is the smallest value of $E[e^{t(X - a)}] = e^{-ta}M_X(t)$ over real $t$. The same theorem adds that the base cannot be improved: for every $\varepsilon$ between 0 and $m(a)$, $P(S_n \ge na)$ exceeds $(m(a) - \varepsilon)^n$ for all large $n$. Moment Generating and Characteristic Functions is the home of $M_X$: which distributions have one, and how to compute it.

A draw from a finite population is bounded, so its moment generating function is a finite sum, finite for every $t$. For the blood pressures, $M_X(t) = \frac{1}{5{,}247}\sum_i e^{tx_i}$, and minimizing $e^{-180t}M_X(t)$ over $t$ numerically gives $P(X \ge 180) \le 0.1334$, at $t = 0.0628$. The table collects the bounds at four thresholds.

Threshold $a$ (mm Hg) 160 180 200 220
Standard deviations above the mean 1.672 2.665 3.659 4.653
Markov, $\mu/a$ 0.7897 0.7020 0.6318 0.5744
Chebyshev, $\sigma^2/(a - \mu)^2$ 0.3579 0.1408 0.0747 0.0462
One-sided Chebyshev 0.2636 0.1234 0.0695 0.0441
Fourth moment 0.5994 0.0927 0.0261 0.0100
Chernoff, best $t$ 0.3992 0.1334 0.0322 0.0050
True share with $X \ge a$ 0.0665 0.0181 0.0048 0.0010

No bound is best everywhere. At 160 the one-sided inequality is the smallest, at 180 and 200 the fourth-moment bound, and at 220 the Chernoff bound. Every bound is well above the truth at every threshold, and the smallest bound at each threshold is between 3.96 and 5.48 times the true share. The table is a computation on one population, and the ranking would differ for another.

The share of the 5,247 adults with systolic pressure at or above each threshold (lowest curve) and five bounds on it, on a logarithmic scale. Notice that the bounds cross: the one-sided Chebyshev bound is lowest up to about 174 mm Hg, the fourth-moment bound from there to about 207, and the Chernoff bound beyond (dotted vertical lines mark the two crossings). Every bound stays above the true share at every threshold.
The share of the 5,247 adults with systolic pressure at or above each threshold (lowest curve) and five bounds on it, on a logarithmic scale. Notice that the bounds cross: the one-sided Chebyshev bound is lowest up to about 174 mm Hg, the fourth-moment bound from there to about 207, and the Chernoff bound beyond (dotted vertical lines mark the two crossings). Every bound stays above the true share at every threshold.

Example (a sample mean: where the exponential bound pays). The Chernoff bound comes into its own for sums. Let $\bar{X}_n$ be the mean of $n$ independent draws with the moment generating function $M_X$. The event $\bar{X}_n \ge a$ is the event $S_n \ge na$ for the sum $S_n = X_1 + \cdots + X_n$, and the moment generating function of a sum of independent variables is the product of theirs (Moment Generating and Characteristic Functions), so

$$P(\bar{X}_n \ge a) \le e^{-tna}\,M_X(t)^n = \big(e^{-ta}M_X(t)\big)^n.$$

The bound is the $n$th power of the bound for one draw, so it falls geometrically in $n$ whenever some $t$ makes $e^{-ta}M_X(t) < 1$. For the mean of $n = 400$ blood pressures and $a = \mu + 4$, the best $t$ gives $e^{-ta}M_X(t) = 0.98168$ and a bound of $0.98168^{400} = 0.00061$. Chebyshev's inequality, with the variance $\sigma^2/400$ of the mean, gives $405.01/(400 \times 16) = 0.0633$ for a miss of 4 mm Hg or more in either direction, and the one-sided inequality gives 0.0595 for the upper tail. Concentration Inequalities: Hoeffding and Bernstein develops this use.

Jensen's inequality

The inequalities so far bound probabilities. Jensen's inequality compares two expected values: that of a function of $X$, and the function of the mean. The blood pressures give an instance. Their mean is 126.36 mm Hg, and the mean of their squares is 16,371.81, which is more than $126.36^2 = 15{,}966.81$. The difference is the variance, 405.01, so here the direction is no surprise. Jensen's inequality says the same direction holds for every convex function. It is named for Jensen (1906), who proved it for a continuous convex function and a weighted mean of finitely many numbers with positive weights (his formula (5), p. 180), which is the expected value of a variable with finitely many values, and then for integrals (formula (5′), p. 186).

A function $\varphi$ on an interval $I$ is convex if its graph lies on or below each of its chords: for all $x, y$ in $I$ and $0 < \lambda < 1$,

$$\varphi\big(\lambda x + (1 - \lambda)y\big) \le \lambda\,\varphi(x) + (1 - \lambda)\,\varphi(y).$$

It is strictly convex if the inequality is strict whenever $x \ne y$, and concave if $-\varphi$ is convex. (In this section $\varphi$ is a convex function, not a characteristic function.)

Theorem (Jensen's inequality). Let $\varphi$ be a convex function on an interval $I$, and let $X$ be a random variable with $P(X \in I) = 1$, a finite mean $\mu$, and $E[\lvert\varphi(X)\rvert]$ finite. Then $\mu$ lies in $I$ and

$$\varphi\big(E[X]\big) \le E\big[\varphi(X)\big].$$

If $\varphi$ is strictly convex, equality holds if and only if $P(X = \mu) = 1$.

The idea: a convex function lies above a straight line that touches its graph at $\mu$. The expected value of a straight-line function of $X$ is the line's value at $\mu$, by linearity, and that is $\varphi(\mu)$.

The line is called a supporting line, and its existence is the one piece of analysis the proof needs.

Lemma (supporting line). Let $\varphi$ be convex on an interval $I$ and let $m$ be a point of $I$ that is not an endpoint. Then there is a number $c$ such that $\varphi(x) \ge \varphi(m) + c\,(x - m)$ for every $x$ in $I$.

Proof. Take $x < m < y$ in $I$. Then $m = \lambda x + (1 - \lambda)y$ with $\lambda = (y - m)/(y - x)$, which lies between 0 and 1. Convexity gives $\varphi(m) \le \lambda\varphi(x) + (1 - \lambda)\varphi(y)$. Multiply by $y - x > 0$:

$$(y - x)\,\varphi(m) \le (y - m)\,\varphi(x) + (m - x)\,\varphi(y).$$

Write $y - x = (y - m) + (m - x)$ on the left and collect terms: $(y - m)\big(\varphi(m) - \varphi(x)\big) \le (m - x)\big(\varphi(y) - \varphi(m)\big)$. Dividing by the positive number $(y - m)(m - x)$,

$$\frac{\varphi(m) - \varphi(x)}{m - x} \le \frac{\varphi(y) - \varphi(m)}{y - m}.$$

So every slope of a chord ending at $m$ from the left is at most every slope of a chord starting at $m$ to the right. Both kinds of chord exist because $m$ is not an endpoint. The left slopes are therefore bounded above, and we let $c$ be their least upper bound. Then $c$ is at most every right slope, which says $\varphi(y) \ge \varphi(m) + c\,(y - m)$ for $y > m$. And every left slope is at most $c$, so for $x < m$, $\varphi(m) - \varphi(x) \le c\,(m - x)$, which is $\varphi(x) \ge \varphi(m) + c\,(x - m)$. At $x = m$ both sides are $\varphi(m)$. $\square$

Proof of the theorem. The mean lies in $I$. Suppose $I$ has a left endpoint $\ell$. Then $X \ge \ell$ with probability 1, so $\mu \ge \ell$ by monotonicity. If $\mu = \ell$, the nonnegative variable $X - \ell$ has mean 0, so $X = \ell$ with probability 1 by the lemma on nonnegative variables. That requires $\ell$ to be in $I$, and then $\varphi(X) = \varphi(\mu)$ with probability 1 and the inequality holds with equality. The same reasoning applies to a right endpoint, with $r - X$ for the endpoint $r$. In every other case $\mu$ lies in $I$ and is not an endpoint.

The inequality. Let $c$ be the slope of a supporting line at $m = \mu$. With probability 1,

$$\varphi(X) \ge \varphi(\mu) + c\,(X - \mu).$$

Take expected values. By monotonicity and linearity, $E[\varphi(X)] \ge \varphi(\mu) + c\,(E[X] - \mu) = \varphi(\mu)$.

Equality. Let $\varphi$ be strictly convex. If $P(X = \mu) = 1$ both sides equal $\varphi(\mu)$. Conversely, suppose $E[\varphi(X)] = \varphi(\mu)$. If $\mu$ is an endpoint of $I$ we have already seen that $X = \mu$ with probability 1. Otherwise the variable $D = \varphi(X) - \varphi(\mu) - c\,(X - \mu)$ is nonnegative with mean 0, so $D = 0$ with probability 1. It remains to show that $D = 0$ only where $X = \mu$, that is, that the supporting line touches the graph of a strictly convex function only at $\mu$. Suppose $\varphi(x) = \varphi(\mu) + c\,(x - \mu)$ at some $x \ne \mu$, and let $z = (x + \mu)/2$. Strict convexity gives $\varphi(z) < \frac12\varphi(x) + \frac12\varphi(\mu) = \varphi(\mu) + c\,(z - \mu)$, which contradicts the supporting line inequality at $z$. So $X = \mu$ with probability 1. $\square$

For a concave function the inequality reverses, $E[\varphi(X)] \le \varphi(E[X])$, by applying the theorem to $-\varphi$.

Example (which way the inequality goes). The claim "the average of a function is the function of the average" is true for straight-line functions only, and the two-point case shows the direction for the rest. Let $X$ be $x$ with probability $\lambda$ and $y$ with probability $1 - \lambda$. Then $E[\varphi(X)] = \lambda\varphi(x) + (1 - \lambda)\varphi(y)$ is a point on the chord, $\varphi(E[X])$ is the point on the graph below it, and Jensen's inequality for this $X$ is the definition of convexity. The theorem extends the definition from two points to any distribution.

Recognizing convex functions

To apply the theorem we need to know that a function is convex. The usual test is the second derivative.

Lemma (second derivative test). If $\varphi$ is continuous on an interval $I$ and twice differentiable inside it with $\varphi'' \ge 0$ there, then $\varphi$ is convex on $I$. If $\varphi'' > 0$ inside $I$, then $\varphi$ is strictly convex.

Proof. Take $x < y$ in $I$ and $m = \lambda x + (1 - \lambda)y$ with $0 < \lambda < 1$. By the mean value theorem there are points $\xi_1$ in $(x, m)$ and $\xi_2$ in $(m, y)$ with

$$\frac{\varphi(m) - \varphi(x)}{m - x} = \varphi'(\xi_1), \qquad \frac{\varphi(y) - \varphi(m)}{y - m} = \varphi'(\xi_2).$$

Since $\varphi'' \ge 0$, $\varphi'$ is nondecreasing, and $\xi_1 < \xi_2$, so the first slope is at most the second. Multiplying out as in the proof of the supporting line lemma, in reverse, that slope inequality is $(y - x)\varphi(m) \le (y - m)\varphi(x) + (m - x)\varphi(y)$, and dividing by $y - x$ gives $\varphi(m) \le \lambda\varphi(x) + (1 - \lambda)\varphi(y)$ because $m - x = (1 - \lambda)(y - x)$ and $y - m = \lambda(y - x)$. If $\varphi'' > 0$ then $\varphi'$ is increasing and every inequality is strict. $\square$

So $x^2$, $e^{tx}$, $-\ln x$ on $x > 0$, $1/x$ on $x > 0$, and $x^p$ on $x \ge 0$ for $p > 1$ are strictly convex, and $\ln x$ and $\sqrt{x}$ are strictly concave. The function $\lvert x\rvert$ is convex without being differentiable at 0: $\lvert\lambda x + (1 - \lambda)y\rvert \le \lambda\lvert x\rvert + (1 - \lambda)\lvert y\rvert$ is the triangle inequality.

Consequences

Each line below is Jensen's inequality for one function, for a variable $X$ with the stated expected values finite.

  • $\varphi(x) = x^2$: $(E[X])^2 \le E[X^2]$, which says that a variance is not negative. Equality only for a constant.
  • $\varphi(x) = \lvert x\rvert$: $\lvert E[X]\rvert \le E[\lvert X\rvert]$.
  • $\varphi(x) = \ln x$, concave, $X > 0$: $E[\ln X] \le \ln E[X]$. Exponentiating, $e^{E[\ln X]} \le E[X]$: the geometric mean is at most the arithmetic mean.
  • $\varphi(x) = 1/x$, $X > 0$: $1/E[X] \le E[1/X]$, so the harmonic mean $1/E[1/X]$ is at most the arithmetic mean.
  • $\varphi(x) = \sqrt{x}$, concave, applied to the sample variance $s_n^2$: $E[s_n] \le \sqrt{E[s_n^2]}$. When $s_n^2$ is unbiased for $\sigma^2$, the sample standard deviation $s_n$ is therefore biased downward, $E[s_n] < \sigma$, unless $s_n$ is constant.

For the blood pressures the geometric mean is 124.87 mm Hg and the harmonic mean 123.47, both below the arithmetic mean 126.36. Problem 3 computes them and the bias of $s_n$ for samples of two.

The finite form. Let $w_1, \ldots, w_K$ be nonnegative weights with sum 1 and $a_1, \ldots, a_K$ points of $I$. The random variable equal to $a_k$ with probability $w_k$ has mean $\sum_k w_ka_k$, and Jensen's inequality for it reads

$$\varphi\Big(\sum_{k=1}^K w_ka_k\Big) \le \sum_{k=1}^K w_k\,\varphi(a_k).$$

With the concave logarithm this is $\ln\sum_k w_ka_k \ge \sum_k w_k\ln a_k$, the form in which the site's derivations of the expectation-maximization algorithm use it.

Theorem (moments of higher order dominate). For $0 < r < s$ and any random variable $X$ with $E[\lvert X\rvert^s]$ finite,

$$\big(E[\lvert X\rvert^r]\big)^{1/r} \le \big(E[\lvert X\rvert^s]\big)^{1/s}.$$

Proof. Put $U = \lvert X\rvert^r \ge 0$ and $\varphi(u) = u^{s/r}$ on $u \ge 0$, which is convex by the second derivative test because $s/r > 1$. The mean of $U$ is finite because $\lvert x\rvert^r \le 1 + \lvert x\rvert^s$ for every $x$, and $\varphi(U) = \lvert X\rvert^s$ has a finite mean by hypothesis. Jensen's inequality gives $\big(E[\lvert X\rvert^r]\big)^{s/r} \le E[\lvert X\rvert^s]$. Taking the power $1/s$ of both sides gives the claim. $\square$

Three consequences are used later. A finite moment of order $s$ implies finite moments of every lower order, so a finite variance implies a finite mean. With $r = 1$, $s = 2$ applied to $X - \mu$: the mean absolute deviation $E[\lvert X - \mu\rvert]$ is at most $\sigma$. With $r = 2$, $s = 4$: $\sigma^4 \le E[(X - \mu)^4]$, which the proof of the strong law uses. For the blood pressures the values of $\big(E[\lvert X - \mu\rvert^r]\big)^{1/r}$ at $r = 1, 2, 3, 4$ are 15.47, 20.12, 24.79 and 29.60 mm Hg, increasing as the theorem requires.

The Cauchy–Schwarz inequality

The last inequality concerns two variables. For the 5,247 adults let $X$ be the recorded age in years and $Y$ the systolic pressure of the adult drawn. The mean of the product is $E[XY] = 6{,}483.33$. How large could it have been, given only the separate distributions of age and pressure? The Cauchy–Schwarz inequality answers: at most $\sqrt{E[X^2]\,E[Y^2]} = \sqrt{2{,}832.28 \times 16{,}371.81} = 6{,}809.52$.

Theorem (Cauchy–Schwarz inequality). Let $X$ and $Y$ be random variables with $E[X^2]$ and $E[Y^2]$ finite. Then $E[\lvert XY\rvert]$ is finite and

$$\big(E[XY]\big)^2 \le E[X^2]\,E[Y^2].$$

Equality holds if and only if there are constants $\alpha$ and $\beta$, not both zero, with $P(\alpha X + \beta Y = 0) = 1$.

The idea: the expected value of a square is not negative. Apply that to $Y - tX$, the part of $Y$ left over after subtracting a multiple of $X$, with the multiple chosen to make the leftover as small as possible.

Proof. Since $(\lvert X\rvert - \lvert Y\rvert)^2 \ge 0$, we have $\lvert XY\rvert \le \frac12(X^2 + Y^2)$, so $E[\lvert XY\rvert] \le \frac12\big(E[X^2] + E[Y^2]\big)$ is finite.

Suppose first that $E[X^2] = 0$. Then $X = 0$ with probability 1 by the lemma on nonnegative variables, so $XY = 0$ with probability 1, both sides of the inequality are 0, and $\alpha = 1$, $\beta = 0$ gives $\alpha X + \beta Y = 0$ with probability 1.

Suppose now that $E[X^2] > 0$. For every real $t$, by monotonicity and linearity,

$$0 \le E\big[(Y - tX)^2\big] = E[Y^2] - 2t\,E[XY] + t^2\,E[X^2].$$

Take $t = t_0 = E[XY]/E[X^2]$, the value that minimizes the right side. It becomes $E[Y^2] - (E[XY])^2/E[X^2]$, and multiplying $0 \le E[Y^2] - (E[XY])^2/E[X^2]$ by $E[X^2]$ gives the inequality.

Equality. Still with $E[X^2] > 0$: equality holds exactly when $E[(Y - t_0X)^2] = 0$, and by the lemma that is exactly when $Y = t_0X$ with probability 1, a relation of the stated form with $\alpha = t_0$, $\beta = -1$. Conversely, suppose $\alpha X + \beta Y = 0$ with probability 1 and $\alpha$, $\beta$ are not both 0. If $\beta \ne 0$ then $Y = \gamma X$ with $\gamma = -\alpha/\beta$, and both sides of the inequality equal $\gamma^2(E[X^2])^2$. If $\beta = 0$ then $\alpha \ne 0$ and $X = 0$ with probability 1, which contradicts $E[X^2] > 0$. $\square$

For age and pressure the two sides are $6{,}483.33^2$ and $6{,}809.52^2$, in the ratio 0.9065. The ratio is the square of the cosine of the angle between the two columns of data regarded as vectors, and it is high mostly because both variables are positive. Removing the means changes the picture, and gives the most familiar corollary.

Corollary (covariance and correlation). If $X$ and $Y$ have finite variances $\sigma_X^2$ and $\sigma_Y^2$, then $\lvert\operatorname{Cov}(X, Y)\rvert \le \sigma_X\sigma_Y$. If both variances are positive, the correlation $\rho = \operatorname{Cov}(X, Y)/(\sigma_X\sigma_Y)$ satisfies $-1 \le \rho \le 1$, with $\lvert\rho\rvert = 1$ if and only if $Y = \mu_Y + \gamma\,(X - \mu_X)$ with probability 1 for some constant $\gamma \ne 0$.

Proof. Apply the theorem to $X - \mu_X$ and $Y - \mu_Y$: $\operatorname{Cov}(X, Y)^2 = \big(E[(X - \mu_X)(Y - \mu_Y)]\big)^2 \le \sigma_X^2\sigma_Y^2$. In the equality case with $\sigma_X^2 > 0$, the proof gives $Y - \mu_Y = t_0(X - \mu_X)$ with probability 1, and $t_0 \ne 0$ because $\sigma_Y^2 > 0$. Conversely such a relation gives $\operatorname{Cov}(X, Y) = \gamma\sigma_X^2$ and $\sigma_Y = \lvert\gamma\rvert\sigma_X$, so $\lvert\rho\rvert = 1$. $\square$

Expectation, Variance and Covariance proves the bound on $\rho$ by another route, the variance of a sum of standardized variables, and treats correlation in full. For age and pressure, $\rho = 0.4866$ and $\rho^2 = 0.2368$, far from the uncentered ratio 0.9065.

Corollary (sums of numbers). For real numbers $a_1, \ldots, a_n$ and $b_1, \ldots, b_n$,

$$\Big(\sum_{i=1}^n a_ib_i\Big)^2 \le \Big(\sum_{i=1}^n a_i^2\Big)\Big(\sum_{i=1}^n b_i^2\Big),$$

with equality if and only if one of the two lists is a constant multiple of the other.

Proof. Let $J$ be an index drawn uniformly from $1, \ldots, n$, and put $X = a_J$ and $Y = b_J$. Then $E[XY] = \frac1n\sum a_ib_i$, $E[X^2] = \frac1n\sum a_i^2$ and $E[Y^2] = \frac1n\sum b_i^2$, and the theorem, multiplied by $n^2$, is the claim. Every index has positive probability, so "$\alpha X + \beta Y = 0$ with probability 1" means $\alpha a_i + \beta b_i = 0$ for every $i$. $\square$

In vector language this is $\lvert\vec{a}\cdot\vec{b}\rvert \le \lVert\vec{a}\rVert\,\lVert\vec{b}\rVert$, the reason a cosine similarity lies between $-1$ and 1.

Corollary (the standard deviation of a sum). If $X$ and $Y$ have finite variances, then $\sigma_{X+Y} \le \sigma_X + \sigma_Y$.

Proof. $\operatorname{Var}(X + Y) = \sigma_X^2 + 2\operatorname{Cov}(X, Y) + \sigma_Y^2 \le \sigma_X^2 + 2\sigma_X\sigma_Y + \sigma_Y^2 = (\sigma_X + \sigma_Y)^2$, by the first corollary. $\square$

Corollary (a lower bound from two moments). If $Y \ge 0$ with probability 1 and $0 < E[Y^2] < \infty$, then

$$P(Y > 0) \ge \frac{(E[Y])^2}{E[Y^2]}.$$

Proof. $Y = Y\,\mathbf{1}\{Y > 0\}$, so by the theorem applied to $Y$ and the indicator, $(E[Y])^2 \le E[Y^2]\,E[\mathbf{1}\{Y > 0\}^2] = E[Y^2]\,P(Y > 0)$. $\square$

This is the one result of the article that bounds a probability from below (Durrett, 2019, Exercise 1.6.6, sets it as an exercise). Problem 4 applies it.

Where the site uses these inequalities

The uses below are the reason this article exists. The list was made by searching the site's pages in November 2026.

What is proved here, what is cited, what is computed

Proved here, in full: Markov's inequality, its general form and its equality case; Chebyshev's inequality, its equality case, the sharper form with a truncated second moment, and the one-sided inequality and the only distribution that attains it; the bound on the distance between median and mean; the moment bound of order $r$; the Chernoff bound; the supporting line lemma, Jensen's inequality and its equality case; the second derivative test for convexity; the monotonicity of $\big(E[\lvert X\rvert^r]\big)^{1/r}$; the Cauchy–Schwarz inequality, its equality case and four corollaries. The proofs use only linearity, monotonicity, the indicator property and the countable additivity of probability.

Taken from other pages: the three properties of expected values, from Expectation, Variance and Covariance, and the multiplication rule for moment generating functions of independent variables, from Moment Generating and Characteristic Functions. From calculus: the mean value theorem and the existence of a least upper bound.

Cited: nothing is cited for a proof. Durrett (2019) and Grinstead and Snell (1997) are cited for the places where the same results appear, and Jensen (1906) and Chernoff (1952) for the names and for the forms in which those two papers state the results.

Computed: every number about the blood pressures is an exact property of the 5,247 values, computed by the script named at the end. No sampling is involved. The comparisons between bounds (which is smallest where) are facts about this population and are not general.

Worked problems

These problems were written for this article; none comes from a textbook. They are posed on the article's blood pressure population, on the ages of the same adults, and on the obesity count of The Binomial Distribution. The script tools/experiments/statistics/probability-inequalities.py in the site's repository computes every number and asserts it. Decimals are rounded from unrounded values.

Problem 1 (the pressure each bound certifies as rare). For the blood pressure population, with $\mu = 126.36$ and $\sigma^2 = 405.01$, find for each of Markov's inequality, Chebyshev's inequality and the one-sided inequality the smallest threshold $a$ at which the bound guarantees $P(X \ge a) \le 0.05$. Compare with the smallest threshold at which the true share is at most 0.05.

Solution. Markov. The bound $\mu/a$ is at most 0.05 when $a \ge \mu/0.05 = 126.36/0.05 = 2{,}527.20$ mm Hg. No blood pressure is anywhere near that, so from the mean alone nothing useful can be certified at the 5% level.

Chebyshev. For $a > \mu$ the event $X \ge a$ is contained in $\lvert X - \mu\rvert \ge a - \mu$, so $P(X \ge a) \le \sigma^2/(a - \mu)^2$. This is at most 0.05 when $(a - \mu)^2 \ge 20\sigma^2$, that is, when $a \ge \mu + \sqrt{20}\,\sigma = 126.36 + 4.4721 \times 20.1247 = 216.36$ mm Hg.

One-sided. The bound $\sigma^2/(\sigma^2 + (a - \mu)^2)$ is at most 0.05 when $\sigma^2 + (a - \mu)^2 \ge 20\sigma^2$, that is, when $(a - \mu)^2 \ge 19\sigma^2$, so $a \ge \mu + \sqrt{19}\,\sigma = 126.36 + 4.3589 \times 20.1247 = 214.08$ mm Hg. The one-sided inequality always gives the lower threshold of the two, because it needs $19\sigma^2$ where Chebyshev's needs $20\sigma^2$.

The truth. Sorting the values, 262 of the 5,247 adults, a share 0.0499, have a pressure of 165 mm Hg or more. The next value below 165 that occurs among the adults is $164\tfrac23$, and 273 adults, a share 0.0520, have a pressure of $164\tfrac23$ or more. The true share is therefore at most 0.05 for every threshold above $164\tfrac23$ mm Hg and exceeds 0.05 at $164\tfrac23$ and below.

The guarantees of 216.36 and 214.08 mm Hg are about 50 mm Hg above the true cut. They are what can be said for every distribution with this mean and variance. For one of them the one-sided threshold is exact: the two-point distribution built as in Problem 2(c) with 214.08 in place of 180. With $a = \sqrt{19}\,\sigma = 87.72$ it takes the value 214.08 with probability $\sigma^2/(\sigma^2 + a^2) = 1/20 = 0.05$ and the value $\mu - \sigma^2/a = 126.36 - 4.62 = 121.74$ with probability 0.95, and it has mean 126.36 and variance 405.01.

Problem 2 (distributions that attain the bounds). Construct (a) a nonnegative variable with mean 126.36 for which Markov's inequality at 180 holds with equality; (b) a variable with mean 126.36 and variance 405.01 for which Chebyshev's inequality at two standard deviations holds with equality; (c) a variable with that mean and variance for which the one-sided inequality at 180 holds with equality. Verify the mean and variance in each case.

Solution. (a) By the equality case of Markov's inequality the variable must take only the values 0 and 180. Let $P(X = 180) = \pi$. The mean is $180\pi = 126.36$, so $\pi = 0.7020$ and $P(X = 0) = 0.2980$. Then $P(X \ge 180) = 0.7020 = \mu/180$.

(b) By the equality case of Chebyshev's inequality with $a = 2\sigma = 40.25$, the variable takes the values $\mu - 2\sigma = 86.11$, $\mu = 126.36$ and $\mu + 2\sigma = 166.61$, with probabilities $\sigma^2/(2a^2) = 1/8$, $1 - 1/4 = 3/4$ and $1/8$. Its mean is $\mu + 2\sigma\big(\tfrac18 - \tfrac18\big) = \mu$. Its variance is $(2\sigma)^2\big(\tfrac18 + \tfrac18\big) = \sigma^2$. And $P(\lvert X - \mu\rvert \ge 2\sigma) = \tfrac18 + \tfrac18 = \tfrac14 = 1/2^2$. The blood pressures themselves have 0.0463 beyond two standard deviations.

(c) Put $a = 180 - \mu = 53.64$. The construction after the one-sided theorem takes $X - \mu = a$ with probability $\pi = \sigma^2/(\sigma^2 + a^2) = 0.1234$, and $X - \mu = -\sigma^2/a = -7.55$ with probability $1 - \pi = 0.8766$. So $X$ is 180 with probability 0.1234 and 118.81 with probability 0.8766. Its mean is $\mu + \big(a\sigma^2 - \sigma^2a\big)/(\sigma^2 + a^2) = \mu$. Its variance is

$$a^2\pi + \frac{\sigma^4}{a^2}(1 - \pi) = \frac{a^2\sigma^2 + \sigma^4}{\sigma^2 + a^2} = \sigma^2.$$

And $P(X \ge 180) = 0.1234$, the one-sided bound. With the unrounded values the script confirms mean 126.36 and variance 405.01 for all three constructions. In each case the extremal distribution is concentrated on two or three points, with one of them exactly at the threshold: a bound is tight only for a distribution built for it.

Problem 3 (three means, and the bias of the sample standard deviation). (a) Compute the arithmetic, geometric and harmonic means and the root mean square of the blood pressures and place them in the order Jensen's inequality requires. (b) Two adults are drawn with replacement, and $s_2$ is the sample standard deviation of their two pressures. Show that $E[s_2^2] = \sigma^2$ and $E[s_2] < \sigma$, and compute $E[s_2]$ exactly.

Solution. (a) Let $X$ be the pressure of one adult drawn at random. All values are positive, between $72\tfrac23$ and 238 mm Hg.

  • Arithmetic mean: $E[X] = 126.36$.
  • Geometric mean: $E[\ln X] = 4.8273$, and $e^{4.8273} = 124.87$. By Jensen's inequality for the concave logarithm, $E[\ln X] \le \ln E[X] = 4.8391$.
  • Harmonic mean: $E[1/X] = 0.008099$, and $1/0.008099 = 123.47$. By Jensen's inequality for the convex function $1/x$, $E[1/X] \ge 1/E[X] = 0.007914$.
  • Root mean square: $\sqrt{E[X^2]} = \sqrt{16{,}371.81} = 127.95$. By Jensen's inequality for $x^2$, $E[X^2] \ge (E[X])^2$.

The geometric mean is also at least the harmonic mean: apply the inequality between the geometric and arithmetic means to the positive variable $1/X$, which gives $e^{-E[\ln X]} \le E[1/X]$, and take reciprocals. So

$$123.47 \;<\; 124.87 \;<\; 126.36 \;<\; 127.95,$$

harmonic, geometric, arithmetic, root mean square. Each inequality is strict because the functions are strictly convex or concave and the pressures are not all equal.

(b) For two values, $\bar{X} = (X_1 + X_2)/2$ and each deviation from it is $\pm(X_1 - X_2)/2$, so

$$s_2^2 = \frac{(X_1 - \bar{X})^2 + (X_2 - \bar{X})^2}{2 - 1} = \frac{(X_1 - X_2)^2}{2}, \qquad s_2 = \frac{\lvert X_1 - X_2\rvert}{\sqrt{2}}.$$

The draws are independent with the same mean, so $E[(X_1 - X_2)^2] = \operatorname{Var}(X_1 - X_2) = 2\sigma^2$, and $E[s_2^2] = \sigma^2$. The square root is strictly concave and $s_2^2$ is not constant, so Jensen's inequality gives $E[s_2] < \sqrt{E[s_2^2]} = \sigma$.

The exact value needs $E[\lvert X_1 - X_2\rvert]$, the average of $\lvert x_i - x_j\rvert$ over all $5{,}247^2$ ordered pairs of adults, a pair of the same adult included. With the values sorted in increasing order, $x_{(1)} \le \cdots \le x_{(N)}$, the value $x_{(i)}$ appears with a plus sign in the pairs where it is the larger (there are $i - 1$ smaller indices) and with a minus sign where it is the smaller ($N - i$ larger indices), in each of the two orders. So

$$E\big[\lvert X_1 - X_2\rvert\big] = \frac{2}{N^2}\sum_{i=1}^N (2i - N - 1)\,x_{(i)} = 21.86 \text{ mm Hg},$$

and $E[s_2] = 21.86/\sqrt{2} = 15.46$ mm Hg, against $\sigma = 20.12$: a ratio of 0.7683. On average the standard deviation of a sample of two understates $\sigma$ by 23%, although its square is exactly unbiased. The script confirms the pair formula against a direct average over all pairs of the distinct values.

Problem 4 (age and pressure). For the 5,247 adults let $X$ be the recorded age and $Y$ the systolic pressure. (a) Verify the Cauchy–Schwarz inequality for $E[XY]$ and for the covariance. (b) What is the largest covariance the two variables could have had with the same variances, and what would the data have to look like? (c) Let $W = \max(Y - 140, 0)$ be the excess of the pressure over 140 mm Hg. Use $E[W] = 3.543$ and $E[W^2] = 109.54$ to bound from below the share of adults with a pressure above 140.

Solution. (a) The script gives $E[X^2] = 2{,}832.28$, $E[Y^2] = 16{,}371.81$ and $E[XY] = 6{,}483.33$. Then $(E[XY])^2 = 4.2034 \times 10^7$ and $E[X^2]E[Y^2] = 4.6370 \times 10^7$, and the first is smaller, in the ratio 0.9065. For the centered variables, $\sigma_X = 18.59$ years, $\sigma_Y = 20.12$ mm Hg and $\operatorname{Cov}(X, Y) = 182.01$, against $\sigma_X\sigma_Y = 374.05$. The ratio is the correlation, $\rho = 0.4866$.

(b) The corollary gives $\operatorname{Cov}(X, Y) \le \sigma_X\sigma_Y = 374.05$, attained if and only if $Y = \mu_Y + \gamma(X - \mu_X)$ for every adult with a constant $\gamma > 0$. Matching the variances requires $\gamma = \sigma_Y/\sigma_X = 1.083$ mm Hg per year. So the bound would be attained only if every adult's pressure lay exactly on one straight line in age, rising 1.083 mm Hg per year. The observed covariance is 0.4866 of that.

(c) $W \ge 0$, and $W > 0$ exactly when $Y > 140$. The lower bound from two moments gives

$$P(Y > 140) = P(W > 0) \ge \frac{(E[W])^2}{E[W^2]} = \frac{3.543^2}{109.54} = 0.1146.$$

The true share with a pressure above 140 is 0.2049. The bound uses two summary numbers of the excess and no count of people, and it is of the right order. It would be attained only if $W$ were constant on the event $W > 0$, that is, if every adult above 140 were above it by the same amount; this is the equality case of the theorem for $W$ and the indicator of $W > 0$.

Problem 5 (a binomial tail four ways). Of the 5,107 adults of The Binomial Distribution, 2,099 are obese, a share $p = 0.4110$. One hundred adults are drawn with replacement, and $X$ is the number who are obese. Bound $P(X \ge 60)$ by Markov's inequality, Chebyshev's inequality, the one-sided inequality and the Chernoff bound, and compare with the exact probability.

Solution. $X$ is binomial with $n = 100$ and $p = 0.4110$, so $E[X] = np = 41.10$ and $\operatorname{Var}(X) = np(1 - p) = 24.21$ (The Binomial Distribution). Write $q = 1 - p$. The threshold is $60 - 41.10 = 18.90$ above the mean.

Markov. $P(X \ge 60) \le 41.10/60 = 0.6850$.

Chebyshev. $P(X \ge 60) \le P(\lvert X - 41.10\rvert \ge 18.90) \le 24.21/18.90^2 = 0.0678$.

One-sided. $P(X \ge 60) \le 24.21/(24.21 + 18.90^2) = 0.0635$.

Chernoff. The moment generating function of a binomial count is $M_X(t) = (q + pe^t)^n$ (Moment Generating Functions and Slutsky's Theorem, Problem 1). The bound $e^{-60t}(q + pe^t)^{100}$ holds for every $t > 0$, and we choose $t$ to make it small. Write $\theta = 60/100 = 0.6$ for the threshold as a share. The logarithm of the bound is $h(t) = -n\theta t + n\ln(q + pe^t)$, with derivative

$$h'(t) = -n\theta + \frac{n\,pe^t}{q + pe^t},$$

which is zero when $pe^t(1 - \theta) = \theta q$, that is, at

$$e^{t} = \frac{\theta q}{(1 - \theta)p} = \frac{0.6 \times 0.5890}{0.4 \times 0.4110} = 2.1496, \qquad t = 0.7653.$$

This $t$ is positive because $\theta > p$. At this $t$, $q + pe^t = q + \theta q/(1 - \theta) = q/(1 - \theta)$, and $e^{-t\theta} = \big((1 - \theta)p/(\theta q)\big)^{\theta}$, so the bound is

$$\Big(\frac{(1 - \theta)p}{\theta q}\Big)^{n\theta}\Big(\frac{q}{1 - \theta}\Big)^{n} = \Big(\frac{p}{\theta}\Big)^{n\theta}\Big(\frac{q}{1 - \theta}\Big)^{n(1 - \theta)} = \Big(\frac{0.4110}{0.6}\Big)^{60}\Big(\frac{0.5890}{0.4}\Big)^{40} = 0.00073.$$

(Any $t > 0$ gives a valid bound, so no proof that this $t$ is the minimizer is needed for the conclusion. It is the minimizer: $h'$ is increasing, because $pe^t/(q + pe^t)$ increases with $t$.) The exponent can be written $-n\big[\theta\ln(\theta/p) + (1 - \theta)\ln((1 - \theta)/q)\big]$. The bracket is the Kullback–Leibler divergence between a coin with success probability $\theta$ and one with probability $p$, defined in Cross-Entropy and Kullback–Leibler Divergence; here it is 0.0722 nats.

Exact. Summing the binomial probabilities in exact fractions, $P(X \ge 60) = 0.00011$.

Method Uses Bound on $P(X \ge 60)$
Markov mean 0.6850
Chebyshev mean, variance 0.0678
One-sided Chebyshev mean, variance 0.0635
Chernoff moment generating function 0.00073
Exact the whole distribution 0.00011

Each added piece of information tightens the bound, and the step from the variance to the moment generating function is the large one: a factor of 87 here. The Chernoff bound is within a factor of 7 of the truth, and its form $e^{-0.0722n}$ shows how the probability of a share of 0.6 or more falls as the number of draws grows.

Summary

Inequality Statement Needs Equality
Markov $P(X \ge a) \le E[X]/a$ $X \ge 0$, finite mean $X$ is 0 or $a$
Chebyshev $P(\lvert X - \mu\rvert \ge a) \le \sigma^2/a^2$ finite variance $X$ is $\mu$ or $\mu \pm a$
One-sided Chebyshev $P(X - \mu \ge a) \le \sigma^2/(\sigma^2 + a^2)$ finite variance $X - \mu$ is $a$ or $-\sigma^2/a$
Chernoff $P(X \ge a) \le e^{-ta}M_X(t)$, $t > 0$ $M_X(t)$ finite
Jensen $\varphi(E[X]) \le E[\varphi(X)]$ $\varphi$ convex $X$ constant, if $\varphi$ is strictly convex
Cauchy–Schwarz $(E[XY])^2 \le E[X^2]\,E[Y^2]$ finite second moments $\alpha X + \beta Y = 0$

What we have learned

All four inequalities come from two facts: the expected value of a nonnegative variable is not negative, and expected values are linear. Markov's inequality applies them to $X - a\,\mathbf{1}\{X \ge a\}$. Chebyshev's inequality, its one-sided form, the moment bounds and the Chernoff bound are Markov's inequality for a square, a shifted square, a power and an exponential. Jensen's inequality applies the two facts to the gap between a convex function and its supporting line, and the Cauchy–Schwarz inequality to the square of $Y - tX$. Each is attained, by a distribution concentrated on two or three points or on a line, and for a population such as the blood pressures each is far from tight: a pressure of 180 mm Hg or more has probability 0.0181, against bounds from 0.0927 to 0.7020.

The Chernoff bound needs the moment generating function of a distribution and the rule for sums of independent variables. Both are the subject of the next article, Moment Generating and Characteristic Functions.

Every number in this article is computed by tools/experiments/statistics/probability-inequalities.py in the site's repository, which downloads the NHANES files from the CDC and checks their checksums. The figure is drawn by tools/figures/probability-inequalities.py from the same values.

References

  • Chernoff, H. (1952). A measure of asymptotic efficiency for tests of a hypothesis based on the sum of observations. The Annals of Mathematical Statistics, 23(4), 493–507. https://doi.org/10.1214/aoms/1177729330 Theorem 1, p. 495. record
  • Durrett, R. (2019). Probability: Theory and Examples (5th ed.). Cambridge University Press. Theorem 1.6.2 (Jensen's inequality), Theorem 1.6.4 (the general form of Markov's inequality, there called Chebyshev's inequality) and Exercises 1.6.4 and 1.6.6. Theorem numbers are those of the author's version of January 11, 2019. record · 4 pages
  • Grinstead, C. M., and Snell, J. L. (1997). Introduction to Probability (2nd rev. ed.). American Mathematical Society. §8.1 (Chebyshev's inequality and a variable that attains it). record · 20 pages
  • Jensen, J. L. W. V. (1906). Sur les fonctions convexes et les inégalités entre les valeurs moyennes. Acta Mathematica, 30, 175–193. https://doi.org/10.1007/BF02418571 Formula (5), p. 180, and formula (5′), p. 186. record
  • National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018: Demographic Variables and Sample Weights (DEMO_J) and Blood Pressure (BPX_J) data files. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 67 pages