The systolic blood pressures of the 5,247 adults that the statistics course pages use as a population can be recorded to the nearest 8, 4 or 2 mm Hg. Sorted into bins 8 mm Hg wide, the pressures have an entropy of 3.27 bits. In bins of 4 mm Hg the entropy is 4.25 bits, and in bins of 2 mm Hg it is 5.24 bits. Each halving of the bin costs almost exactly one more bit. For a variable with a density this goes on without end, so a measurement on a continuous scale has no finite entropy. Is anything left that belongs to the distribution and not to the bins? Something is. The entropy plus the logarithm of the bin width barely moves, 6.27, 6.25 and 6.24 bits, and the number it settles toward for a density is the differential entropy.
The differential entropy of a random variable $X$ with density $f$ is
$$h(X) = -\int f(x)\ln f(x)\,dx,$$
the discrete formula with the sum replaced by an integral, written $h(f)$ when the density is the object. It is not the entropy of $X$ in the sense of Entropy. It can be negative, and it changes when $X$ is measured in different units.
First we look at the binned pressures and define the differential entropy, with three examples derived. Then we establish what does not carry over from the discrete case: the sign, invariance under a change of variable, and the meaning as a number of bits, which is replaced by a theorem about fine bins. Next we show what does carry over: the divergence between two densities and the mutual information between two variables keep every property they had, and do not depend on units at all. Along the way we prove that the normal distribution has the greatest differential entropy of any distribution with its variance. The article assumes Entropy and Cross-Entropy and Kullback–Leibler Divergence, and densities as Random Variables and Distribution Functions defines them.
This article is the home of differential entropy on this site. Maximum Entropy Distributions builds on its definitions and on the form of Gibbs' inequality proved here.
Notation and scope
Random variables $X$, $Y$ have densities $f$, $g$, or $f_X$, $f_Y$ where two variables are in play, and a joint density $f(x, y)$. Integrals with no limits run over the whole line, and a term with $f(x) = 0$ contributes nothing, as $0\ln 0 = 0$ in the discrete case. Following the unit's convention, calculus is done with the natural logarithm, so $h$ is in nats; where a number of bits is wanted the logarithm is $\log_2$ and the unit is stated. One nat is 1.4427 bits. The normal distribution is described in words, by its mean and its standard deviation or variance. Its density with mean $\mu$ and standard deviation $\sigma$ is $\frac{1}{\sigma\sqrt{2\pi}}e^{-(x-\mu)^2/(2\sigma^2)}$, and the standard normal density is $\phi$.
As in the Probability Theory unit, densities are functions that can be integrated in the Riemann sense, continuous except at finitely many points, and no measure theory is used. One consequence is that statements of equality between densities are made at points of continuity. A differential entropy need not exist: the integral can diverge to $+\infty$ or $-\infty$, or have no value at all (Polyanskiy and Wu, 2025, Section 2.3, warn of all three and give an example of the last). Every theorem below assumes the entropies it mentions are finite.
Entropy in bins
Each adult's pressure is the mean of up to three readings, in mm Hg. The population has mean 126.36, variance 405.01 and standard deviation 20.12, with a long right tail (skewness 1.04). Cut the scale into bins of width $\Delta$ starting at multiples of $\Delta$, and let $X^\Delta$ be the bin a pressure falls in. $X^\Delta$ is a discrete variable, and its entropy $H(X^\Delta)$ is computed from the bin counts as in Entropy:
| Bin width $\Delta$ (mm Hg) | Occupied bins | $H(X^\Delta)$, bits | $H(X^\Delta) + \log_2\Delta$ |
|---|---|---|---|
| 32 | 6 | 1.39 | 6.39 |
| 16 | 11 | 2.31 | 6.31 |
| 8 | 21 | 3.27 | 6.27 |
| 4 | 41 | 4.25 | 4.25 + 2 = 6.25 |
| 2 | 75 | 5.24 | 6.24 |
| 1 | 134 | 6.16 | 6.16 |
The third column grows by about one bit per halving, because halving the bins splits each bin's probability roughly in two. The last column removes that growth. It is steady between 6.24 and 6.27 bits for bins of 8, 4 and 2 mm Hg. (The bottom row breaks the pattern, for a reason that belongs to the data and is explained after the theorem on fine bins.) The differential entropy is the quantity that the last column approximates.
Definition and three examples
Why should $-\int f\ln f$ be that quantity? Here is the idea. In a narrow bin around $x$ the probability is about $f(x)\Delta$. The entropy of the bins is then about $-\sum f(x)\Delta\,\ln\big(f(x)\Delta\big)$, which separates into $-\sum f(x)\ln f(x)\,\Delta$, a Riemann sum for $-\int f\ln f$, and $-\ln\Delta\sum f(x)\Delta$, which is about $-\ln\Delta$. So $H(X^\Delta) \approx h(X) - \ln\Delta$. The section "The limit of fine bins" makes this a theorem. First, three densities for which the integral can be done.
Uniform. If $X$ is uniform on an interval of length $w$, then $f = 1/w$ there and
$$h(X) = -\int \frac{1}{w}\ln\frac{1}{w}\,dx = \ln w.$$
Exponential. If $X$ has density $f(x) = \lambda e^{-\lambda x}$ for $x \ge 0$, with mean $1/\lambda$, then $\ln f(x) = \ln\lambda - \lambda x$ and
$$h(X) = -E[\ln\lambda - \lambda X] = -\ln\lambda + \lambda\cdot\frac{1}{\lambda} = 1 - \ln\lambda = 1 + \ln(\text{mean}).$$
Normal. If $X$ is normal with mean $\mu$ and standard deviation $\sigma$, then $\ln f(x) = -\tfrac12\ln(2\pi\sigma^2) - \frac{(x - \mu)^2}{2\sigma^2}$, and since $E[(X - \mu)^2] = \sigma^2$,
$$h(X) = \tfrac12\ln(2\pi\sigma^2) + \frac{\sigma^2}{2\sigma^2} = \tfrac12\ln(2\pi\sigma^2) + \tfrac12 = \tfrac12\ln(2\pi e\,\sigma^2). \qquad (1)$$
For the standard normal this is 1.4189 nats, or 2.0471 bits. The mean does not appear: sliding a density along the line does not change $-\int f\ln f$. Shannon (1948, Section 20) gives the definition and these three values. The script checks each closed form in this article against a numerical integral.
In all three cases $h$ is the logarithm of a width. The uniform case says it directly, and suggests reading $e^{h(X)}$, or $2^{h}$ with $h$ in bits, as the effective width of a distribution: the length of the interval on which a uniform variable has the same differential entropy. For a normal distribution the effective width is $\sqrt{2\pi e}\,\sigma = 4.1327\,\sigma$. This is the continuous counterpart of the perplexity of Perplexity and Source Coding, which reads $2^{H}$ as an effective number of outcomes.
What does not carry over
Three properties of discrete entropy fail for $h$. Each failure has the same cause: a density is a probability per unit of length, so $f(x)$ carries a unit, and its logarithm depends on what the unit is.
The sign
Discrete entropy is never negative (Entropy, the bounds). Differential entropy can be.
Example (a negative entropy). A variable uniform on $[0, \tfrac12]$ has $h = \ln\tfrac12 = -0.6931$ nats, which is $-1$ bit. Nothing is wrong. The density is 2 on that interval, the logarithm of a density above 1 is positive, and $h$ averages the negative of it. By formula (1) a normal distribution has negative differential entropy as soon as $\sigma < 1/\sqrt{2\pi e} = 0.2420$; with $\sigma = 0.1$ it is $-0.8836$ nats. A negative value does not mean less than no uncertainty. It means an effective width below one unit: the zero of the scale is the uniform distribution on an interval of length 1, as Shannon (1948, Section 20) observes.
Change of variables
A one-to-one relabeling of the outcomes leaves discrete entropy unchanged, because entropy depends only on the probabilities (Entropy, the entropy of a function). For densities, a one-to-one map stretches some regions and compresses others, and the entropy changes by the average logarithm of the stretch.
Theorem (change of variables). Let $X$ have density $f$, and let $T$ be a function that is continuously differentiable and strictly monotone, with $T'(x) \ne 0$, on an open interval that contains $X$ with probability 1. Then $Y = T(X)$ has
$$h(Y) = h(X) + E\big[\ln\lvert T'(X)\rvert\big],$$
provided the entropies and the expectation are finite.
Proof. By the change of variables formula for densities (Random Variables and Distribution Functions, Theorem 5), $Y$ has density $f_Y(y) = f(x)/\lvert T'(x)\rvert$ at $y = T(x)$. Substitute $y = T(x)$, for which $dy = \lvert T'(x)\rvert\,dx$, in the integral that defines $h(Y)$:
$$h(Y) = -\int f_Y(y)\ln f_Y(y)\,dy = -\int \frac{f(x)}{\lvert T'(x)\rvert}\ln\frac{f(x)}{\lvert T'(x)\rvert}\,\lvert T'(x)\rvert\,dx = -\int f(x)\ln f(x)\,dx + \int f(x)\ln\lvert T'(x)\rvert\,dx. \quad\square$$
Corollary (shift and scale). For constants $a \ne 0$ and $b$, $h(aX + b) = h(X) + \ln\lvert a\rvert$. Shifting changes nothing, and multiplying by $a$ adds $\ln\lvert a\rvert$.
Example (the entropy depends on the unit). A normal distribution with the variance of the pressures, 405.01, has $h = \tfrac12\log_2(2\pi e \times 405.01) = 6.378$ bits when pressure is in mm Hg. Expressed in centimeters of mercury every value is divided by 10, and the corollary gives $6.378 - \log_2 10 = 6.378 - 3.322 = 3.056$ bits. The same people and the same pressures have two different differential entropies. Neither is the "information in a blood pressure"; each is the logarithm of an effective width, 83.2 mm Hg in one unit and 8.32 cm Hg in the other.
A map that is not linear changes the entropy by an amount that depends on the distribution. If $X$ is normal with mean $\mu$ and standard deviation $\sigma$ and $Y = e^X$, then $T'(x) = e^x$ and $h(Y) = h(X) + E[X] = \tfrac12\ln(2\pi e\sigma^2) + \mu$. For $\mu = 0.5$ and $\sigma = 0.4$ that is $0.5026 + 0.5 = 1.0026$ nats, which the script confirms by integrating the density of $Y$.
Shannon (1948, Section 20) states the general rule for several variables: the new entropy is the old one less the expected logarithm of the Jacobian of the inverse map. He draws the conclusion used throughout this article. In the continuous case entropy is measured relative to the coordinate system. But quantities that are differences of two entropies do not depend on the coordinates, because both terms change by the same amount.
The meaning as a number of bits
Perplexity and Source Coding proves that $H(X)$ in bits is the least average number of binary digits that records an outcome of a discrete variable. A value of a continuous variable cannot be recorded exactly in any finite number of digits, and $h(X)$ is not that number. What $h$ gives is the number of digits needed to record $X$ to a stated precision, which is the theorem of the next section.
The limit of fine bins
Theorem (entropy of a quantized variable). Let $X$ have a density $f$ that is continuous on a closed interval $[a, b]$ and zero outside it. For $\Delta > 0$ let $X^\Delta$ be the index $i$ of the bin $[i\Delta, (i+1)\Delta)$ that contains $X$. Then
$$H(X^\Delta) + \ln\Delta \to h(X) \quad \text{as } \Delta \to 0.$$
The plan is the calculation sketched above, done with care at the two ends of the interval, where a bin may be cut by $a$ or $b$.
Proof. Write $\varphi(t) = -t\ln t$ for $t > 0$ and $\varphi(0) = 0$. It is continuous on $[0, \infty)$, since $t\ln t \to 0$ as $t \to 0$. Let $M$ be the maximum of $f$ on $[a, b]$, and let $K$ be the maximum of $\lvert\varphi\rvert$ on $[0, M]$. The bin probabilities are $p_i = \int_{i\Delta}^{(i+1)\Delta} f(x)\,dx$, and since they sum to 1,
$$H(X^\Delta) + \ln\Delta = \sum_i \varphi(p_i) + \ln\Delta\sum_i p_i = \sum_i\big[\varphi(p_i) + p_i\ln\Delta\big].$$
Call a bin interior if it lies inside $[a, b]$, and an edge bin if it contains $a$ or $b$ without lying inside; there are at most two edge bins, and every other bin has $p_i = 0$. For an interior bin, the mean value theorem for integrals gives a point $x_i$ in the bin with $p_i = f(x_i)\Delta$. Then
$$\varphi(p_i) + p_i\ln\Delta = -f(x_i)\Delta\big[\ln f(x_i) + \ln\Delta\big] + f(x_i)\Delta\ln\Delta = \varphi\big(f(x_i)\big)\Delta$$
(if $f(x_i) = 0$ both sides are 0). So
$$H(X^\Delta) + \ln\Delta = \sum_{\text{interior}}\varphi\big(f(x_i)\big)\Delta + \sum_{\text{edge}}\big[\varphi(p_i) + p_i\ln\Delta\big].$$
The first sum runs over the interior bins, which cover $[a, b]$ except for at most two end pieces, each shorter than $\Delta$. It tends to $\int_a^b \varphi(f(x))\,dx = h(X)$, by two estimates. The function $\varphi(f(x))$ is continuous on the closed interval $[a, b]$, hence uniformly continuous: for every $\varepsilon > 0$ there is a $\delta > 0$ such that its values at two points less than $\delta$ apart differ by less than $\varepsilon$. When $\Delta < \delta$, the term $\varphi(f(x_i))\Delta$ of an interior bin differs from the integral of $\varphi(f)$ over that bin by less than $\varepsilon\Delta$, because $x_i$ lies in the bin. Summed over the interior bins, whose total length is at most $b - a$, the differences come to less than $\varepsilon(b - a)$. And the integral of $\varphi(f)$ over the two end pieces is at most $2K\Delta$ in absolute value. So the first sum differs from $h(X)$ by less than $\varepsilon(b - a) + 2K\Delta$, which is as small as desired for small $\Delta$. For an edge bin, $p_i \le M\Delta$. So $p_i \to 0$, and $\varphi(p_i) \to 0$ by the continuity of $\varphi$ at 0; and $\lvert p_i\ln\Delta\rvert \le M\Delta\lvert\ln\Delta\rvert \to 0$. The edge terms vanish in the limit. $\square$
In bits the statement is $H(X^\Delta) \approx h(X) - \log_2\Delta$, with $h$ in bits. Recording a variable to within a bin of width $\Delta = 2^{-k}$ costs about $h(X) + k$ bits: the differential entropy, plus one bit for each binary digit of precision. That is the operational meaning of $h$, and it is why $h$ alone has none. It is the constant in a cost that grows without bound.
The theorem is proved here for a continuous density on a closed interval. It holds far more widely. Polyanskiy and Wu (2025, equation 2.21), crediting Theorem 1 of Rényi (1959), state that $H(X^\Delta) + \ln\Delta \to h(X)$ whenever $h(X)$ is defined and the entropy of the whole-number part of $X$ is finite; they write it with $\Delta = 1/m$. Rényi's paper was not consulted for this article. Cover and Thomas (2006) treat the relation in their Section 8.3, "Relation of Differential Entropy to Discrete Entropy". Three exact cases show the convergence, two of them on densities the theorem above does not cover.
Uniform. For $X$ uniform on $[0, 1]$ and $\Delta = 1/m$, the $m$ bins are equally likely, so $H(X^\Delta) = \ln m$ and $H(X^\Delta) + \ln\Delta = \ln m - \ln m = 0 = h(X)$ for every $m$, not only in the limit.
Exponential. Let $X$ be exponential with mean 1. The bin $[i\Delta, (i+1)\Delta)$ has probability $e^{-i\Delta} - e^{-(i+1)\Delta} = (1 - \theta)^i\theta$ with $\theta = 1 - e^{-\Delta}$: the binned variable is geometric. The entropy of a geometric distribution with parameter $\theta$ is $\big[-\theta\ln\theta - (1-\theta)\ln(1-\theta)\big]/\theta$ nats (Entropy, Problem 3; shifting the support from $1, 2, \dots$ to $0, 1, \dots$ does not change it). With $\ln(1 - \theta) = -\Delta$ this is
$$H(X^\Delta) = -\ln\big(1 - e^{-\Delta}\big) + \frac{\Delta e^{-\Delta}}{1 - e^{-\Delta}}.$$
As $\Delta \to 0$, $(1 - e^{-\Delta})/\Delta \to 1$, so the second term tends to 1 and $-\ln(1 - e^{-\Delta}) + \ln\Delta = -\ln\frac{1 - e^{-\Delta}}{\Delta} \to 0$. Hence $H(X^\Delta) + \ln\Delta \to 1 = h(X)$, exactly as the theorem says, for a density on an unbounded interval. Numerically, $H(X^\Delta) + \ln\Delta$ is 1.0407 for $\Delta = 1$, 1.0104 for $\Delta = 0.5$, 1.0004 for $\Delta = 0.1$ and 1.0000 for $\Delta = 0.01$ (to four places), while $H(X^\Delta)$ itself climbs from 1.0407 to 5.6052 nats.
Standard normal. With bin probabilities from the normal distribution function:
| $\Delta$ | $H(X^\Delta)$, bits | $H(X^\Delta) + \log_2\Delta$ |
|---|---|---|
| 2 | 1.2677 | 2.2677 |
| 1 | 2.1048 | 2.1048 |
| 1/2 | 3.0620 | 2.0620 |
| 1/4 | 4.0508 | 2.0508 |
| 1/16 | 6.0473 | 2.0473 |
| 1/64 | 8.0471 | 2.0471 |
The last column converges to $h = 2.0471$ bits, and is already within 0.004 bits of it when the bins are a quarter of a standard deviation wide.
Back to the pressures. A normal density with the pressures' mean and variance, binned as the pressures were, gives $H + \log_2\Delta$ equal to 6.52, 6.42, 6.39, 6.38, 6.38 and 6.38 bits for $\Delta$ from 32 down to 1: it settles at its differential entropy, 6.378 bits. Under that model, recording a pressure to the nearest 2 mm Hg would take about $6.378 - \log_2 2 = 5.38$ bits. The pressures themselves give 6.39, 6.31, 6.27, 6.25, 6.24 and 6.16, and at 2 mm Hg they took 5.24 bits.
Two things separate the two rows. Between $\Delta = 16$ and $\Delta = 2$ the pressures sit 0.11 to 0.13 bits below the normal density at every bin width. That gap is real, and the section on the Gaussian explains it. At $\Delta = 1$ the pressures fall further, and that drop is an artifact that matters for every use of the theorem on data.
Example (data have no differential entropy). The theorem is about a density. A dataset is not one: it is a finite list, and at a fine enough scale it is discrete. Each reading here was recorded as an even whole number, and each adult's value is a mean of up to three. So every value is a multiple of $\tfrac13$ mm Hg, and only 237 distinct values occur. Their entropy is 6.97 bits. However small $\Delta$ is made, $H(X^\Delta)$ for these data never exceeds 6.97 bits, and in any dataset of $n$ values it cannot exceed $\log_2 n$, here 12.36 bits. So $H(X^\Delta) + \log_2\Delta$ computed from data eventually falls without limit as $\Delta$ shrinks. The plateau at bin widths of 2 to 8 mm Hg, 6.24 to 6.27 bits, is an estimate of the differential entropy of a smooth density that the data are imagined to come from. It is only as good as that picture. It corresponds to an effective width of about 76 mm Hg (76.3 at $\Delta = 4$).
Two variables: joint and conditional entropy
For a pair with joint density $f(x, y)$, the joint differential entropy is $h(X, Y) = -\iint f(x, y)\ln f(x, y)\,dx\,dy$. With the conditional density $f(y \mid x) = f(x, y)/f_X(x)$, the conditional differential entropy is
$$h(Y \mid X) = -\iint f(x, y)\ln f(y \mid x)\,dx\,dy = \int f_X(x)\,h(Y \mid X = x)\,dx,$$
the average over $x$ of the entropy of the conditional density. The chain rule holds as in the discrete case, for the same reason: $\ln f(x, y) = \ln f_X(x) + \ln f(y \mid x)$, and multiplying by $-f(x, y)$ and integrating gives
$$h(X, Y) = h(X) + h(Y \mid X).$$
If $X$ and $Y$ are independent, $f(y \mid x) = f_Y(y)$ and so $h(X, Y) = h(X) + h(Y)$. These identities are bookkeeping and survive unchanged. Two further facts of the discrete case, that conditioning lowers entropy on average and that joint entropy is at most the sum of the marginal entropies, also survive, and are proved below from the mutual information. For random vectors, a linear map $A$ adds $\ln\lvert\det A\rvert$ to the differential entropy, and the normal distribution in $d$ dimensions with covariance matrix $\Sigma$ has $h = \tfrac12\ln\big((2\pi e)^d\det\Sigma\big)$; both are stated by Shannon (1948, Section 20) and by Polyanskiy and Wu (2025, Theorem 2.7 and equation 2.18) and are cited here, not proved. The Multivariate Normal Distribution is the planned home of that distribution.
Divergence for densities
The property of discrete entropy that mattered most was not entropy itself but Gibbs' inequality, which is a statement about two distributions. For densities $f$ and $g$ the Kullback–Leibler divergence is
$$D(f \,\|\, g) = \int f(x)\ln\frac{f(x)}{g(x)}\,dx,$$
where the integrand is read as 0 at points with $f(x) = 0$. If $g = 0$ on an interval where $f > 0$, the divergence is $+\infty$, as in the discrete case. If $g$ vanishes only at isolated points where $f > 0$, the integral is improper and may still be finite: for $f = 1$ and $g(x) = 2x$ on $[0, 1]$ it is $\int_0^1 \ln\frac{1}{2x}\,dx = 1 - \ln 2$. The theorems below assume $g(x) > 0$ wherever $f(x) > 0$. Unlike $h$, the divergence keeps its sign.
Theorem (Gibbs' inequality for densities). Let $f$ and $g$ be densities with $g(x) > 0$ wherever $f(x) > 0$. Then $D(f \,\|\, g) \ge 0$, and $D(f \,\|\, g) = 0$ exactly when $f(x) = g(x)$ at every point where both are continuous.
The proof rests on the same inequality as the discrete one: $\ln t \le t - 1$ for $t > 0$, with equality only at $t = 1$, which Cross-Entropy and Kullback–Leibler Divergence proves. The plan is to write the divergence as the integral of a function that is never negative.
Proof. Define the excess
$$e(x) = g(x) - f(x) - f(x)\ln\frac{g(x)}{f(x)} \quad \text{where } f(x) > 0, \qquad e(x) = g(x) \quad \text{where } f(x) = 0. \qquad (2)$$
Where $f(x) > 0$, put $t = g(x)/f(x)$; then $e(x) = f(x)\,(t - 1 - \ln t)$, which is nonnegative, and zero only if $t = 1$, that is, only if $g(x) = f(x)$. Where $f(x) = 0$, $e(x) = g(x) \ge 0$. So $e \ge 0$ everywhere. Both densities integrate to 1, so $\int (g - f)\,dx = 0$, and adding this zero to the definition of the divergence gives
$$D(f \,\|\, g) = \int \Big[f\ln\frac{f}{g} + g - f\Big]dx = \int e(x)\,dx \ge 0.$$
(The integral of the nonnegative function $e$ is a finite number or $+\infty$, and the divergence with it.)
Equality. If $f = g$ at every point where both are continuous, then $e = 0$ except at finitely many points, and $D = 0$. Conversely, suppose $f(x_0) \ne g(x_0)$ at a point $x_0$ where both are continuous. We show that $e$ is bounded below by a positive number on a short interval around $x_0$, so that $D = \int e > 0$.
- Case 1: $f(x_0) > 0$. Then $g(x_0) > 0$ by hypothesis, $e$ is continuous at $x_0$, and $e(x_0) > 0$ because $g(x_0) \ne f(x_0)$. So $e > e(x_0)/2$ on a short interval around $x_0$.
- Case 2: $f(x_0) = 0$. Then $c = g(x_0) > 0$. By continuity there is a short interval around $x_0$ on which $g > c/2$ and $f < c/4$. At a point of it where $f = 0$, $e = g > c/2$. At a point of it where $f > 0$, the ratio $t = g/f$ is greater than 2. The function $(t - 1 - \ln t)/t$ has derivative $(\ln t)/t^2 > 0$ for $t > 1$, so for $t \ge 2$ it is at least its value at 2, which is $(1 - \ln 2)/2 > 0.15$. Hence $e = f\,(t - 1 - \ln t) \ge 0.15\,f\,t = 0.15\,g > 0.075\,c$. Either way $e > 0.075\,c$ on the whole interval. $\square$
Invariance
Theorem (the divergence does not depend on the coordinates). Let $T$ be as in the change of variables theorem, on an interval that contains the variable with probability 1 under both $f$ and $g$. Let $f_T$ and $g_T$ be the densities of $T(X)$ when $X$ has density $f$ and when it has density $g$. Then $D(f_T \,\|\, g_T) = D(f \,\|\, g)$.
Proof. At $y = T(x)$, $f_T(y) = f(x)/\lvert T'(x)\rvert$ and $g_T(y) = g(x)/\lvert T'(x)\rvert$. The factor cancels in the ratio: $f_T(y)/g_T(y) = f(x)/g(x)$. Substituting $y = T(x)$, $dy = \lvert T'(x)\rvert\,dx$,
$$D(f_T \,\|\, g_T) = \int f_T(y)\ln\frac{f_T(y)}{g_T(y)}\,dy = \int \frac{f(x)}{\lvert T'(x)\rvert}\ln\frac{f(x)}{g(x)}\,\lvert T'(x)\rvert\,dx = D(f \,\|\, g). \quad\square$$
The term $E[\ln\lvert T'(X)\rvert]$ that spoiled the invariance of $h$ appears once in each of the two entropies that the divergence compares, and cancels. This is Shannon's remark made exact. The divergence between two distributions of blood pressure is the same in mm Hg, in cm Hg, or on a logarithmic scale.
Bins
For the binned variable, let $P_i = \int_{B_i} f$ and $Q_i = \int_{B_i} g$ be the probabilities of bin $B_i$ under the two densities. In the entropy, binning produced a term $-\ln\Delta$ that diverges. In the divergence the two such terms cancel, and binning can only lose.
Theorem (binning does not increase the divergence). If $g > 0$ wherever $f > 0$, then for any division of the line into intervals $B_i$,
$$\sum_i P_i\ln\frac{P_i}{Q_i} \le D(f \,\|\, g).$$
Proof. By the proof of Gibbs' inequality, $f\ln\frac{f}{g} = e - g + f$ with $e \ge 0$, so the integral of $f\ln\frac fg$ over a bin $B$ is $\int_B e\,dx - Q + P$. Summing over the bins, of which there may be infinitely many, the nonnegative integrals $\int_B e$ add up to $\int e = D(f \,\|\, g)$ and the $P$'s and the $Q$'s each add up to 1. So
$$D(f \,\|\, g) = \sum_B \int_B f\ln\frac{f}{g}\,dx,$$
and it is enough to prove $\int_B f\ln\frac{f}{g}\,dx \ge P\ln\frac{P}{Q}$ for each bin.
Fix a bin $B$ with $P = \int_B f > 0$. Then $f$ is positive at some point of $B$ where it is continuous, hence on a short interval, and $g$ is positive there by assumption, so $Q = \int_B g > 0$ as well. If $\int_B f\ln\frac{f}{g}\,dx = +\infty$ there is nothing to prove, so suppose it is finite. On the points of $B$ where $g = 0$ we have $f = 0$, so those points contribute to neither integral. Let $W$ be a random variable with density $g(x)/Q$ on the part of $B$ where $g > 0$, and let $r(x) = f(x)/g(x)$. Then $E[r(W)] = \int_B \frac{f}{g}\cdot\frac{g}{Q}\,dx = P/Q$. The function $\psi(t) = t\ln t$, with $\psi(0) = 0$, is convex on $[0, \infty)$, since $\psi''(t) = 1/t > 0$, and $E[\psi(r(W))] = \frac1Q\int_B f\ln\frac fg\,dx$ is finite. By Jensen's inequality (Probability Inequalities, section "Jensen's inequality"),
$$\int_B f\ln\frac{f}{g}\,dx = Q\int_B \psi\big(r(x)\big)\frac{g(x)}{Q}\,dx = Q\,E\big[\psi(r(W))\big] \ge Q\,\psi\!\left(\frac{P}{Q}\right) = P\ln\frac{P}{Q}.$$
For a bin with $P = 0$, the term $P\ln\frac{P}{Q}$ is 0 by convention. On such a bin $f$, which is nonnegative with integral 0, is zero at every point of continuity, so $\int_B f\ln\frac{f}{g}\,dx = 0$ as well. $\square$
This is the continuous form of the theorem on merging outcomes in Cross-Entropy and Kullback–Leibler Divergence. More is true: the divergence between two distributions is the least upper bound of the divergences between their binned versions over all finite divisions into bins (Polyanskiy and Wu, 2025, Theorem 4.5, where it is attributed to Gelfand, Yaglom and Perez). That statement is cited, not proved here. It is the reason the divergence of densities deserves the same name as the discrete one: it is what the discrete divergence converges to, with no correction term.
Two normal densities
Let $f$ be normal with mean $\mu_1$ and standard deviation $\sigma_1$, and $g$ normal with mean $\mu_2$ and standard deviation $\sigma_2$. Then
$$\ln\frac{f(x)}{g(x)} = \ln\frac{\sigma_2}{\sigma_1} - \frac{(x - \mu_1)^2}{2\sigma_1^2} + \frac{(x - \mu_2)^2}{2\sigma_2^2}.$$
Take the expectation under $f$. $E[(X - \mu_1)^2] = \sigma_1^2$, and $E[(X - \mu_2)^2] = \sigma_1^2 + (\mu_1 - \mu_2)^2$, the variance plus the squared distance of the mean from $\mu_2$. So
$$D(f \,\|\, g) = \ln\frac{\sigma_2}{\sigma_1} + \frac{\sigma_1^2 + (\mu_1 - \mu_2)^2}{2\sigma_2^2} - \frac12 \ \text{nats}. \qquad (3)$$
With $\sigma_1 = \sigma_2 = \sigma$ this is $(\mu_1 - \mu_2)^2/(2\sigma^2)$, the case worked in Cross-Entropy and Kullback–Leibler Divergence (Problem 4). The formula is unchanged if all four parameters are multiplied by the same constant, as the invariance theorem requires.
On the pressures. Of the 5,247 adults, 2,547 are men, with mean 127.41 and standard deviation 18.37 mm Hg, and 2,700 are women, with mean 125.37 and standard deviation 21.60. For the two normal densities with those parameters, formula (3) gives, from men to women,
$$\ln\frac{21.60}{18.37} + \frac{18.37^2 + (127.41 - 125.37)^2}{2 \times 21.60^2} - \frac12 = 0.1621 + 0.3660 - 0.5 = 0.0281 \ \text{nats},$$
which is 0.0406 bits. (The three terms are computed from the unrounded means and standard deviations; the rounded ones shown give 0.1620 and 0.3661.) In the other direction it is 0.0513 bits. The divergence is not symmetric, and here the difference in spread, not the 2 mm Hg difference in means, produces most of it. Binned, the divergence from the men's density to the women's is 0.0063 bits with bins of 64 mm Hg, 0.0290 with 32, 0.0371 with 16, 0.0397 with 8, 0.0403 with 4 and 0.0405 with 2. It rises toward 0.0406 and never passes it, as the two statements about bins require. These are divergences between two fitted normal curves, not between the two groups' actual distributions, which are skewed.
The Gaussian has the greatest entropy for its variance
The binned pressures sat below the binned normal density at every bin width. That is an instance of a theorem, and Gibbs' inequality for densities proves it in three lines.
Theorem (the normal maximizes differential entropy at a given variance). Let $X$ have a density, variance $\sigma^2$, and finite differential entropy. Then
$$h(X) \le \tfrac12\ln(2\pi e\,\sigma^2),$$
with equality exactly when $X$ is normal.
Here is the idea. Compare the density of $X$ with the normal density that has the same mean and variance. The cross-entropy of $X$ against that normal density depends on $X$ only through its mean and variance, because the logarithm of a normal density is a quadratic. So it equals the normal's own entropy, and the difference between the two entropies is a divergence, which cannot be negative.
Proof. Let $f$ be the density of $X$ and $\mu$ its mean, and let $g$ be the normal density with mean $\mu$ and variance $\sigma^2$. Since $\ln g(x) = -\tfrac12\ln(2\pi\sigma^2) - \frac{(x - \mu)^2}{2\sigma^2}$ and $E[(X - \mu)^2] = \sigma^2$,
$$-\int f(x)\ln g(x)\,dx = \tfrac12\ln(2\pi\sigma^2) + \frac{\sigma^2}{2\sigma^2} = \tfrac12\ln(2\pi e\,\sigma^2),$$
which by formula (1) is $h(g)$. The normal density is positive everywhere, so Gibbs' inequality applies:
$$0 \le D(f \,\|\, g) = \int f\ln f\,dx - \int f\ln g\,dx = -h(X) + \tfrac12\ln(2\pi e\,\sigma^2).$$
Equality holds exactly when $D(f \,\|\, g) = 0$, that is, when $f = g$ at every point of continuity. $\square$
The proof gives more than the inequality. The shortfall is itself a divergence:
$$\tfrac12\ln(2\pi e\,\sigma^2) - h(X) = D(f \,\|\, g), \qquad (4)$$
the divergence of the distribution from the normal with its mean and variance. It is a measure of how far from normal a distribution is that, by the invariance theorem, does not depend on the unit. For three familiar shapes it can be computed from the closed forms. At variance 1, the normal has $h = 1.4189$ nats, and:
| Density with variance 1 | $h$, nats | Gap to the normal, nats | Gap, bits |
|---|---|---|---|
| Laplace (two-sided exponential) | 1.3466 | 0.0724 | 0.104 |
| Uniform, width $\sqrt{12}$ | 1.2425 | 0.1765 | 0.255 |
| Exponential, mean 1 | 1.0000 | 0.4189 | 0.604 |
The gaps are $\tfrac12\ln\frac{\pi}{e}$, $\tfrac12\ln\frac{\pi e}{6}$ and $\tfrac12\ln\frac{2\pi}{e}$. The script confirms each by integrating $f\ln(f/g)$ directly. (The Laplace density with scale $b$ is $\frac{1}{2b}e^{-\lvert x\rvert/b}$, with variance $2b^2$ and entropy $1 + \ln 2b$, by the same calculation as the exponential.)
Back to the pressures. The bound for a variance of 405.01 is 6.378 bits, an effective width of 83.2 mm Hg. The binned pressures plateau at 6.24 to 6.27 bits, about 0.12 bits below (0.12 at $\Delta = 4$). By identity (4) that gap estimates the divergence of the pressure distribution from the normal: a small number, smaller than the gap of a uniform distribution, for a distribution whose skewness of 1.04 is plain in the figure. Divergence from the normal weighs the whole shape, and a long thin tail that holds little probability moves it only slightly.
Shannon (1948, Section 20) proved the theorem by the calculus of variations. The proof by comparison with the normal density is the one Polyanskiy and Wu (2025, Theorem 2.8) give. Maximum Entropy Distributions shows that the same three lines prove a whole family of results, of which this is one member: the uniform, exponential and geometric distributions maximize entropy under other constraints, for the same reason.
Mutual information for densities
For a pair with joint density $f(x, y)$ and marginal densities $f_X$ and $f_Y$, the mutual information is the divergence of the joint density from the product of the marginals, as in Mutual Information:
$$I(X; Y) = \iint f(x, y)\ln\frac{f(x, y)}{f_X(x)\,f_Y(y)}\,dx\,dy.$$
Mutual information keeps the properties that differential entropy lost. To prove that, Gibbs' inequality is needed for densities in the plane, and the class of densities has to be stated again. A joint density as simple as the uniform density on a square is discontinuous along whole line segments, not at finitely many points.
In this section a joint density is a nonnegative function $f(x, y)$ with $\iint f\,dx\,dy = 1$ that is continuous except possibly on finitely many horizontal and vertical lines, and whose marginal densities $f_X(x) = \int f(x, y)\,dy$ and $f_Y(y) = \int f(x, y)\,dx$ are densities of the kind used on the line. The product $f_X(x)\,f_Y(y)$ is then a joint density of the same kind: it is continuous except on the vertical lines through the discontinuities of $f_X$ and the horizontal lines through those of $f_Y$. The class contains every density used in this article: the bivariate normal density below, which is continuous everywhere, the density of a signal and its noisy copy in Problem 5, and products of densities on the line. It does not contain, for example, the uniform density on a disk. The cited sources treat arbitrary joint distributions with measure theory. A finite set of lines has zero area, so values on those lines do not affect any double integral.
Theorem (Gibbs' inequality in the plane). Let $f$ and $g$ be joint densities with $g > 0$ wherever $f > 0$, except possibly at points of finitely many horizontal and vertical lines. Then
$$D(f \,\|\, g) = \iint f(x, y)\ln\frac{f(x, y)}{g(x, y)}\,dx\,dy \ge 0,$$
with equality exactly when $f = g$ at every point where both are continuous.
Proof. Define the excess $e(x, y)$ from $f(x, y)$ and $g(x, y)$ by formula (2), at every point off the exceptional lines of the hypothesis; on those lines, which have zero area, it may be left undefined. As on the line, $e \ge 0$ wherever it is defined, and $e = 0$ at a point with $f > 0$ only if $g = f$ there. Since $\iint (g - f)\,dx\,dy = 1 - 1 = 0$,
$$D(f \,\|\, g) = \iint \Big[f\ln\frac{f}{g} + g - f\Big]dx\,dy = \iint e\,dx\,dy \ge 0.$$
If $f = g$ at every point where both are continuous, then $e = 0$ except on finitely many lines, and $D = 0$. Conversely, suppose $f \ne g$ at a point $(x_0, y_0)$ where both are continuous. Only continuity at that one point is used. If $f_0 = f(x_0, y_0) > 0$, write $2\eta = \lvert g(x_0, y_0) - f_0\rvert > 0$. By continuity at the point there is a small disk around it on which $f_0/2 < f < 2f_0$ and $\lvert g - f\rvert > \eta$. Take a point of the disk off the exceptional lines, so that $g > 0$ there and $e$ is defined. The ratio $t = g/f$ satisfies $\lvert t - 1\rvert = \lvert g - f\rvert/f > \tau$, with $\tau = \eta/(2f_0)$. The function $k(t) = t - 1 - \ln t$ decreases on $(0, 1]$ and increases on $[1, \infty)$, and is positive except at $t = 1$. If $t > 1$, then $t > 1 + \tau$ and $k(t) > k(1 + \tau)$. If $t < 1$, then $t < 1 - \tau$, which can happen only when $\tau < 1$, and then $k(t) > k(1 - \tau)$. Let $\kappa$ be $k(1 + \tau)$, or the smaller of $k(1 + \tau)$ and $k(1 - \tau)$ when $\tau < 1$; it is positive. Hence $e = f\,k(t) > \tfrac12 f_0\,\kappa$ at every such point of the disk. If $f(x_0, y_0) = 0$, let $c = g(x_0, y_0) > 0$, and take a disk on which $g > c/2$ and $f < c/4$. At its points with $f = 0$, $e = g > c/2$. At its points with $f > 0$, the ratio $t = g/f$ exceeds 2, and the inequality $t - 1 - \ln t \ge 0.15\,t$ for $t \ge 2$, established in the proof on the line, gives $e \ge 0.15\,g > 0.075\,c$. In both cases $e$ exceeds a positive constant at every point of a disk except those on the finitely many exceptional lines, and the double integral of $e$ is at least that constant times the area of the disk. So $D > 0$. $\square$
Theorem (mutual information for densities). Let $(X, Y)$ have a joint density $f$ in the sense above. Then:
(a) $I(X; Y) \ge 0$, with equality exactly when $f(x, y) = f_X(x)\,f_Y(y)$ at every point where both sides are continuous, and that condition holds exactly when $X$ and $Y$ are independent.
(b) When the entropies are finite, $I(X; Y) = h(X) + h(Y) - h(X, Y) = h(Y) - h(Y \mid X) = h(X) - h(X \mid Y)$. Consequently $h(Y \mid X) \le h(Y)$ and $h(X, Y) \le h(X) + h(Y)$.
(c) If $U = S(X)$ and $V = T(Y)$, for maps $S$ and $T$ as in the change of variables theorem, then $I(U; V) = I(X; Y)$.
(d) If the line of $x$ values is divided into intervals $B_i$ and the line of $y$ values into intervals $B'_j$, and $X^B$ and $Y^B$ are the indices of the intervals that contain $X$ and $Y$, then $I(X^B; Y^B) \le I(X; Y)$.
Proof. (a) At a point of continuity with $f(x_0, y_0) > 0$, $f$ is positive on a small disk, so the integrals that define $f_X(x_0)$ and $f_Y(y_0)$ are positive. So the product of the marginals is positive wherever $f$ is, except possibly at points of the finitely many lines on which $f$ is discontinuous. That is the hypothesis of Gibbs' inequality in the plane, which, applied to $f$ and the product $f_X f_Y$, gives $I \ge 0$ and the case of equality.
It remains to connect that case with independence. If $f = f_Xf_Y$ at every point where both are continuous, the two functions differ only on finitely many lines, so $f_Xf_Y$ is also a joint density of the pair. By Random Variables and Distribution Functions, Theorem 6, the product formula $P(X \in A, Y \in B) = P(X \in A)\,P(Y \in B)$ then holds for all intervals $A$ and $B$, and by the step from intervals to all sets that that page cites, $X$ and $Y$ are independent. Conversely, if $X$ and $Y$ are independent, the same theorem says that $f_Xf_Y$ is a joint density of the pair. Two joint densities $f_1$ and $f_2$ of the same pair agree at every point where both are continuous. If they differed at such a point, say $f_1 - f_2 = 2\delta > 0$ there, then by continuity $f_1 - f_2 > \delta$ on a small square around the point, and the integrals of $f_1$ and $f_2$ over the square would differ by at least $\delta$ times its area. But both integrals are the probability that $(X, Y)$ falls in the square. So $f = f_Xf_Y$ at every point where both are continuous.
(b) Split the logarithm: $\ln\frac{f(x, y)}{f_X(x)f_Y(y)} = \ln f(x, y) - \ln f_X(x) - \ln f_Y(y)$. Multiply by $f(x, y)$ and integrate. The first term gives $-h(X, Y)$. The second gives $-\iint f(x, y)\ln f_X(x)\,dy\,dx = -\int f_X(x)\ln f_X(x)\,dx = h(X)$, after the integration over $y$, and the third gives $h(Y)$ in the same way with the roles of $x$ and $y$ exchanged. The other two forms follow from the chain rule $h(X, Y) = h(X) + h(Y \mid X) = h(Y) + h(X \mid Y)$. The two inequalities are $I \ge 0$ rewritten. The form $I(X; Y) = h(Y) - h(Y \mid X)$ needs only those two entropies to be finite, not $h(X)$ or $h(X, Y)$: split the logarithm instead as $\ln\frac{f(x, y)}{f_X(x)f_Y(y)} = \ln f(y \mid x) - \ln f_Y(y)$, multiply by $f(x, y)$ and integrate.
(c) Suppose first that $S$ and $T$ are increasing. Then $P(U \le u, V \le v) = P\big(X \le S^{-1}(u),\, Y \le T^{-1}(v)\big)$, and differentiating once in $u$ and once in $v$ gives the joint density of $(U, V)$ at $u = S(x)$, $v = T(y)$ as $f(x, y)/(S'(x)\,T'(y))$. If a map is decreasing, the corresponding inequality is reversed and the derivative is negative, and the two changes of sign leave $f(x, y)/(\lvert S'(x)\rvert\,\lvert T'(y)\rvert)$. By the change of variables formula on the line the marginal densities are $f_X(x)/\lvert S'(x)\rvert$ and $f_Y(y)/\lvert T'(y)\rvert$. The factors cancel in the ratio of the joint density to the product of the marginals. Substituting $u = S(x)$ and $v = T(y)$ in the double integral for $I(U; V)$, with $du\,dv = \lvert S'(x)\rvert\,\lvert T'(y)\rvert\,dx\,dy$, turns it into the double integral for $I(X; Y)$.
(d) Let $R = B_i \times B'_j$ be one of the rectangles, with $P_{ij} = \iint_R f$. The product of the marginals has integral $\int_{B_i} f_X\,dx \cdot \int_{B'_j} f_Y\,dy = P_i\,P'_j$ over $R$, the product of the probabilities of the two intervals. So the binned product of the marginals is the product of the binned marginals, which is what makes rectangles the right bins. If $P_{ij} > 0$ then $P_i P'_j > 0$, since $P_i \ge P_{ij}$ and $P'_j \ge P_{ij}$. If the integral of $f\ln\frac{f}{f_Xf_Y}$ over $R$ is $+\infty$ there is nothing to prove for this rectangle. Otherwise let $W$ be a random point of $R$ with density $f_X f_Y/(P_iP'_j)$, and let $r = f/(f_Xf_Y)$, so that $E[r(W)] = P_{ij}/(P_iP'_j)$. By Jensen's inequality for the convex function $\psi(t) = t\ln t$, applied to the random variable $r(W)$,
$$\iint_R f\ln\frac{f}{f_Xf_Y}\,dx\,dy = P_iP'_j\,E\big[\psi(r(W))\big] \ge P_iP'_j\,\psi\!\left(\frac{P_{ij}}{P_iP'_j}\right) = P_{ij}\ln\frac{P_{ij}}{P_iP'_j}.$$
A rectangle with $P_{ij} = 0$ contributes 0 on both sides, because $f$ is then zero at every point of continuity in it. As in the proof for the line, $I(X; Y)$ is the sum of the left sides over all rectangles: the integrand is the excess minus $f_Xf_Y$ plus $f$, the excess is nonnegative, and the other two terms sum to $-1 + 1$. The sum of the right sides is $I(X^B; Y^B)$. $\square$
More than (d) is true: the mutual information is the least upper bound of the mutual information between binned versions of $X$ and $Y$ (Polyanskiy and Wu, 2025, equation 4.29; cited, not proved here). The entropies show why no correction term appears. Binning $X$ and $Y$ with width $\Delta$ adds $-\ln\Delta$ to $H(X^\Delta)$ and to $H(Y^\Delta)$ and $-2\ln\Delta$ to the joint entropy, and the three corrections cancel in $H(X^\Delta) + H(Y^\Delta) - H(X^\Delta, Y^\Delta)$.
So the mutual information between two continuous variables is a number of bits in the full discrete sense: it is what any sufficiently fine pair of measuring scales would find.
The bivariate normal
Let $X$ and $Z$ be independent standard normal variables, let $-1 < \rho < 1$, and let
$$Y = \rho X + \sqrt{1 - \rho^2}\,Z.$$
Then $Y$ is standard normal, because a sum of independent normal variables is normal with the variances added (Moment Generating and Characteristic Functions), and $\rho^2 + (1 - \rho^2) = 1$. The correlation of $X$ and $Y$ is $E[XY] = \rho E[X^2] + \sqrt{1 - \rho^2}\,E[XZ] = \rho$. Given $X = x$, the variable $Y$ is $\rho x$ plus $\sqrt{1 - \rho^2}$ times a standard normal variable that is independent of $X$, so its conditional density is normal with mean $\rho x$ and variance $1 - \rho^2$. In terms of densities, the joint density of $(X, Z)$ is $\phi(x)\phi(z)$. Substituting $z = (y - \rho x)/\sqrt{1 - \rho^2}$ at fixed $x$, with $dz = dy/\sqrt{1 - \rho^2}$, gives the joint density of $(X, Y)$ as $\phi(x)$ times that normal density in $y$. It is continuous everywhere, and its marginals are the standard normal density, so it is a joint density in the sense of this section. By formula (1) the conditional entropy is the same for every $x$:
$$h(Y \mid X) = \tfrac12\ln\big(2\pi e\,(1 - \rho^2)\big), \qquad h(Y) = \tfrac12\ln(2\pi e).$$
Therefore
$$I(X; Y) = h(Y) - h(Y \mid X) = -\tfrac12\ln(1 - \rho^2).$$
By the invariance under changes of unit, the same value holds after each variable is shifted and rescaled, that is, for every pair that is jointly normal with correlation $\rho$. It is 0 for $\rho = 0$, 0.2075 bits for $\rho = 0.5$, 1.1980 bits for $\rho = 0.9$, and grows without bound as $\lvert\rho\rvert \to 1$: a variable shares infinitely many bits with an exact copy of itself, which is the divergence of the bin entropies seen once more.
Two readings on one adult. For 4,811 of the adults both a first and a second reading were recorded. Their correlation is 0.9662. If the pair were jointly normal, the mutual information would be $-\tfrac12\log_2(1 - 0.9662^2) = -\tfrac12\log_2 0.0665 = 1.96$ bits. That is a statement about a model. Computed directly from the readings sorted into bins, by the plug-in method of Mutual Information:
| Bin width (mm Hg) | Occupied cells | Mutual information of the binned readings, bits |
|---|---|---|
| 32 | 16 | 0.79 |
| 16 | 34 | 1.27 |
| 8 | 84 | 1.64 |
| 4 | 234 | 1.86 |
| 2 | 645 | 2.02 |
The coarse bins lose information, as binning must. Finer bins approach the model's 1.96 bits, and the finest, which are the even-numbered readings themselves, pass it. That last value should not be read as evidence against the model. A plug-in estimate from 4,811 pairs spread over 645 cells is biased, and Mutual Information shows in its section "Measuring it from counts" that the bias of such an estimate can run either way. Here it runs upward, and its size can be estimated in two ways. A bootstrap resamples the 4,811 pairs with replacement and recomputes the plug-in value; over 30 seeded resamples the value exceeds the original by a median of 0.07 bits (interquartile range 0.061 to 0.094), and that excess estimates the bias of the original. A count of the cells gives a similar figure. The plug-in entropy of $N$ observations from a distribution on $m$ cells is too low on average by about $(m - 1)/(2N)$ nats when $m$ is fixed and $N$ is large (Paninski, 2003, equation 4.6). Adding that amount back, with the number of occupied cells in place of $m$, is what Paninski (2003, Section 3) calls the Miller–Madow bias correction. Applied to each of the three entropies in $I = H(X) + H(Y) - H(X, Y)$, it puts the upward bias of a plug-in mutual information at about (occupied cells − occupied rows − occupied columns + 1)/(2 × pairs) nats. The table has 645 occupied cells in 68 occupied rows and 68 occupied columns, so the estimate is $(645 - 68 - 68 + 1)/(2 \times 4{,}811)$ nats, which is 0.076 bits. It is a first-order approximation for many observations per cell, and with about 7.5 pairs per occupied cell it is a rough one. Subtracting the bootstrap estimate brings the finest value to about 1.95 bits, beside the model's 1.96. Both corrections are estimates, not proofs. No bin width is both fine enough to lose nothing and coarse enough to be estimated well from this many pairs. The model and the counts agree that a first reading is worth about 2 bits about the second.
What is proved, what is cited, what is computed
Proved here: the three entropies in closed form; the change of variables theorem and its corollary; the limit of the binned entropy for a continuous density on a closed interval, and exactly for the uniform and the exponential; the chain rule; Gibbs' inequality for densities with its case of equality, on the line and, for joint densities continuous off finitely many horizontal and vertical lines, in the plane; the invariance of the divergence; that binning does not increase the divergence; that mutual information for such joint densities is nonnegative, vanishes exactly under independence, does not depend on the units and is not increased by binning; the divergence between two normal densities; that the normal maximizes differential entropy at a given variance, with the gap identified as a divergence; and the mutual information of the bivariate normal.
Borrowed, with the proof on a linked page: $\ln t \le t - 1$ (Cross-Entropy and Kullback–Leibler Divergence); the change of variables formula for densities (Random Variables and Distribution Functions); Jensen's inequality (Probability Inequalities); that sums of independent normal variables are normal (Moment Generating and Characteristic Functions). The criterion for independence in terms of joint densities is Theorem 6 of Random Variables and Distribution Functions, proved there for intervals; the extension from intervals to all sets is cited there.
Cited, not proved: the general form of the limit of binned entropy (as stated by Polyanskiy and Wu, who credit Rényi, 1959); divergence and mutual information for joint distributions outside the class of joint densities used here; that divergence and mutual information are least upper bounds over binnings; the formulas for random vectors; the first-order bias of a plug-in entropy, $(m - 1)/(2N)$ nats, behind the Miller–Madow correction (as stated by Paninski, 2003), used for the two readings; the examples of densities with no finite differential entropy.
Computed: every number about the blood pressures. The bias of the finest binned mutual information is an estimate from a bootstrap of 30 resamples (seeds 16,000,000 to 16,000,029, median and interquartile range) and from a count of cells; neither is a proof. The 5,247 adults are treated as a population, as on the course pages; they are a survey sample, and nothing here estimates a quantity for the United States. The plateau of the binned entropy and the binned mutual informations are estimates that depend on the bin width, and the page says where they break down.
Worked problems
These problems are posed for this article, on small exact densities and on the blood pressure data, one for each main result. The script tools/experiments/statistics/differential-entropy.py checks every number.
Problem 1 (a triangular density). $X$ has the density $f(x) = x$ for $0 \le x \le 1$ and $f(x) = 2 - x$ for $1 \le x \le 2$. Find $h(X)$, the variance of $X$, and the divergence of $X$ from the normal distribution with its mean and variance.
Solution. $h(X) = \tfrac12$ nat, the variance is $\tfrac16$, and the divergence is 0.0231 nats.
The density is symmetric about 1, so the two halves contribute equally to the entropy: $h(X) = -2\int_0^1 x\ln x\,dx$. Integrating by parts, $\int_0^1 x\ln x\,dx = \big[\tfrac{x^2}{2}\ln x\big]_0^1 - \int_0^1 \tfrac{x}{2}\,dx = 0 - \tfrac14$, where the bracket vanishes at 0 because $x^2\ln x \to 0$. So $h(X) = \tfrac12$. The mean is 1 by symmetry, and the variance is $2\int_0^1 (1 - x)^2 x\,dx = 2\int_0^1 (x - 2x^2 + x^3)\,dx = 2\big(\tfrac12 - \tfrac23 + \tfrac14\big) = \tfrac16$. The normal distribution with variance $\tfrac16$ has $h = \tfrac12\ln\frac{2\pi e}{6} = 0.5231$ nats by formula (1). By identity (4) the divergence from it is $0.5231 - 0.5 = 0.0231$ nats. The triangle is closer to a normal curve than the Laplace density is (0.0724), and much closer than the uniform (0.1765).
Problem 2 (bins that do not line up). $X$ is uniform on $[0, \tfrac34]$. Compute $H(X^\Delta) + \ln\Delta$ for $\Delta = \tfrac12$ and for $\Delta = \tfrac14$, and compare with $h(X)$.
Solution. $h(X) = \ln\tfrac34 = -0.2877$ nats. With $\Delta = \tfrac14$ the binned value reproduces it exactly, and with $\Delta = \tfrac12$ it gives $-0.0566$.
With $\Delta = \tfrac14$ the three bins $[0, \tfrac14)$, $[\tfrac14, \tfrac12)$, $[\tfrac12, \tfrac34)$ each have probability $\tfrac13$. So $H = \ln 3$ and $H + \ln\tfrac14 = \ln\tfrac34$. With $\Delta = \tfrac12$ the bin $[0, \tfrac12)$ has probability $\tfrac23$ and the bin $[\tfrac12, 1)$, which the interval only half fills, has probability $\tfrac13$. Then $H = -\tfrac23\ln\tfrac23 - \tfrac13\ln\tfrac13 = 0.6365$ nats, and $H + \ln\tfrac12 = 0.6365 - 0.6931 = -0.0566$. The difference from $-0.2877$ is the edge term in the proof of the theorem on fine bins: a bin cut by the end of the interval does not have probability $f(x)\Delta$. The theorem says such terms vanish as $\Delta \to 0$. It does not say the approximation is good at a given $\Delta$, and for a density with jumps at the ends it converges unevenly.
Problem 3 (changing the temperature scale). A temperature $C$ in degrees Celsius has differential entropy $h(C)$. What is the differential entropy of the same temperature in degrees Fahrenheit, $F = 1.8\,C + 32$? What happens to the mutual information between two temperatures, and to the divergence between two distributions of temperature?
Solution. $h(F) = h(C) + \ln 1.8 = h(C) + 0.5878$ nats, which is 0.8480 bits more. The mutual information and the divergence do not change.
By the corollary on shift and scale, the shift by 32 has no effect and the factor 1.8 adds $\ln 1.8$. A Fahrenheit degree is smaller, so the same distribution is 1.8 times as wide when counted in degrees, and its effective width is multiplied by 1.8. By the theorem on fine bins, recording a temperature to the nearest degree therefore takes about 0.848 bits more in Fahrenheit, when one degree is small compared with the spread of the temperatures, because the nearest Fahrenheit degree is a finer precision. The theorem is a limit, so the figure is approximate at any actual bin width. If two temperatures are both converted, their mutual information is unchanged, by part (c) of the theorem on mutual information. The divergence between two distributions of temperature is unchanged, by the invariance theorem. Statements of the form "this distribution has differential entropy 2.3" mean nothing until the unit is given; statements about divergence and mutual information need none.
Problem 4 (what the first reading tells about the second). Under the jointly normal model for the 4,811 pairs of readings, with correlation 0.9662 and a standard deviation of 19.70 mm Hg for the second reading, find the differential entropy of the second reading, its conditional differential entropy given the first, and the effective widths that correspond to them.
Solution. $h = 6.35$ bits, $h(\text{second} \mid \text{first}) = 4.39$ bits, and the difference is the mutual information, 1.96 bits. The effective width falls from 81 mm Hg to 21.
By formula (1) in bits, $h = \tfrac12\log_2(2\pi e \times 19.70^2) = 6.35$. Under the model the conditional distribution of the second reading given the first is normal with variance $\sigma^2(1 - \rho^2)$, by the construction of the bivariate normal rescaled to standard deviation $\sigma$. Its standard deviation is $19.70 \times \sqrt{0.0665} = 5.08$ mm Hg, and its entropy is $\tfrac12\log_2(2\pi e \times 5.08^2) = 4.39$ bits. The difference, $6.35 - 4.39 = 1.96$ bits, is $-\tfrac12\log_2(1 - \rho^2)$. The effective widths are $4.1327 \times 19.70 = 81$ mm Hg and $4.1327 \times 5.08 = 21$ mm Hg: knowing the first reading narrows the second by a factor of $1/\sqrt{1 - \rho^2} = 2^{1.96} = 3.9$. Both entropies depend on the unit. Their difference does not.
Problem 5 (the Gaussian channel). A signal $X$ with a density (in the sense of this article: continuous except at finitely many points) and with $E[X^2] \le P$ is received as $Y = X + Z$, where the noise $Z$ is normal with mean 0 and variance $N$ and is independent of $X$. Show that $I(X; Y) \le \tfrac12\ln(1 + P/N)$, with equality when $X$ is normal with mean 0 and variance $P$. Evaluate the bound for $P/N = 1$ and $P/N = 15$.
Solution. The bound is 0.5000 bits for $P/N = 1$ and 2.0000 bits for $P/N = 15$.
Write $\phi_N$ for the density of the noise. Given $X = x$, the received value is $x + Z$, whose density is the noise density shifted by $x$. So the joint density of $(X, Y)$ is $f_X(x)\,\phi_N(y - x)$, which is continuous except on the vertical lines through the discontinuities of $f_X$, and $h(Y \mid X = x) = h(Z)$ for every $x$. Hence $h(Y \mid X) = h(Z) = \tfrac12\ln(2\pi e N)$ and $I(X; Y) = h(Y) - h(Y \mid X) = h(Y) - h(Z)$, a form that needs only $h(Y)$ and $h(Y \mid X)$ to be finite.
The density of $Y$ is $f_Y(y) = \int f_X(x)\,\phi_N(y - x)\,dx$, an average of values of $\phi_N$. It is continuous, because $\phi_N$ is uniformly continuous: moving $y$ a little moves every $\phi_N(y - x)$ by at most the same small amount. And it is at most the largest value of $\phi_N$, which is $1/\sqrt{2\pi N}$. So $-\ln f_Y \ge \tfrac12\ln(2\pi N)$ everywhere, and $h(Y) \ge \tfrac12\ln(2\pi N)$: the integral that defines $h(Y)$ has a value, finite or $+\infty$, and it is not $-\infty$. It is not $+\infty$ either. Because $X$ and $Z$ are independent, $\operatorname{Var}(Y) = \operatorname{Var}(X) + N \le E[X^2] + N \le P + N$, which is finite and at least $N > 0$. Let $g$ be the normal density with the mean and variance of $Y$. By the proof of Gibbs' inequality, $D(f_Y \,\|\, g) = \int e \ge 0$, and $-\int f_Y\ln g = \tfrac12\ln\big(2\pi e\operatorname{Var}(Y)\big)$ as in the proof of the theorem on the normal. So $-h(Y) = D(f_Y \,\|\, g) + \int f_Y\ln g \ge -\tfrac12\ln\big(2\pi e\operatorname{Var}(Y)\big)$, which is finite. Hence $h(Y)$ is finite, and the theorem on the normal's maximal entropy applies to $Y$. By the theorem on the normal's maximal entropy, $h(Y) \le \tfrac12\ln\big(2\pi e\operatorname{Var}(Y)\big) \le \tfrac12\ln\big(2\pi e(P + N)\big)$. Therefore
$$I(X; Y) \le \tfrac12\ln\big(2\pi e(P + N)\big) - \tfrac12\ln(2\pi e N) = \tfrac12\ln\left(1 + \frac{P}{N}\right).$$
If $X$ is normal with mean 0 and variance $P$, then $Y$ is normal with variance $P + N$, both inequalities are equalities, and the bound is attained. In bits, $\tfrac12\log_2 2 = 0.5$ and $\tfrac12\log_2 16 = 2$.
For comparison the script evaluates $h(Y) - h(Z)$ numerically for two other inputs of the same power. A signal uniform on an interval gives 0.4962 bits at $P/N = 1$ and 1.8931 bits at $P/N = 15$. A signal that takes only the two values $\pm\sqrt{P}$, each with probability $\tfrac12$, gives 0.4859 bits and 0.9998 bits. (A two-valued signal has no density; for it the same identity holds with $h(Y \mid X)$ averaged over the two values, a mixed case that this article does not develop.) At low power all three inputs are nearly as good. At high power the two-valued signal stays just under 1 bit, which is the entropy of a fair coin, while the normal input reaches 2. (That a discrete input cannot convey more than its own entropy is the bound $I(X; Y) \le H(X)$ of Mutual Information, proved there for two discrete variables; for this mixed case it is observed in the computed values, not proved.)
Shannon (1948, Section 25, Theorem 17) obtained the corresponding result for a channel of bandwidth $W$, a capacity of $W\log\frac{P + N}{N}$ per second, by the same argument: the received signal has the greatest entropy for its power when it is Gaussian. He showed that this is the capacity of the channel, the greatest rate at which binary digits can be sent with an arbitrarily small frequency of errors. That second statement is the noisy channel coding theorem, which is cited here, not proved.
What we have learned
A variable with a density has no finite entropy: binned ever more finely, its entropy grows by one bit for each halving of the bins. The differential entropy $h(X) = -\int f\ln f$ is the part of that growth that belongs to the distribution, the limit of the bin entropy plus the logarithm of the bin width. It is the logarithm of an effective width, so it can be negative and it changes with the unit of measurement. What carries over from the discrete case in full is everything that compares two distributions. The divergence between densities is nonnegative, vanishes only for equal densities, does not depend on the coordinates, and is approached from below by the divergence between binned versions; mutual information inherits all of that. Among distributions with a given variance the normal has the greatest differential entropy, $\tfrac12\ln(2\pi e\sigma^2)$, and the shortfall of any other is its divergence from the normal. For the blood pressures that shortfall is about 0.12 bits.
The proof of that last theorem used only one fact about the normal density, that its logarithm is a quadratic function. Maximum Entropy Distributions turns that observation into a method: whenever the logarithm of a density is a combination of the functions whose averages are fixed, that density has the greatest entropy under those constraints.
References
- Cover, T. M., and Thomas, J. A. (2006). Elements of Information Theory, 2nd edition. Wiley. Chapter 8, "Differential Entropy" (Section 8.3, "Relation of Differential Entropy to Discrete Entropy"; Section 8.5, "Relative Entropy and Mutual Information"). record · 8 pages
- National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018: Demographic Variables and Sample Weights (DEMO_J) and Blood Pressure (BPX_J) data files. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 67 pages
- Paninski, L. (2003). Estimation of entropy and mutual information. Neural Computation, 15(6), 1191–1253. Section 3, "Previous Work" (the Miller–Madow bias correction), and equation 4.6. record · 2 pages
- Polyanskiy, Y., and Wu, Y. (2025). Information Theory: From Coding to Learning. Cambridge University Press. doi:10.1017/9781108966351. Read in the authors' prepublication version (file dated August 16, 2024), which states that its equation and theorem numbers match those of the printed book. Section 2.3, "Differential entropy" (Definition 2.6, Theorems 2.7 and 2.8, equations 2.18 and 2.21); Theorem 4.5 and equation 4.29. record
- Rényi, A. (1959). On the dimension and entropy of probability distributions. Acta Mathematica Academiae Scientiarum Hungaricae, 10(1–2), 193–215. record
- Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical Journal, 27(3), 379–423, and 27(4), 623–656. Section 20, "Entropy of a continuous distribution", and Section 25, "Channel capacity with an average power limitation". record · 6 pages