Maximum Entropy Distributions

Which distribution has the greatest entropy among all that satisfy given constraints, and why it always has the same form. One theorem, proved by the divergence argument, shows that a distribution whose log-probability is a combination of the constrained statistics is the maximizer; the uniform, geometric, exponential and normal distributions are its instances, and the general solution is an exponential family. The maximizer is also the safest model, the softmax function is the maximum entropy distribution under an expected-score constraint, and fitting such a model by maximum likelihood is the same problem seen from the other side. The running examples are the gaps between Reuters stories that contain a term and the characters that follow a capital T in the Tiny Shakespeare text. Worked problems on a die with a given mean, a density on an interval with a given mean, a bound on a mean, and a geometric code.

In the Reuters-21578 collection the term oil occurs in 1,038 stories, the last of them story 10,782, so the gaps between consecutive stories that contain it average $10{,}782/1{,}038 = 10.39$. Index Compression must choose a code for those gaps, and Perplexity and Source Coding shows that choosing a code is choosing a distribution. If the mean gap is all that is known, which distribution on the positive integers should be assumed? The one of greatest entropy, which is the geometric distribution with that mean. It has 4.75 bits of entropy, and no distribution with mean 10.39 has more. An ideal code built for it, one whose lengths are the logarithms of its probabilities without rounding, costs 4.75 bits per gap on average whatever the true distribution with that mean is.

A maximum entropy distribution for a set of constraints is a distribution that satisfies the constraints and has entropy at least as great as every other distribution that does. The constraints in this article are of one kind: the expected values of chosen functions of the outcome, such as its mean or its mean square.

First we state the principle and its simplest case, no constraint at all. Then we prove one theorem that covers every case: a distribution whose log-probability is a linear combination of the constrained functions is the maximizer. Next we read off the geometric, uniform, exponential and normal distributions, and identify the general solution as an exponential family. Then we show that the maximizer is the safest model in a precise sense. Finally we treat the softmax function as a maximum entropy distribution, and prove that fitting it by maximum likelihood solves the same problem from the other side. The article assumes Entropy and Gibbs' inequality from Cross-Entropy and Kullback–Leibler Divergence; the sections on densities use Differential Entropy and the Gaussian.

Notation

Notation follows the Information Theory unit. $p$ and $q$ are mass functions on an alphabet $\mathcal{X}$, $H(p)$ is the entropy, $H(p, q) = -\sum_x p(x)\ln q(x)$ the cross-entropy and $D(p \,\|\, q) = H(p, q) - H(p)$ the Kullback–Leibler divergence. For densities $f$ and $g$, $h(f) = -\int f\ln f$ is the differential entropy and $D(f \,\|\, g) = \int f\ln\frac fg$. Calculus is done in nats, with $\ln$; where a code length is meant the unit is bits, with $\log_2$, and one nat is 1.4427 bits. $E_p[\cdot]$ is an expectation under $p$. The geometric distribution has the parametrization of Entropy: $P(X = k) = (1 - \theta)^{k-1}\theta$ for $k = 1, 2, \dots$, with mean $1/\theta$.

The principle, and the case of no constraint

Suppose all that is known about a random quantity is a short list of facts. Many distributions agree with the facts. The principle of maximum entropy says to use the one with the greatest entropy. Jaynes (1957) proposed it as a general rule of inference. His abstract says of the maximum entropy estimate: "It is the least biased estimate possible on the given information; i.e., it is maximally noncommittal with regard to missing information." Shannon (1948, Section 10) had used the idea for codes. When only some of the statistics of a source are to be used in designing a code, he writes, "we consider the source with the maximum entropy subject to the statistical conditions we wish to retain."

The principle is a choice, not a theorem. What can be proved is which distribution it selects, and one property of that distribution, proved below, that makes the choice a safe one.

With no facts at all on a finite alphabet of $n$ outcomes, the answer is the uniform distribution. Entropy proves it as the upper bound $H(p) \le \ln n$, by the identity

$$D(p \,\|\, u) = \ln n - H(p) \ge 0,$$

where $u$ is the uniform mass function. The proof has a shape worth noticing, because every proof in this article repeats it. The cross-entropy of any $p$ against $u$ is $-\sum_x p(x)\ln\frac1n = \ln n$, the same for every $p$, and equal to the entropy of $u$ itself. So the entropy of $p$ falls short of the entropy of $u$ by exactly a divergence, which cannot be negative.

The theorem

Constraints on averages change the answer in a way that can be guessed from the uniform case. The uniform distribution works because its log-probability is constant, so that every distribution has the same cross-entropy against it. If the mean of $X$ is fixed, the same will be true of a distribution whose log-probability is a constant plus a multiple of $x$: its cross-entropy involves $p$ only through the mean of $X$, which all the candidates share.

Theorem (maximum entropy under expectation constraints). Let $\mathcal{X}$ be a finite or countably infinite alphabet, let $T_1, \dots, T_m$ be real functions on $\mathcal{X}$, and let $c_1, \dots, c_m$ be numbers. Let $\mathcal{P}$ be the set of mass functions $p$ on $\mathcal{X}$ with

$$E_p[T_j(X)] = \sum_x p(x)\,T_j(x) = c_j \quad \text{for } j = 1, \dots, m.$$

Suppose there are numbers $\lambda_1, \dots, \lambda_m$ for which $Z = \sum_x \exp\big(\sum_j \lambda_j T_j(x)\big)$ is finite and the mass function

$$q(x) = \frac{1}{Z}\exp\Big(\sum_{j=1}^{m}\lambda_j T_j(x)\Big) \qquad (1)$$

belongs to $\mathcal{P}$. Then every $p$ in $\mathcal{P}$ satisfies

$$H(p) \le H(q) = \ln Z - \sum_j \lambda_j c_j,$$

with equality only for $p = q$.

Proof. Take any $p$ in $\mathcal{P}$. By (1), $\ln q(x) = \sum_j \lambda_j T_j(x) - \ln Z$, so

$$H(p, q) = -\sum_x p(x)\ln q(x) = \ln Z - \sum_j \lambda_j\sum_x p(x)\,T_j(x) = \ln Z - \sum_j \lambda_j c_j. \qquad (2)$$

The right side does not depend on which member of $\mathcal{P}$ was taken. In particular it holds for $p = q$, where the cross-entropy is the entropy: $H(q) = \ln Z - \sum_j\lambda_j c_j$. So $H(p, q) = H(q)$ for every $p$ in $\mathcal{P}$, and

$$H(q) - H(p) = H(p, q) - H(p) = D(p \,\|\, q) \ge 0 \qquad (3)$$

by Gibbs' inequality, with equality exactly when $p = q$ (Cross-Entropy and Kullback–Leibler Divergence, section "Gibbs' inequality"). The linked theorem is stated for a finite alphabet. On a countably infinite alphabet the step is justified as follows. By the inequality $\ln t \le t - 1$, each term of the series $\sum_x p(x)\ln\frac{q(x)}{p(x)}$ is at most $q(x) - p(x)$, and the series $\sum_x \big(q(x) - p(x)\big)$ converges, to $1 - 1 = 0$. So the first series either converges or diverges to $-\infty$, and in both cases its value is at most 0. That series is $H(p) - H(p, q)$, and $H(p, q)$ is finite by (2). If $H(p)$ were infinite, the value would be $+\infty$, which is not at most 0. So every $p$ in $\mathcal{P}$ has finite entropy, and $H(p) \le H(p, q)$. For equality, the differences $\big(q(x) - p(x)\big) - p(x)\ln\frac{q(x)}{p(x)}$ are nonnegative and must sum to 0, so each is 0, which by the case of equality in $\ln t \le t - 1$ means $q(x) = p(x)$ for every $x$. $\square$

Three remarks fix what the theorem does and does not say.

  • The gap is a divergence. Identity (3) is stronger than the inequality. Every distribution that meets the constraints has entropy below the maximum by exactly its divergence from the maximizer.
  • The theorem is a sufficient condition. It says that if a distribution of the form (1) meets the constraints, it is the unique maximizer. It does not say that one exists. The section "When form (1) does not exist" gives constraints for which none does; the entropy then has no maximum, or a maximum that is not of form (1). The softmax section proves existence for one constraint on a finite alphabet.
  • The form is forced by the method, not assumed for convenience. Jaynes (1957, Section 2, equations 2-9 to 2-13) reached (1) by Lagrange multipliers. He writes the solution as $p_i = \exp\{-[\lambda_0 + \lambda_1 f_1(x_i) + \cdots + \lambda_m f_m(x_i)]\}$ and calls $Z$ the partition function. He gives the maximum entropy as $\lambda_0 + \lambda_1\langle f_1\rangle + \cdots + \lambda_m\langle f_m\rangle$ with $\lambda_0 = \ln Z$. That is the value in the theorem, with his opposite sign for the multipliers. The calculus shows where to look. The divergence argument is what proves that the stationary point is the maximum, with no second-order conditions to check.

Corollary (inequality constraints). Suppose some constraints are relaxed to $E_p[T_j(X)] \le c_j$. If $q$ of the form (1) meets every constraint with equality and $\lambda_j \le 0$ for each relaxed constraint, then $q$ still has the greatest entropy.

Proof. For $p$ in the larger set, (2) becomes $H(p, q) = \ln Z - \sum_j\lambda_j E_p[T_j]$. For a relaxed constraint, $E_p[T_j] \le c_j$ and $\lambda_j \le 0$ give $-\lambda_j E_p[T_j] \le -\lambda_j c_j$, so $H(p, q) \le \ln Z - \sum_j \lambda_j c_j = H(q)$. Then $H(p) \le H(p, q) \le H(q)$ by Gibbs' inequality. $\square$

The script tests the theorem on an alphabet of six outcomes with two constraints. It draws 200 distributions at random that satisfy both constraints. All have less entropy than the distribution of form (1), and in each case the shortfall equals the divergence to twelve decimal places.

A given mean on the positive integers: the geometric distribution

Take $\mathcal{X} = \{1, 2, 3, \dots\}$ and one constraint, $E[X] = \mu$ with $\mu > 1$. Form (1) is $q(k) = e^{\lambda k}/Z$. The sum $Z = \sum_{k \ge 1} e^{\lambda k}$ is finite exactly when $\lambda < 0$. Writing $r = e^{\lambda}$, with $0 < r < 1$, gives $Z = r/(1 - r)$ and

$$q(k) = (1 - r)\,r^{k-1},$$

the geometric distribution with $\theta = 1 - r$. Its mean is $1/\theta$, so the constraint is met by $\theta = 1/\mu$, which is possible for every $\mu > 1$. By the theorem:

Among all distributions on the positive integers with mean $\mu$, the geometric distribution with $\theta = 1/\mu$ has the greatest entropy. That entropy is $\big[-\theta\ln\theta - (1 - \theta)\ln(1 - \theta)\big]/\theta$ (Entropy, Problem 3), which in terms of the mean is

$$H = \mu\ln\mu - (\mu - 1)\ln(\mu - 1) \ \text{nats}.$$

Since $\lambda = \ln(1 - \theta) < 0$, the corollary applies as well: the same distribution has the greatest entropy among all distributions with mean at most $\mu$.

On the gaps of oil. With $\mu = 10.39$, $\theta = 0.0963$ and the geometric distribution has 4.75 bits of entropy. The 1,038 gaps take 60 distinct values, and the entropy of their own frequencies is 4.66 bits. The difference, 0.09 bits, is by identity (3) the divergence of the list's gap distribution from the geometric one. It is small, and the figure shows where it comes from.

The share of the 1,038 gaps of oil that exceed g (solid steps), against the geometric distribution with the same mean (dashed), on a logarithmic scale where a geometric distribution is a straight line. Notice that up to about 20 the gaps run slightly below the line, and beyond it they stay above: both very short and long gaps are more common than the geometric distribution allows.
The share of the 1,038 gaps of oil that exceed g (solid steps), against the geometric distribution with the same mean (dashed), on a logarithmic scale where a geometric distribution is a straight line. Notice that up to about 20 the gaps run slightly below the line, and beyond it they stay above: both very short and long gaps are more common than the geometric distribution allows.

The list has more gaps of exactly 1 than the geometric distribution predicts (13.9 percent against 9.6) and more long gaps (0.77 percent above 60, against 0.23; the largest gap is 106). Stories about oil come in runs, with quiet stretches between them. A geometric distribution describes a term scattered through the collection at random. It is the model behind the Golomb codes of Index Compression, which that article reports are optimal prefix codes for geometrically distributed gaps when the parameter is well chosen, and behind the Rice codes, which it says come close. The Rice code it fits to this list spends 4.82 bits per gap, close to the 4.75 that the geometric model allows.

Example (pooling terms destroys the fit). For all 725,793 gaps of the collection pooled together, the mean gap is 289.7 and the geometric distribution with that mean has 9.62 bits of entropy. The pooled gaps have 7.19 bits, 2.43 bits less. Nothing contradicts the theorem, which says only that 9.62 is an upper bound. But a bound that loose shows the mean is a poor summary of the pooled gaps. They mix the short gaps of frequent terms with the long gaps of rare ones. Maximum entropy returns the best distribution for the stated facts; it does not say the stated facts are the right ones.

Example (the Poisson distribution is not the answer). On the counts $0, 1, 2, \dots$ with a given mean $m$, it is tempting to expect the Poisson distribution, the standard model of a count with a given mean. The theorem says otherwise. Form (1) with $T(k) = k$ on $\{0, 1, 2, \dots\}$ is again geometric, $q(k) = (1 - r)r^k$ with $r = m/(m + 1)$, and its entropy is $(m + 1)\ln(m + 1) - m\ln m$. For $m = 10$ that is 3.3510 nats, against 2.5614 nats for the Poisson distribution with mean 10; for $m = 1$ it is 1.3863 against 1.3048. The Poisson distribution does solve a problem of this kind, with a different starting point, which the section on exponential families identifies.

Densities: the uniform, the exponential and the normal

For a variable with a density the theorem and its proof are unchanged, with integrals for sums, once the entropy is the differential entropy and Gibbs' inequality is the one for densities. Both are established in Differential Entropy and the Gaussian, which also explains what differential entropy does and does not measure.

Theorem (densities). Let $S$ be an interval, bounded or not, and let $\mathcal{P}$ be the set of densities $f$ that vanish outside $S$, have finite differential entropy, and satisfy $E_f[T_j(X)] = c_j$ for $j = 1, \dots, m$. Suppose there are $\lambda_1, \dots, \lambda_m$ for which $Z = \int_S \exp\big(\sum_j\lambda_j T_j(x)\big)\,dx$ is finite and the density $g(x) = \frac1Z\exp\big(\sum_j \lambda_j T_j(x)\big)$ on $S$ belongs to $\mathcal{P}$. Then every $f$ in $\mathcal{P}$ has

$$h(f) \le h(g) = \ln Z - \sum_j\lambda_j c_j, \qquad h(g) - h(f) = D(f \,\|\, g),$$

with equality exactly when $f = g$ at every point where both are continuous.

Proof. $g$ is positive on $S$ and $f$ vanishes outside $S$, so $D(f \,\|\, g)$ is defined. As in (2), $-\int_S f\ln g\,dx = \ln Z - \sum_j \lambda_j E_f[T_j] = \ln Z - \sum_j\lambda_j c_j$ for every $f$ in $\mathcal{P}$, including $g$, for which it is $h(g)$. Hence $h(g) - h(f) = -\int f\ln g + \int f\ln f = D(f \,\|\, g) \ge 0$, by Gibbs' inequality for densities, which also gives the case of equality. $\square$

Corollary (inequality constraints, densities). If some constraints are relaxed to $E_f[T_j(X)] \le c_j$, and $g$ of the stated form meets every constraint with equality and has $\lambda_j \le 0$ for each relaxed constraint, then $g$ still has the greatest differential entropy.

Proof. For $f$ in the larger set, $-\int_S f\ln g\,dx = \ln Z - \sum_j\lambda_j E_f[T_j]$. For a relaxed constraint, $E_f[T_j] \le c_j$ and $\lambda_j \le 0$ give $-\lambda_j E_f[T_j] \le -\lambda_j c_j$. So $-\int_S f\ln g\,dx \le \ln Z - \sum_j\lambda_j c_j = h(g)$. By Gibbs' inequality for densities, $h(f) = -\int f\ln f \le -\int_S f\ln g$. Together, $h(f) \le h(g)$. $\square$

Three choices of $S$ and of the constraints give three familiar distributions.

No constraint on a bounded interval: the uniform. With $S = [a, b]$ and no functions $T_j$, form (1) is a constant, $g = 1/(b - a)$, and

$$h(f) \le \ln(b - a)$$

for every density on $[a, b]$. On $[0, 1]$ the uniform density has $h = 0$; the density $6x(1 - x)$ has $h = -0.1251$ nats and the triangular density with its peak at $\tfrac12$ has $h = -0.1931$.

A given mean on the half line: the exponential. With $S = [0, \infty)$ and $E[X] = \mu$, form (1) is $g(x) = e^{\lambda x}/Z$, which can be normalized only for $\lambda < 0$: $Z = -1/\lambda$. Its mean is $-1/\lambda$, so $\lambda = -1/\mu$ and

$$g(x) = \frac{1}{\mu}e^{-x/\mu}, \qquad h(f) \le \ln Z - \lambda\mu = \ln\mu + 1.$$

Among densities on $[0, \infty)$ with mean 1, the exponential has $h = 1$ nat. The script integrates three others with the same mean: the half-normal density has $h = 0.9516$, the gamma density with shape 2 has 0.8841, and the uniform density on $[0, 2]$ has $\ln 2 = 0.6931$. In each case it confirms that the shortfall equals the divergence from the exponential, computed as a separate integral.

The exponential is the continuous form of the geometric result. Differential Entropy and the Gaussian shows that an exponential variable sorted into bins of equal width is exactly geometric. So the two maximum entropy distributions for a given mean, one on the integers and one on the half line, are the same distribution seen at two resolutions.

A given mean and variance on the line: the normal. With $S$ the whole line, $T_1(x) = x$ and $T_2(x) = x^2$, the constraints $E[X] = \mu$ and $E[X^2] = \sigma^2 + \mu^2$ fix the mean and the variance. Form (1) is $g(x) = \exp(\lambda_1 x + \lambda_2 x^2)/Z$, which can be normalized only for $\lambda_2 < 0$. Completing the square,

$$\lambda_1 x + \lambda_2 x^2 = \lambda_2\Big(x + \frac{\lambda_1}{2\lambda_2}\Big)^2 - \frac{\lambda_1^2}{4\lambda_2},$$

so $g$ is proportional to $\exp\big(-(x - m)^2/(2s^2)\big)$ with $m = -\lambda_1/(2\lambda_2)$ and $s^2 = -1/(2\lambda_2)$: a normal density with mean $m$ and variance $s^2$. The constraints are met by $\lambda_2 = -1/(2\sigma^2)$ and $\lambda_1 = \mu/\sigma^2$. So the normal distribution with mean $\mu$ and variance $\sigma^2$ has the greatest differential entropy among all distributions with that mean and variance, and since its entropy $\tfrac12\ln(2\pi e\sigma^2) = 1.4189 + \ln\sigma$ nats does not depend on $\mu$, among all distributions with that variance. Differential Entropy and the Gaussian proves this case directly and measures the shortfall for the uniform, Laplace and exponential shapes and for a population of blood pressures. Here it is one line of a table:

Outcomes What is fixed Maximum entropy distribution Its entropy, nats
$n$ outcomes nothing uniform $\ln n$
$1, 2, 3, \dots$ mean $\mu$ geometric, $\theta = 1/\mu$ $\mu\ln\mu - (\mu - 1)\ln(\mu - 1)$
interval $[a, b]$ nothing uniform $\ln(b - a)$
half line $[0, \infty)$ mean $\mu$ exponential $1 + \ln\mu$
whole line mean and variance $\sigma^2$ normal $\tfrac12\ln(2\pi e\sigma^2)$

The three continuous rows are results Shannon stated (1948, Section 20), for a variable limited to a volume, for a variable on a half line with a fixed first moment, and for a variable with a fixed standard deviation. Cover and Thomas (2006) treat the general theorem and these examples in their Chapter 12, "Maximum Entropy".

When form (1) does not exist

The theorem needs a distribution of form (1) that meets the constraints. Sometimes there is none. Then the entropy may have no maximum, as in the first two examples, or a maximum that is not of form (1), as in the third.

Example (no constraint on an infinite alphabet). On the positive integers with no constraint, the uniform distribution on $\{1, \dots, n\}$ has entropy $\ln n$, which grows without bound. So no distribution of finite entropy is a maximizer. (Some distributions on the positive integers have infinite entropy, as Entropy notes for countable alphabets, and each of those attains the supremum trivially; they are not maximizers in any useful sense.) Form (1) would be a constant, which cannot be normalized over infinitely many outcomes.

Example (a mean alone on the whole line). Among densities on the whole line with mean 0, the normal densities with mean 0 have entropy $\tfrac12\ln(2\pi e\sigma^2)$, as large as desired. So no density of finite entropy is a maximizer. Form (1) would be $e^{\lambda x}/Z$, and $\int e^{\lambda x}\,dx$ over the whole line diverges for every $\lambda$. A mean limits the entropy on the half line, where large values are held back by the constraint. It does not on the line, where a distribution can spread in both directions and keep its mean.

Example (a constraint at the edge of what is possible). A die whose mean is 6 must show 6 every time. There is exactly one distribution in $\mathcal{P}$, and it is trivially the maximizer, but it is not of form (1), which gives every face a positive probability. The softmax section shows that it is the limit of form (1) as the multiplier grows.

A further caution applies to densities. Differential entropy depends on the unit and the scale of measurement, so the maximum entropy density does too. The uniform density on $[1, 4]$ is the maximum entropy density for $X$ on that interval, but then $\sqrt{X}$ on $[1, 2]$ is not uniform, although nothing was assumed about it either. The discrete problem has no such ambiguity. Jaynes (1957, p. 622, footnote 6) notes that the problems of the continuous case are "fundamentally more complicated" and restricts his paper to the discrete one. The remedy is the subject of the next section: measure a distribution against a stated reference distribution instead of against length.

The general solution is an exponential family

The entropy of $p$ on a finite alphabet is $\ln n - D(p \,\|\, u)$: maximizing entropy is minimizing the divergence from the uniform distribution. Replacing the uniform distribution by any reference distribution gives the general problem and the general solution.

Theorem (minimum divergence from a reference). Let $r$ be a mass function with $r(x) > 0$ on $\mathcal{X}$, and let $\mathcal{P}$ be as before. Suppose $Z = \sum_x r(x)\exp\big(\sum_j\lambda_j T_j(x)\big)$ is finite and

$$q(x) = \frac{1}{Z}\,r(x)\exp\Big(\sum_j\lambda_j T_j(x)\Big) \qquad (4)$$

belongs to $\mathcal{P}$. Then every $p$ in $\mathcal{P}$ has $D(p \,\|\, r) \ge D(q \,\|\, r) = \sum_j\lambda_j c_j - \ln Z$, with equality only for $p = q$. More precisely, $D(p \,\|\, r) = D(q \,\|\, r) + D(p \,\|\, q)$.

Proof. By (4), $\ln\frac{q(x)}{r(x)} = \sum_j\lambda_j T_j(x) - \ln Z$. For $p$ in $\mathcal{P}$,

$$D(p \,\|\, r) = \sum_x p(x)\ln\frac{p(x)}{q(x)} + \sum_x p(x)\ln\frac{q(x)}{r(x)} = D(p \,\|\, q) + \sum_j\lambda_j c_j - \ln Z.$$

With $p = q$ this reads $D(q \,\|\, r) = \sum_j\lambda_j c_j - \ln Z$. Substituting back gives the decomposition, and $D(p \,\|\, q) \ge 0$ with equality only for $p = q$. $\square$

A family of distributions of the form (4), indexed by $\lambda = (\lambda_1, \dots, \lambda_m)$, is an exponential family: the $\lambda_j$ are its natural parameters, the $T_j$ its sufficient statistics, and $\ln Z$, as a function of $\lambda$, its log-partition function. Exponential Families is the planned home of their properties, and Sufficient Statistics of that term. The theorem says what an exponential family is for. As the constraint values $c$ vary, its members are the distributions closest to the reference, in divergence, among all distributions with given averages of the statistics. With a uniform reference they are the maximum entropy distributions. The geometric, exponential and normal families are exponential families with statistics $x$, $x$ and $(x, x^2)$.

The Poisson distribution, placed. Take the reference $r(k) = e^{-1}/k!$ on $\{0, 1, 2, \dots\}$, the Poisson distribution with mean 1, and the statistic $T(k) = k$. Form (4) is $q(k) \propto e^{\lambda k}/k!$, the Poisson distribution with mean $e^{\lambda}$. So the Poisson distribution with mean $m$ is the distribution with mean $m$ that is closest to the Poisson distribution with mean 1. It is not the maximum entropy distribution with mean $m$, because the factor $1/k!$ in the reference is not constant.

For densities the statement and the proof are these. Let $r$ be a reference density, positive on $S$, let $\mathcal{P}$ be the set of densities on $S$ with $E_f[T_j] = c_j$, and suppose $g(x) = \frac1Z\,r(x)\exp\big(\sum_j\lambda_j T_j(x)\big)$, with $Z = \int_S r\exp\big(\sum_j\lambda_jT_j\big)\,dx$ finite, belongs to $\mathcal{P}$. Then $\ln\frac{g}{r} = \sum_j\lambda_jT_j - \ln Z$, and for $f$ in $\mathcal{P}$,

$$D(f \,\|\, r) = \int f\ln\frac{f}{g}\,dx + \int f\ln\frac{g}{r}\,dx = D(f \,\|\, g) + \sum_j\lambda_jc_j - \ln Z.$$

With $f = g$ this gives $D(g \,\|\, r) = \sum_j\lambda_jc_j - \ln Z$, so $D(f \,\|\, r) = D(g \,\|\, r) + D(f \,\|\, g) \ge D(g \,\|\, r)$, by Gibbs' inequality for densities. This form removes the dependence on units noted above. By the invariance theorem of Differential Entropy and the Gaussian, the divergence from a reference does not change when the variable is transformed, provided the reference is transformed with it.

The maximum entropy distribution is the safest model

Why prefer the distribution of greatest entropy? One answer can be proved. Recall from Perplexity and Source Coding that a model $q$ is a code, and that when the outcomes really follow $p$ the code costs $H(p, q)$ per outcome, to within one bit.

Theorem (the maximizer equalizes the cost and minimizes the worst case). Let $q$ in $\mathcal{P}$ have the form (1). Then (a) $H(p, q) = H(q)$ for every $p$ in $\mathcal{P}$: the code built for $q$ costs the same whichever member of $\mathcal{P}$ is the truth. (b) For any other model $r$ on the same alphabet, some member of $\mathcal{P}$, namely $q$, makes $r$ cost more than $H(q)$. So $\sup_{p \in \mathcal{P}} H(p, r) > H(q)$.

Proof. Part (a) is identity (2). For (b), $q$ itself is a member of $\mathcal{P}$, and $H(q, r) = H(q) + D(q \,\|\, r) > H(q)$ by Gibbs' inequality, since $r \ne q$. $\square$

So the maximum entropy distribution is the minimax model. Among all models, it has the smallest worst-case cost over the distributions that agree with the known facts, and its worst case is its own entropy. A model with lower entropy is betting on structure that the facts do not guarantee, and it loses the bet against at least one distribution in $\mathcal{P}$, namely $q$.

On the gaps of oil. Part (a) can be seen in the numbers. The list's own frequencies have 4.66 bits of entropy and are 0.09 bits from the geometric distribution, so the ideal code for the geometric model costs $4.66 + 0.09 = 4.75$ bits per gap on this list, 4,928 bits in all. That is exactly the entropy of the geometric distribution. It would be the same for any list of positive integers with mean 10.39, however its gaps were arranged. The code cannot do better than 4.75 bits on such a list, and it cannot do worse.

Softmax

A classifier or a language model produces a score $z_k$, a logit, for each of $K$ classes, and turns the scores into probabilities with the softmax function, $p_k = e^{z_k}/\sum_j e^{z_j}$ (Activation Functions defines it and its temperature). That is form (1) on the alphabet of classes, with the score as the statistic. So the softmax is a maximum entropy distribution, and the theorem says for which constraint.

Theorem (softmax as maximum entropy). Let $z_1, \dots, z_K$ be scores that are not all equal, and let $c$ be a number strictly between the smallest and the largest score. Then there is exactly one $\beta$ for which the distribution

$$p_k(\beta) = \frac{e^{\beta z_k}}{\sum_{j=1}^{K} e^{\beta z_j}}$$

has expected score $\sum_k p_k(\beta)\,z_k = c$, and that distribution has the greatest entropy among all distributions on the $K$ classes with expected score $c$.

The second claim is the main theorem with $T(k) = z_k$ and $\lambda = \beta$. The first, that a suitable $\beta$ exists and is unique, needs a lemma about the normalizing sum, which is used again in the next section.

Lemma (derivatives of the log-partition function). Let $w_x > 0$ and $s_x$ be numbers indexed by a finite alphabet, and let $A(\beta) = \ln\sum_x w_x e^{\beta s_x}$. Let $E_\beta$ and $\operatorname{Var}_\beta$ be the mean and variance under the distribution $p_\beta(x) = w_x e^{\beta s_x}/e^{A(\beta)}$. Then

$$A'(\beta) = E_\beta[s], \qquad A''(\beta) = \operatorname{Var}_\beta(s) \ge 0.$$

Proof. Write $Z(\beta) = \sum_x w_x e^{\beta s_x}$. Then $Z'(\beta) = \sum_x s_x w_x e^{\beta s_x}$ and $Z''(\beta) = \sum_x s_x^2 w_x e^{\beta s_x}$, so $Z'/Z = E_\beta[s]$ and $Z''/Z = E_\beta[s^2]$. Hence $A' = Z'/Z = E_\beta[s]$ and

$$A'' = \frac{Z''}{Z} - \Big(\frac{Z'}{Z}\Big)^2 = E_\beta[s^2] - \big(E_\beta[s]\big)^2 = \operatorname{Var}_\beta(s). \quad\square$$

Proof of the theorem. Let $m(\beta) = \sum_k p_k(\beta)z_k$ be the expected score. By the lemma with $w = 1$ and $s = z$, $m(\beta) = A'(\beta)$ and $m'(\beta) = \operatorname{Var}_\beta(z)$. Every class has positive probability under $p(\beta)$ and the scores are not all equal, so the variance is positive and $m$ is strictly increasing.

Next, $m(\beta) \to z_{\max}$ as $\beta \to \infty$. Dividing numerator and denominator by $e^{\beta z_{\max}}$, $p_k(\beta) = e^{-\beta(z_{\max} - z_k)}/\sum_j e^{-\beta(z_{\max} - z_j)}$. The denominator is at least 1, because a class with the largest score contributes 1. For a class with $z_k < z_{\max}$ the numerator tends to 0. So the total probability of the classes below the maximum tends to 0, and the expected score tends to $z_{\max}$. Applying this to the scores $-z_k$ shows $m(\beta) \to z_{\min}$ as $\beta \to -\infty$.

So $m$ is continuous, strictly increasing, and takes values arbitrarily close to $z_{\min}$ and to $z_{\max}$. By the intermediate value theorem it takes each value $c$ strictly between them, and by strict monotonicity only once. The distribution $p(\beta)$ at that $\beta$ has the form (1) and meets the constraint, so by the main theorem it is the unique maximum entropy distribution. $\square$

The multiplier $\beta$ is the reciprocal of the temperature $T$ with which language models sample, $p_k \propto e^{z_k/T}$. The theorem gives the temperature a meaning.

  • At $\beta = 0$, an infinite temperature, the distribution is uniform and the expected score is the plain average of the scores. A constraint equal to that average is no constraint.
  • As $\beta$ grows from 0, the temperature falls, the expected score rises toward the top score, and the entropy falls. The fall is strict, by the lemma. The theorem gives the entropy as $H(\beta) = A(\beta) - \beta A'(\beta)$, with $A(\beta) = \ln\sum_k e^{\beta z_k}$, so $H'(\beta) = A'(\beta) - A'(\beta) - \beta A''(\beta) = -\beta\operatorname{Var}_\beta(z)$, which is negative for $\beta > 0$ and positive for $\beta < 0$. So the entropy is greatest at $\beta = 0$.
  • In the limit $\beta \to \infty$ all probability is on the classes with the top score. If $j$ classes share it, each of them has probability $1/\sum_i e^{-\beta(z_{\max} - z_i)}$, which tends to $1/j$, so the entropy tends to $\ln j$. It tends to 0 when the top score is unique.
  • At every temperature in between, the softmax is the most uncertain distribution that achieves its own expected score. Lowering the temperature is the least committal way of raising the average score of what is sampled.
  • A negative $\beta$ gives expected scores below the average, favoring low-scoring classes.

The characters after T

In the training split of the Tiny Shakespeare text a capital T occurs 5,971 times, followed by 22 different characters. The Bigram Language Model reports the row: h follows 46.2 percent of the time, then o, E and the colon. Let $p^*$ be that row of relative frequencies and take the scores $z_k = \ln p^*_k$, which are the logits of the row up to an added constant. Sampling at temperature $T$ uses $p_k \propto (p^*_k)^{1/T}$.

Temperature $T$ Entropy, bits Perplexity Probability of h Expected score $E[\ln p^*]$, nats
0.5 0.93 1.9 85.4% −1.01
1 2.87 7.3 46.2% −1.99
2 4.03 16.3 19.4% −3.07

The uniform distribution on the 22 characters would have $\log_2 22 = 4.46$ bits and an expected score of $-4.26$; the largest possible expected score is that of h alone, $\ln 0.462 = -0.77$. Each row of the table is, by the theorem, the distribution of greatest entropy among all distributions on the 22 characters with that row's expected score. Because the score is a log-probability under the row, the constraint has a second reading. The expected score is minus the cross-entropy against $p^*$, so each row is the most uncertain distribution with a given cross-entropy against the model's own row.

To see the theorem act, take a different distribution with the same expected score as the $T = 2$ row: a mixture of $p^*$ and the uniform distribution, with weight 0.523 on $p^*$, chosen so that its expected score is also $-3.07$. Its entropy is 3.96 bits, 0.06 bits below the 4.03 of the softmax. The script confirms that the 0.06 is the divergence of the mixture from the softmax, as identity (3) requires.

The greatest entropy available at each expected score for the 22 characters that follow T, traced by the softmax family as the temperature varies, with three temperatures marked. Notice that the curve peaks at the uniform distribution, where the constraint is the plain average score, and that the mixture with the same expected score as the T = 2 point (square) lies just below the curve, as every other distribution with that score must.
The greatest entropy available at each expected score for the 22 characters that follow T, traced by the softmax family as the temperature varies, with three temperatures marked. Notice that the curve peaks at the uniform distribution, where the constraint is the plain average score, and that the mixture with the same expected score as the T = 2 point (square) lies just below the curve, as every other distribution with that score must.

Maximum likelihood from the other side

In practice the constraint values are not chosen. They come from data: the average of each statistic in a sample. And the model of form (1) is not found by solving the constraints directly. It is fitted by maximum likelihood, that is, by minimizing the cross-entropy loss. These are the same computation.

Theorem (maximum likelihood is maximum entropy). Let $\mathcal{X}$ be finite, let $\hat{p}$ be the empirical distribution of a sample, and let $\hat{c}_j = E_{\hat{p}}[T_j]$ be the sample averages of the statistics. For the family $q_\lambda(x) = \exp\big(\sum_j\lambda_j T_j(x)\big)/Z(\lambda)$, the average log-likelihood of the sample is

$$\ell(\lambda) = \sum_x \hat{p}(x)\ln q_\lambda(x) = \sum_j\lambda_j\hat{c}_j - \ln Z(\lambda) = -H(\hat{p}, q_\lambda).$$

(a) $\ell$ is concave, and $\hat\lambda$ maximizes it exactly when the model reproduces the sample averages: $E_{q_{\hat\lambda}}[T_j] = \hat{c}_j$ for every $j$.

(b) In that case $q_{\hat\lambda}$ is the maximum entropy distribution among all distributions whose averages of the $T_j$ equal the sample's, and its entropy is the minimized loss: $H(q_{\hat\lambda}) = -\ell(\hat\lambda) = \min_\lambda H(\hat{p}, q_\lambda)$.

Proof. The formula for $\ell$ follows from $\ln q_\lambda(x) = \sum_j\lambda_j T_j(x) - \ln Z(\lambda)$, as in (2).

(a) Fix $\lambda$ and a direction $v = (v_1, \dots, v_m)$, and consider $\ell$ along the line $\lambda + tv$. Then $\ln Z(\lambda + tv) = \ln\sum_x w_x e^{t s_x}$ with $w_x = \exp\big(\sum_j\lambda_j T_j(x)\big)$ and $s_x = \sum_j v_j T_j(x)$. By the lemma its first derivative in $t$ at $t = 0$ is $E_{q_\lambda}[s]$ and its second derivative is a variance, which is nonnegative. So

$$\frac{d}{dt}\ell(\lambda + tv)\Big|_{t=0} = \sum_j v_j\big(\hat{c}_j - E_{q_\lambda}[T_j]\big), \qquad \frac{d^2}{dt^2}\ell(\lambda + tv) \le 0.$$

A function whose second derivative along every line is nonpositive is concave. If $\hat\lambda$ maximizes $\ell$, the first derivative vanishes in every direction $v$, which forces $E_{q_{\hat\lambda}}[T_j] = \hat{c}_j$ for each $j$. Conversely, if the averages match at $\hat\lambda$, then for any other $\lambda'$ the function $t \mapsto \ell(\hat\lambda + t(\lambda' - \hat\lambda))$ is concave with derivative 0 at $t = 0$, so it is at most its value at $t = 0$ everywhere, and $\ell(\lambda') \le \ell(\hat\lambda)$.

(b) $q_{\hat\lambda}$ has the form (1) and meets the constraints $E[T_j] = \hat{c}_j$, so the main theorem applies, and gives $H(q_{\hat\lambda}) = \ln Z(\hat\lambda) - \sum_j\hat\lambda_j\hat{c}_j = -\ell(\hat\lambda)$. $\square$

So there are two descriptions of one distribution. Among all distributions that match the sample's averages, it is the one of greatest entropy. Among all distributions of the exponential form, it is the one of greatest likelihood. The maximum of the entropy and the minimum of the cross-entropy loss are the same number. Cross-Entropy and Kullback–Leibler Divergence proves that maximizing the likelihood is minimizing the divergence $D(\hat{p} \,\|\, q_\lambda)$, and Maximum Likelihood Estimation treats the method in general.

The theorem here is for a single categorical variable. A classifier or a language model is a conditional model, with one softmax for each input, and the statistics are features of the input and the class together. Berger, Della Pietra and Della Pietra (1996, Sections 3.3 to 3.5) set out that case. They maximize the conditional entropy subject to the model's expected features matching the sample's, obtain the form $p(y \mid x) = \exp\big(\sum_i\lambda_i f_i(x, y)\big)/Z_\lambda(x)$, and state that the dual of that problem is maximizing the likelihood of the same form, which is softmax regression; they give the statement and refer to the theory of Lagrange duality for its proof. That conditional result is cited here, not proved. Multinomial and Softmax Regression is the planned home of the model.

On a table from the course. Mutual Information and feature selection study the 7,769 Reuters training stories by whether they contain the term wheat (243 do) and whether they are about grain (433 are); 227 are both. Take the four cells as the alphabet and the two indicators, "contains wheat" and "is about grain", as the statistics. The sample averages are the two margins, $243/7{,}769$ and $433/7{,}769$. Form (1) with two indicator statistics is $q \propto e^{\lambda_1[\text{wheat}] + \lambda_2[\text{grain}]}$, a product of a factor for the term and a factor for the class: the two are independent under $q$. Matching the margins gives $\lambda_1 = \ln\frac{243}{7{,}526} = -3.4331$ and $\lambda_2 = \ln\frac{433}{7{,}336} = -2.8298$, and the fitted table is the product of the margins, with 13.5 stories expected in the cell that holds 227.

By the theorem this independent table is at once the maximum likelihood fit of the independence model and the maximum entropy distribution with the observed margins. Its entropy is 0.5110 bits, and the observed table's is 0.3871 bits. By identity (3) the difference, 0.124 bits, is the divergence of the observed table from the independent one, which is the mutual information between term and class that those two articles report. Mutual information is the entropy a joint distribution gives up, relative to the most uncertain distribution with its margins.

What is proved, what is cited, what is computed

Proved here: the maximum entropy theorem for discrete distributions and for densities, with the identity that the shortfall is a divergence; the corollary for inequality constraints; the geometric, uniform, exponential and normal cases; the minimum-divergence theorem with a reference distribution and its decomposition; the minimax property of the maximizer; the existence and uniqueness of the softmax multiplier for one constraint; the derivatives of the log-partition function; and the equivalence of maximum likelihood and maximum entropy for one categorical variable.

Borrowed, with the proof on a linked page: Gibbs' inequality for mass functions (Cross-Entropy and Kullback–Leibler Divergence) and for densities, with the definition of differential entropy (Differential Entropy and the Gaussian); the entropy of the geometric distribution (Entropy); the cost of a code built for a model (Perplexity and Source Coding).

Cited, not proved: the conditional form of the maximum entropy and maximum likelihood equivalence (Berger, Della Pietra and Della Pietra, 1996); the optimality of Golomb codes for geometric gaps, as reported in Index Compression. The principle itself, that the maximum entropy distribution should be used, is Jaynes's proposal and is not a mathematical claim; what is proved is the minimax property.

Computed: every number about the Reuters gaps, the characters after T, and the wheat table. The comparisons between the gaps and the geometric distribution describe one term and one collection.

Worked problems

These problems are posed for this article, on small exact cases and on the Reuters gaps, one for each main result. The script tools/experiments/statistics/maximum-entropy-distributions.py checks every number.

Problem 1 (a die with a given mean). A die shows faces 1 to 6, and all that is known is that its long-run average is 4.5, not the 3.5 of a fair die. Find the maximum entropy distribution and its entropy.

Solution. The probabilities are 0.0544, 0.0788, 0.1142, 0.1654, 0.2398 and 0.3475 for faces 1 to 6, and the entropy is 1.6136 nats (2.328 bits), against $\ln 6 = 1.7918$ nats (2.585 bits) for a fair die.

This is the softmax theorem with scores $z_k = k$ and $c = 4.5$, which lies strictly between 1 and 6, so there is exactly one $\beta$. The distribution is $p_k = e^{\beta k}/\sum_j e^{\beta j}$: each face is more probable than the one before by the same factor $e^{\beta}$. The expected score $m(\beta)$ is 3.5 at $\beta = 0$ and increases with $\beta$, so the solution has $\beta > 0$. No formula gives it, but because $m$ is increasing, bisection finds it: $\beta = 0.3710$, a factor of $e^{\beta} = 1.4493$ between successive faces. The probabilities follow, and the entropy is $\ln Z - \beta c$ by the theorem, which the script confirms against $-\sum p\ln p$.

Any other die with mean 4.5 has less entropy, by its divergence from this one. A die that shows only 3 and 6, equally often, also has mean 4.5; its entropy is $\ln 2 = 0.6931$ nats, and the shortfall of $1.6136 - 0.6931$ is its divergence from the maximum entropy die. The fair die has more entropy than either, 1.7918 nats, but it is not a candidate: its mean is 3.5.

Problem 2 (a density on an interval with a given mean). Find the density of greatest differential entropy on $[0, 1]$ with mean 0.3.

Solution. It is the truncated exponential $g(x) = e^{\lambda x}/Z$ with $\lambda = -2.6721$, falling from 2.8705 at $x = 0$ to 0.1984 at $x = 1$, and its differential entropy is $-0.2528$ nats.

By the theorem for densities with $S = [0, 1]$ and $T(x) = x$, the maximizer has the form $g(x) = e^{\lambda x}/Z$ with $Z = \int_0^1 e^{\lambda x}\,dx = (e^{\lambda} - 1)/\lambda$, if the mean can be matched. The mean of $g$ is

$$\frac{1}{Z}\int_0^1 x\,e^{\lambda x}\,dx = \frac{e^{\lambda}}{e^{\lambda} - 1} - \frac{1}{\lambda},$$

by integration by parts. This mean increases with $\lambda$. To see it, write $Z(\lambda) = \int_0^1 e^{\lambda x}\,dx$. Differentiating under the integral sign, which is allowed for a smooth integrand on a bounded interval, $Z' = \int_0^1 x e^{\lambda x}\,dx$ and $Z'' = \int_0^1 x^2 e^{\lambda x}\,dx$. So the mean is $Z'/Z$, and its derivative is $Z''/Z - (Z'/Z)^2$, the variance of $g$, which is positive. The mean tends to $\tfrac12$ as $\lambda \to 0$, the mean of the uniform density: $\frac{e^\lambda}{e^\lambda - 1} = 1 + \frac{1}{e^\lambda - 1}$, and $\frac{1}{e^\lambda - 1} - \frac1\lambda \to -\tfrac12$, from $e^\lambda - 1 = \lambda + \lambda^2/2 + \cdots$. It tends to 0 as $\lambda \to -\infty$, where both $e^\lambda$ and $1/\lambda$ vanish. A mean of 0.3 is below $\tfrac12$, so $\lambda < 0$, and bisection gives $\lambda = -2.6721$. The entropy is $\ln Z - \lambda \times 0.3 = -0.2528$ nats. It is negative because the density is concentrated: its effective width, in the sense of Differential Entropy and the Gaussian, is $e^{-0.2528} = 0.78$ of the unit interval. It is below the 0 of the uniform density, which is not a candidate, and above every other density on $[0, 1]$ with mean 0.3. For comparison, the beta density proportional to $x^{0.5}(1 - x)^{2.5}$ also has mean 0.3, and its differential entropy is $-0.3556$ nats.

Problem 3 (a bound on a mean). A waiting time is positive and its mean is known to be at most 10.4 minutes. What is the greatest differential entropy it can have? What if the mean is known to be exactly 5 minutes?

Solution. At most $1 + \ln 10.4 = 3.3418$ nats, attained by the exponential distribution with mean 10.4; with mean exactly 5, at most $1 + \ln 5 = 2.6094$ nats.

For the bound, use the corollary on inequality constraints. The exponential density with mean 10.4 has the form (1) with $\lambda = -1/10.4 < 0$ and meets the constraint with equality, so it has the greatest entropy among all densities on $[0, \infty)$ with mean at most 10.4. The sign of $\lambda$ matters. Entropy increases with the mean, so a distribution allowed a mean of up to 10.4 uses all of it. If the constraint were instead that the mean is at least 10.4, the corollary would not apply, and there would be no maximum. The exponential densities with larger means have entropy $1 + \ln\mu$, without bound. With the mean fixed at 5, the theorem gives the exponential with mean 5. The values are in nats for a time in minutes; in seconds each would be larger by $\ln 60$, by the scaling rule for differential entropy.

Problem 4 (a code for a given mean). A postings list has 1,038 gaps with mean 10.39, like that of oil. Without knowing anything else about the gaps, how many bits will an ideal code for the geometric distribution with that mean spend on the list, and how does that compare with the codes of Index Compression?

Solution. It spends 4.75 bits per gap, 4,928 bits in all, whatever the gaps are.

By part (a) of the minimax theorem, the cross-entropy of any distribution with mean 10.39 against the geometric distribution with that mean equals the geometric entropy, $\mu\log_2\mu - (\mu - 1)\log_2(\mu - 1)$ with $\mu = 10.39$, which is 4.75 bits. The total for 1,038 gaps is 4,928 bits. The reason is visible in the code lengths: the ideal length for a gap $k$ is $-\log_2\big[(1 - \theta)^{k-1}\theta\big]$, a linear function of $k$, so the total length of a list depends on the gaps only through their sum, which the mean fixes. Index Compression reports 5,000 bits for the Rice code with parameter 3 and 5,728 bits for the gamma code on this list. The Rice code is a geometric code with whole-number lengths and a parameter restricted to a power of two, and it comes within 72 bits of the ideal. The entropy of the list's own frequencies is lower, 4.66 bits per gap, but a code that reached it would have to know those frequencies. The geometric code needs one number, the mean.

What we have learned

One argument settles every maximum entropy problem with constraints on averages. If a distribution's log-probability is a linear combination of the constrained statistics, then every distribution that meets the constraints has the same cross-entropy against it. Gibbs' inequality then makes it the unique maximizer, with every other candidate short by exactly its divergence. The uniform, geometric, exponential and normal distributions are the cases of no constraint, a mean on the positive integers, a mean on the half line, and a mean and variance on the line. The general solution, with a reference distribution, is an exponential family. The maximizer is the model with the smallest worst-case cost over everything the constraints allow. The softmax function is the maximizer under a constraint on the expected score, with the temperature as the multiplier, and fitting it by maximum likelihood finds the same distribution from the other side.

This article completes the Information Theory unit. Its results are used where the collection goes next: Exponential Families and Multinomial and Softmax Regression in the unit on generalized linear models take up the family and the model identified here, and Statistics of Model Evaluation turns to the uncertainty in the losses and perplexities that the unit has shown how to interpret.

References

  • Berger, A. L., Della Pietra, S. A., and Della Pietra, V. J. (1996). A maximum entropy approach to natural language processing. Computational Linguistics, 22(1), 39–71. record
  • Cover, T. M., and Thomas, J. A. (2006). Elements of Information Theory, 2nd edition. Wiley. Chapter 12, "Maximum Entropy" (Section 12.1, "Maximum Entropy Distributions"; Section 12.2, "Examples"). record · 8 pages
  • Jaynes, E. T. (1957). Information theory and statistical mechanics. Physical Review, 106(4), 620–630. record
  • Karpathy, A. (2015). char-rnn [Software and data]. https://github.com/karpathy/char-rnn (the file data/tinyshakespeare/input.txt). record · 3 pages
  • Lewis, D. D. (1997). Reuters-21578 text categorization test collection, distribution 1.0. record · 11 pages
  • Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical Journal, 27(3), 379–423, and 27(4), 623–656. Section 10, "Discussion and examples", and Section 20, "Entropy of a continuous distribution". record · 6 pages