The 1,400 abstracts of the Cranfield collection contain 4,803 distinct terms after stemming, and the average abstract has 111.6 tokens. Language Models for Retrieval ranks abstracts by the probability that each abstract's own word distribution would generate a query, and must therefore estimate, for every abstract, a probability for each of the 4,803 terms, most of which the abstract never uses. Add-one smoothing, the uniform prior of Naive Bayes, is one answer; Dirichlet smoothing, with a prior centered on the whole collection, is another, and its strength $\mu$ must be set. That page found that $\mu$ between 200 and 300 ranks best on Cranfield, far below the 2,000 commonly recommended. How should a prior be chosen when the data say little, and is there a choice that does not depend on who makes it?
A prior distribution encodes what is assumed about a parameter before the data are seen (Priors, Posteriors and Conjugate Families). When real knowledge is available it should go into the prior, and the hierarchical models of Hierarchical Models and Shrinkage learn a prior from related data. This article is about the other case: when the analyst wants a prior that adds little, or a default that others can accept. Gelman et al. (2013, §2.8) note that such priors are sometimes called reference priors and described as vague, flat, diffuse or noninformative, and that their rationale is often said to be to let the data speak for themselves; they go on to list difficulties with them.
First we show that "flat" has no meaning until a scale is named, and define improper priors. Then we derive Jeffreys' prior, which removes the dependence on the scale, and compute it for the standard models. Next we describe weakly informative priors, a deliberate compromise. Finally we read the smoothing methods of text retrieval as priors and choose the strength of the Dirichlet prior on Cranfield in three ways. The article assumes Priors, Posteriors and Conjugate Families and Fisher Information and the Cramér–Rao Bound.
A flat prior is flat on one scale only
The oldest default is the uniform prior, which Laplace used for a proportion. Gelman et al. (2013, §2.4) note that later readers ascribe to him the "principle of insufficient reason": a uniform specification when nothing is known. The difficulty is that a parameter can be described in many ways, and uniformity on one description is not uniformity on another.
Theorem 1 (a prior on a new scale). Let $\theta$ have prior density $\pi(\theta)$, and let $\eta = h(\theta)$ with $h$ differentiable and strictly monotone. Then $\eta$ has density
$$\pi_\eta(\eta) = \pi(\theta(\eta))\,\Big\lvert\frac{d\theta}{d\eta}\Big\rvert.$$
Proof. For increasing $h$, $P(\eta \le t) = P(\theta \le h^{-1}(t))$, and differentiating in $t$ by the chain rule gives $\pi(h^{-1}(t))\,\frac{d}{dt}h^{-1}(t)$. For decreasing $h$, $P(\eta \le t) = P(\theta \ge h^{-1}(t))$, whose derivative is $-\pi(h^{-1}(t))\frac{d}{dt}h^{-1}(t)$, which is positive. Both cases are the stated formula; Random Variables and Distribution Functions proves the change-of-variables rule in general. $\square$
The posterior transforms by the same rule, so a Bayesian analysis gives consistent answers on every scale once the prior is fixed. The problem is only in choosing the prior: "uniform" picks a different distribution on each scale.
Example (uniform on the proportion is not uniform on the log odds). Let $p$ be uniform on $(0, 1)$ and $\varphi = \ln\frac{p}{1-p}$. Then $p = 1/(1 + e^{-\varphi})$ and $dp/d\varphi = p(1-p)$, so by Theorem 1 the density of $\varphi$ is $p(1-p) = e^\varphi/(1 + e^\varphi)^2$, the logistic density, which is far from flat. It puts probability $2/(1 + e^{-2}) - 1 = 0.7616$ on $\lvert\varphi\rvert \le 2$, that is, on $p$ between 0.1192 and 0.8808. A uniform prior on $p$ is a fairly confident statement that the log odds are moderate. Conversely, a flat density on $\varphi$ corresponds on the $p$ scale to a density proportional to $1/[p(1-p)]$, which piles up near 0 and 1. The Likelihood Function made the same observation about areas under a likelihood: they change with the scale, which is why a likelihood is not a probability distribution and why a prior must say on which scale it is flat.
Improper priors
The density $1/[p(1-p)]$ has an infinite integral over $(0, 1)$, so it is not a probability distribution. A function used in Bayes' theorem in place of a prior density whose integral is infinite is an improper prior. Gelman et al. (2013, §2.8) call a prior proper if it does not depend on the data and integrates to 1 (a density with any finite positive integral can be rescaled to integrate to 1). They allow improper priors provided the posterior they produce, likelihood times prior, has a finite integral, which must be checked, and they read the result as an approximation for situations in which the likelihood dominates the prior. The flat prior on a normal mean has a proper posterior given at least one observation, and the prior $1/\sigma^2$ of Priors, Posteriors and Conjugate Families given at least two observations that are not all equal. The prior $1/[p(1-p)]$ is the improper $\text{Beta}(0, 0)$ prior, the limit of $\text{Beta}(a, a)$ as $a \to 0$, which Gelman et al. (2013, §2.8) identify as uniform on the log odds. Whether its posterior exists depends on the data.
Theorem 2 (when the $\text{Beta}(0, 0)$ posterior is proper). With the improper prior $\pi(p) \propto p^{-1}(1-p)^{-1}$ and $x$ successes in $n$ trials, the posterior is proper if and only if $0 < x < n$, and then it is $\text{Beta}(x, n - x)$ with mean $x/n$.
Proof. Likelihood times prior is $p^{x-1}(1-p)^{n-x-1}$. Near $p = 0$ the factor $(1-p)^{n-x-1}$ is close to 1, so the integral near 0 is finite exactly when $\int_0^{1/2}p^{x-1}\,dp$ is finite, that is, when $x - 1 > -1$, or $x > 0$; for $x = 0$ the integral $\int_0^{1/2}p^{-1}\,dp$ diverges like $\ln(1/\varepsilon)$ as its lower end $\varepsilon$ goes to 0. By the same argument at $p = 1$, the integral is finite near 1 exactly when $n - x > 0$. When both hold, the normalized posterior is the $\text{Beta}(x, n-x)$ density, whose mean is $x/(x + n - x) = x/n$. $\square$
So with this prior the posterior mean is the maximum likelihood estimate, and a sample with no events (or no non-events) leaves no posterior at all. Gelman et al. (2013, §2.8) give the same warning. With 1 hypertensive among 40 the posterior is $\text{Beta}(1, 39)$, with mean $1/40 = 0.025$; with none it does not exist.
Jeffreys' prior
Jeffreys (1946) proposed a rule that gives the same answer whatever scale it is applied on. His abstract describes "a certain differential form depending on the values of the parameters in a law of chance" that "is invariant for all transformations of the parameters when the law is differentiable with regard to all parameters", and proposes it as having "the properties required to give a general rule for stating the prior probability in a large class of estimation problems." In the form Gelman et al. (2013, §2.8) give for one parameter, the prior is the square root of the Fisher information.
Here is the idea. A rule assigns a prior to each description of a model. It is consistent if applying the rule on the $\theta$ scale and then transforming by Theorem 1 gives the same distribution as applying the rule directly on the $\eta$ scale (Gelman et al., 2013, §2.8). The Fisher information changes with the scale by exactly the square of the factor in Theorem 1, so its square root changes by exactly that factor.
The Fisher information of one observation is $I(\theta) = E_\theta[u(X;\theta)^2]$, where $u = \frac{\partial}{\partial\theta}\ln f(X \mid \theta)$ is the score (Fisher Information and the Cramér–Rao Bound). That article proves, in the section "Information depends on the parameter chosen", that for a reparametrization $\theta = g(\eta)$ the information about $\eta$ is $I^*(\eta) = I(g(\eta))\,g'(\eta)^2$.
Theorem 3 (Jeffreys' prior is invariant). Define Jeffreys' prior on any one-dimensional parametrization by $\pi_J(\theta) \propto \sqrt{I(\theta)}$. If $\theta = g(\eta)$ with $g$ differentiable and strictly monotone, the Jeffreys prior computed on the $\eta$ scale equals the Jeffreys prior on the $\theta$ scale transformed by Theorem 1.
Proof. Directly on the $\eta$ scale, $\pi_J(\eta) \propto \sqrt{I^*(\eta)} = \sqrt{I(g(\eta))}\,\lvert g'(\eta)\rvert$. Transforming $\pi_J(\theta) \propto \sqrt{I(\theta)}$ by Theorem 1 with $d\theta/d\eta = g'(\eta)$ gives $\sqrt{I(g(\eta))}\,\lvert g'(\eta)\rvert$. The two agree up to the constant of proportionality, which is all a prior density is defined up to. $\square$
For a vector parameter Jeffreys' rule is $\pi_J(\boldsymbol\theta) \propto \sqrt{\det I(\boldsymbol\theta)}$, with $I(\boldsymbol\theta)$ the information matrix of Fisher Information and the Cramér–Rao Bound (section "Several parameters"). It is invariant by the same argument: the score transforms as $\mathbf{u}^* = J^{\mathsf T}\mathbf{u}$, with $J$ the matrix of derivatives $\partial\theta_i/\partial\eta_j$, so $I^*(\boldsymbol\eta) = E[\mathbf{u}^*\mathbf{u}^{*\mathsf T}] = J^{\mathsf T}I(\boldsymbol\theta)J$ and $\sqrt{\det I^*} = \lvert\det J\rvert\sqrt{\det I}$, which is the multivariate change-of-variables factor. Gelman et al. (2013, §2.8) note that the extension to several parameters is "more controversial", and the normal model below shows one reason.
Jeffreys priors for the standard families
Theorem 4 (Jeffreys priors). For one observation:
- Bernoulli with probability $p$: $I(p) = 1/[p(1-p)]$, so $\pi_J(p) \propto p^{-1/2}(1-p)^{-1/2}$, the proper $\text{Beta}(\tfrac12, \tfrac12)$ distribution. The same holds for a binomial count.
- Poisson with mean $\lambda$: $I(\lambda) = 1/\lambda$, so $\pi_J(\lambda) \propto \lambda^{-1/2}$, improper. The posterior after counts with total $t$ in $n$ observations is $\text{Gamma}(t + \tfrac12,\ n)$.
- Normal with mean $\mu$ and known variance $\sigma^2$: $I(\mu) = 1/\sigma^2$, constant, so $\pi_J(\mu)$ is flat.
- Normal with known mean and standard deviation $\sigma$: $I(\sigma) = 2/\sigma^2$, so $\pi_J(\sigma) \propto 1/\sigma$, equivalently $\pi_J(\sigma^2) \propto 1/\sigma^2$.
- Normal with both unknown, $(\mu, \sigma)$: the information matrix is $\text{diag}(1/\sigma^2,\ 2/\sigma^2)$, so $\pi_J(\mu, \sigma) \propto 1/\sigma^2$.
- Categorical with probabilities $\theta_1, \ldots, \theta_K$: $\pi_J \propto \prod_{k=1}^K\theta_k^{-1/2}$, the $\text{Dirichlet}(\tfrac12, \ldots, \tfrac12)$ distribution.
Proof. Parts 1 to 3 use the informations of the standard families computed in Fisher Information and the Cramér–Rao Bound (section "The standard families"): $1/[p(1-p)]$ per Bernoulli trial (so $n/[p(1-p)]$ for a binomial count, which has the same square root up to a constant), $1/\lambda$ for the Poisson, $1/\sigma^2$ for a normal mean. For part 2's posterior, $\lambda^{-1/2}\cdot\lambda^te^{-n\lambda} = \lambda^{t + 1/2 - 1}e^{-n\lambda}$, a gamma kernel with a finite integral whenever $n \ge 1$.
Part 4: the same table gives $I(\sigma) = 2/\sigma^2$ for the standard deviation of a normal with known mean, so $\pi_J(\sigma) \propto 1/\sigma$. On the scale of $\sigma^2$, Theorem 1 with $d\sigma/d\sigma^2 = 1/(2\sigma)$ turns $1/\sigma$ into a density proportional to $1/\sigma^2$.
Part 5: that page's example "the normal distribution with both parameters unknown" (section "Several parameters") gives the information matrix $\text{diag}(1/\sigma^2,\ 2/\sigma^2)$. Its determinant is $2/\sigma^4$, whose square root is proportional to $1/\sigma^2$.
Part 6: take $\theta_1, \ldots, \theta_{K-1}$ as the free parameters, $\theta_K = 1 - \sum_{k<K}\theta_k$. With indicators $y_k$ of the observed class, $\ln f = \sum_{k<K}y_k\ln\theta_k + y_K\ln\theta_K$, and differentiating twice, $\partial^2\ln f/\partial\theta_j\partial\theta_k = -\delta_{jk}y_k/\theta_k^2 - y_K/\theta_K^2$, with $\delta_{jk} = 1$ if $j = k$ and 0 otherwise. Taking minus the expectation with $E[y_k] = \theta_k$ gives $I = D + \theta_K^{-1}\mathbf{1}\mathbf{1}^{\mathsf T}$ with $D = \text{diag}(1/\theta_1, \ldots, 1/\theta_{K-1})$. For any invertible $D$ and vectors $\mathbf{a}, \mathbf{b}$, $\det(D + \mathbf{a}\mathbf{b}^{\mathsf T}) = \det(D)(1 + \mathbf{b}^{\mathsf T}D^{-1}\mathbf{a})$: write $D + \mathbf{a}\mathbf{b}^{\mathsf T} = D(\mathrm{Id} + \mathbf{c}\mathbf{b}^{\mathsf T})$ with $\mathbf{c} = D^{-1}\mathbf{a}$ and $\mathrm{Id}$ the identity matrix of size $K - 1$; for $\mathbf{b} \ne \mathbf{0}$, every vector orthogonal to $\mathbf{b}$ is an eigenvector of $\mathrm{Id} + \mathbf{c}\mathbf{b}^{\mathsf T}$ with eigenvalue 1, which accounts for at least $K - 2$ eigenvalues, and the trace $K - 1 + \mathbf{b}^{\mathsf T}\mathbf{c}$ makes the last one $1 + \mathbf{b}^{\mathsf T}\mathbf{c}$. Here $\mathbf{b}^{\mathsf T}D^{-1}\mathbf{a} = \theta_K^{-1}\sum_{k<K}\theta_k = (1 - \theta_K)/\theta_K$, so
$$\det I = \prod_{k<K}\frac{1}{\theta_k}\cdot\Big(1 + \frac{1 - \theta_K}{\theta_K}\Big) = \prod_{k=1}^{K}\frac{1}{\theta_k},$$
and its square root is $\prod_k\theta_k^{-1/2}$. $\square$
Part 1 is Gelman et al.'s (2013, §2.8) derivation of the Jeffreys prior for the binomial, and it is the prior of the Jeffreys interval in Credible Intervals and How They Differ from Confidence Intervals. For the $(\mu, \sigma)$ normal, the rule that treats the two parameters separately, flat on $\mu$ and $1/\sigma$ on $\sigma$, gives $\pi(\mu, \sigma) \propto 1/\sigma$, which is the prior $\pi(\mu, \sigma^2) \propto 1/\sigma^2$ that Gelman et al. (2013, §3.2) use and that reproduces the $t$ interval (Priors, Posteriors and Conjugate Families, corollary to Theorem 7). The joint rule of part 5 gives $1/\sigma^2$ instead, and the two disagree in a way that matters.
Example (the joint Jeffreys prior for a normal sample undercovers). Under $\pi(\mu, \sigma) \propto 1/\sigma^2$, Theorem 1 gives $\pi(\mu, \sigma^2) \propto \sigma^{-2}\cdot\frac{1}{2\sigma} \propto (\sigma^2)^{-3/2}$. Times the normal likelihood the posterior is proportional to $(\sigma^2)^{-(n/2 + 1)}(\sigma^2)^{-1/2}\exp\{-[(n-1)s^2 + n(\mu - \bar x)^2]/(2\sigma^2)\}$, the normal–inverse-gamma form of Priors, Posteriors and Conjugate Families (Theorem 7) with $\kappa_n = n$, $\mu_n = \bar x$, $\nu_n = n$ and $\nu_n\sigma_n^2 = (n - 1)s^2$. So $\mu$ is Student $t$ with $n$ degrees of freedom, location $\bar x$ and scale $\frac{s}{\sqrt n}\sqrt{(n-1)/n}$, and the central 95% interval is $\bar x \pm t_{n, 0.975}\sqrt{(n-1)/n}\,s/\sqrt n$. Since $(\bar x - \mu)/(s/\sqrt n)$ has the $t_{n-1}$ distribution in repeated samples, the interval's coverage is $P(\lvert T_{n-1}\rvert \le t_{n, 0.975}\sqrt{(n-1)/n})$ at every $\mu$ and $\sigma$: 0.7978 for $n = 2$, 0.9363 for $n = 10$ and 0.9460 for $n = 30$. For Student's ten sleep differences the interval is $(0.758, 2.402)$ hours, using $t_{10, 0.975} = 2.2281$, narrower than the $t$ interval $(0.700, 2.460)$. The extra factor $1/\sigma$ acts like one more observation in the inference about $\sigma$: the posterior has $n$ degrees of freedom where the sampling distribution of the pivot has $n - 1$, so the joint rule does not pay for the degree of freedom used to estimate $\mu$; the rule applied to each parameter separately does. It illustrates the reason Gelman et al. give for calling the multiparameter rule controversial: independent priors on the components can give different results from the joint rule. The coverage argument is our own.
Weakly informative priors
Gelman et al. (2013, §2.9) characterize a prior as weakly informative "if it is proper but is set up so that the information it does provide is intentionally weaker than whatever actual prior knowledge is available." The aim is not to model ignorance but to rule out the absurd: for regression coefficients on the log or logit scale with standardized inputs, they argue, effects larger than about 10 are implausible in most applications, since a difference of 10 on the logit scale moves a probability from 0.01 to 0.99. Such a prior keeps the posterior within reasonable bounds when the data are weak, and has little effect when the data are strong.
Gelman, Jakulin, Pittau and Su (2008) turn this into a default for logistic regression. They first scale each input that is not binary to mean 0 and standard deviation 0.5, then give each coefficient an independent Cauchy prior with center 0 and scale 2.5, and the intercept a Cauchy prior with scale 10 (their §2). Their abstract notes that the prior "has the advantage of always giving answers, even when there is complete separation in logistic regression", the case in which Deviance and Inference for Generalized Linear Models (Theorem 4) proves the maximum likelihood estimate does not exist. They motivate the scale with a single binomial trial with half a success and half a failure, whose likelihood on the logit scale, $e^{\varphi/2}/(1 + e^\varphi)$, is close to a $t$ density with 7 degrees of freedom and scale 2.5; they choose the Cauchy as a more conservative, longer-tailed version (§2.2).
That half-success likelihood is Jeffreys' prior in disguise. By Theorem 1, the $\text{Beta}(\tfrac12, \tfrac12)$ density on $p$ becomes, on the log odds scale, $p^{-1/2}(1-p)^{-1/2}\cdot p(1-p) = p^{1/2}(1-p)^{1/2} = e^{\varphi/2}/(1 + e^\varphi)$. So Gelman and colleagues' default is a longer-tailed version of the Jeffreys prior for a single proportion, placed on each coefficient.
Example (priors for a sample with no events). Suppose that none of the 40 young adults had hypertension, and compare four priors for $p$: the uniform, Jeffreys', and Cauchy priors with center 0 and scales 2.5 and 10 on the log odds. The Cauchy posteriors have no closed form and were computed by numerical integration over the whole real line.
| Prior | Posterior mean, 0 of 40 | Median | 97.5% point | Posterior mean, 10 of 40 | Median | 97.5% point |
|---|---|---|---|---|---|---|
| Uniform, $\text{Beta}(1, 1)$ | 0.0238 | 0.0168 | 0.0860 | 0.2619 | 0.2581 | 0.4030 |
| Jeffreys, $\text{Beta}(\tfrac12, \tfrac12)$ | 0.0122 | 0.0056 | 0.0605 | 0.2561 | 0.2521 | 0.3983 |
| Cauchy on the log odds, scale 2.5 | 0.0051 | 0.00016 | 0.0413 | 0.2570 | 0.2531 | 0.3993 |
| Cauchy on the log odds, scale 10 | 0.0018 | $2.9 \times 10^{-7}$ | 0.0193 | 0.2506 | 0.2464 | 0.3937 |
With 10 events the four posterior means lie between 0.2506 and 0.2619, a spread of 0.011, and the data decide. With no events they disagree by more than a factor of ten. The flat prior on $p$ pulls the estimate up toward one half: its posterior mean, $1/42$, is the rule of succession. Gelman et al. (2013, §2.9) warn of exactly this: a uniform prior on the prevalence of a rare disease, with 0 of 100 cases, grossly overestimates a prevalence known to be below 1 in 10,000 (the posterior mean would be $1/102 = 0.0098$). The Cauchy priors on the log odds do the opposite: a heavy-tailed prior on an unbounded scale keeps substantial probability on log odds of $-10$ and below, so after no events the posterior median of $p$ is tiny, $2.9 \times 10^{-7}$ for scale 10, a claim of near-certain rarity that 40 observations cannot support. Gelman and colleagues choose the scale-10 prior for the intercept precisely because it is weak, and note that it implies that the probability for an average case lies between $10^{-9}$ and $1 - 10^{-9}$ (§2.2). Near a boundary of the parameter space no prior is neutral, and a default should be checked against what the analyst actually believes about the range of plausible values.
Smoothing is a choice of prior
The smoothing methods of text retrieval are posterior means under Dirichlet priors, so choosing a smoothing method is choosing a prior. Under the multinomial model, the tokens of a document are independent draws from a distribution $\boldsymbol\theta$ over the vocabulary of $V$ terms; with a $\text{Dirichlet}(a_1, \ldots, a_V)$ prior and counts $\mathit{tf}_{t,d}$ in a document of length $L_d$, the posterior mean of $\theta_t$ is $(\mathit{tf}_{t,d} + a_t)/(L_d + A)$, $A = \sum_t a_t$ (Naive Bayes; Priors, Posteriors and Conjugate Families, Theorem 8).
| Smoothing | Estimate of $\theta_t$ | Dirichlet prior | Prior sample size |
|---|---|---|---|
| Add-one (Laplace) | $\frac{\mathit{tf}_{t,d} + 1}{L_d + V}$ | all $a_t = 1$ (uniform) | $V$ |
| Add-half | $\frac{\mathit{tf}_{t,d} + \frac12}{L_d + V/2}$ | all $a_t = \frac12$ (Jeffreys, Theorem 4) | $V/2$ |
| Add-$\alpha$ (Lidstone) | $\frac{\mathit{tf}_{t,d} + \alpha}{L_d + V\alpha}$ | all $a_t = \alpha$ | $V\alpha$ |
| Dirichlet | $\frac{\mathit{tf}_{t,d} + \mu P(t \mid M_c)}{L_d + \mu}$ | $a_t = \mu P(t \mid M_c)$ | $\mu$ |
| Jelinek–Mercer, document weight $\lambda$ | $\lambda\frac{\mathit{tf}_{t,d}}{L_d} + (1 - \lambda)P(t \mid M_c)$ | $a_t = \mu_dP(t \mid M_c)$, $\mu_d = L_d\frac{1-\lambda}{\lambda}$ | $\mu_d$, proportional to $L_d$ |
Here $P(t \mid M_c)$ is the collection model, the share of term $t$ among all tokens of the collection. The first four rows are the posterior mean formula with the stated $a_t$. For the last, Language Models for Retrieval shows that Dirichlet smoothing equals Jelinek–Mercer with the document weight $L_d/(L_d + \mu)$. Setting $L_d/(L_d + \mu_d) = \lambda$ and solving gives $\mu_d = L_d(1 - \lambda)/\lambda$: Jelinek–Mercer is a Dirichlet posterior mean whose prior grows in proportion to the document's length, so that every document, long or short, gives the same weight to the collection. A fixed Dirichlet prior lets long documents speak more for themselves.
Not every smoothing method is a posterior mean of this kind. Absolute discounting, also described on that page, subtracts a constant $\delta$ from every positive count, giving $(\mathit{tf}_{t,d} - \delta)/L_d$ plus a collection term for a term the document contains. Take a document containing terms $s$ and $t$, with $\mathit{tf}_{s,d} \ge 2$, and move one occurrence of $s$ to $t$. That changes neither $L_d$ nor the number of distinct terms $U_d$. A Dirichlet prior may depend on $L_d$, as Jelinek–Mercer's does, but not on the counts, so under any such prior the estimate for $t$ rises by exactly $1/(L_d + A)$. Under absolute discounting it rises by $1/L_d$, since the collection term $\delta U_dP(t \mid M_c)/L_d$ is unchanged. The two agree only if $A = 0$, which no Dirichlet prior allows. So absolute discounting is not a Dirichlet posterior mean for any prior.
Example (a uniform prior over a large vocabulary is not uninformative). Under add-one smoothing an average Cranfield abstract, with 111.6 tokens and 68.1 distinct terms, receives 4,803 pseudo-tokens from the prior, more than forty times its own length. The posterior predictive probability that the next token is a term the abstract has not used is the prior mass of the unused terms, $(V - \text{distinct})/(L_d + V)$; averaged over the abstracts, add-one smoothing gives it 96.4%, and add-half smoothing, the Jeffreys prior, still gives 94.3%. How often is a token of an abstract actually the only occurrence of its term in that abstract? For 42.0% of the 156,309 tokens. The uniform prior asserts, before any data, that the next word is almost certainly one never seen, because spreading belief evenly over 4,803 possibilities is a strong statement when the data hold only a hundred tokens. A flat prior is flat on the simplex, not neutral about what the data will look like.
Choosing the strength of the prior on Cranfield
The Dirichlet prior centered on the collection fixes the shape of the prior; its strength $\mu$ remains. Three criteria choose $\mu$ from the abstracts alone, without queries or relevance judgments.
- Marginal likelihood. Treat $\mu$ as a hyperparameter and choose the value under which the observed abstracts are most probable. By Theorem 8 of Priors, Posteriors and Conjugate Families, the probability of an abstract's token sequence under the prior is $Z(\mathbf{a} + \mathbf{tf})/Z(\mathbf{a})$, which in logarithms is $\ln\Gamma(\mu) - \ln\Gamma(\mu + L_d) + \sum_t[\ln\Gamma(\mathit{tf}_{t,d} + \mu P(t \mid M_c)) - \ln\Gamma(\mu P(t \mid M_c))]$. Summing over the 1,400 abstracts and maximizing over $\mu$ is the empirical Bayes method of Hierarchical Models and Shrinkage; comparing models by their marginal likelihoods is the subject of Bayes Factors and Posterior Model Probabilities.
- Leave-one-out prediction. Remove each token in turn from its abstract, predict it from the rest with the posterior mean, $(\mathit{tf}_{t,d} - 1 + \mu P(t \mid M_c))/(L_d - 1 + \mu)$, and sum the logarithms of the predicted probabilities.
- Held-out halves. Split each abstract's tokens at random into two halves, estimate from one, and score the other; repeat for 30 random splits.
The collection model $P(t \mid M_c)$ is computed from all abstracts, including the one being predicted, as on Language Models for Retrieval; it is a slight advantage to every setting alike.
The marginal likelihood is largest at $\mu = 114.6$ and the leave-one-out log-likelihood at $\mu = 123.3$. At $\mu = 114.6$ an average abstract gives its own counts a document weight of $111.6/(111.6 + 114.6) = 0.49$: the data say that an abstract's own words and the collection deserve about equal weight. The held-out halves agree: over the 30 splits (seeds 1 to 30) the best $\mu$ had a median of 113.1 (interquartile range 112.6 to 113.6), although the half-abstracts average only 55.8 tokens. The best strength did not halve with the length of the text, which is what a fixed Dirichlet prior assumes and Jelinek–Mercer smoothing does not.
The table compares the priors by the leave-one-out log-likelihood per token (in nats) and its exponential, the perplexity (Perplexity and Source Coding): the effective number of equally likely choices the model faces for each held-out token. The best symmetric prior, $\text{Dirichlet}(\alpha, \ldots, \alpha)$ with $\alpha$ chosen by the same criterion, has $\alpha = 0.019$.
| Prior | Pseudo-tokens | Log-likelihood per token | Perplexity |
|---|---|---|---|
| Add-one, $\alpha = 1$ | 4,803 | $-7.793$ | 2,425 |
| Add-half, $\alpha = \tfrac12$ (Jeffreys) | 2,401.5 | $-7.525$ | 1,854 |
| Best symmetric, $\alpha = 0.019$ | 91 | $-6.594$ | 731 |
| Collection-centered, $\mu = 123$ | 123 | $-6.044$ | 421 |
| Collection-centered, $\mu = 300$ | 300 | $-6.108$ | 450 |
| Collection-centered, $\mu = 2{,}000$ | 2,000 | $-6.458$ | 638 |
Two things matter, in this order: where the prior is centered and how strong it is. Centering on the collection model instead of the uniform distribution lowers the perplexity from 731 to 421 at the best strength of each, because common words are common in every abstract. Within the uniform family, the strength makes a large difference: the Jeffreys prior is better than add-one but far worse than a symmetric prior with 91 pseudo-tokens instead of its 2,401.5.
How do these choices rank documents? The mean average precision of Dirichlet-smoothed query likelihood over the 225 Cranfield queries, computed as on Language Models for Retrieval, is 0.278 at the marginal-likelihood choice $\mu = 114.6$ and 0.279 at the leave-one-out choice, against 0.284 at $\mu = 200$ and $\mu = 300$, the best values that page found, and 0.266 at $\mu = 2{,}000$. The script reproduces that page's 0.284 and 0.266. So the prior that best predicts an abstract's own words is somewhat weaker than the one that ranks best, and both are far weaker than 2,000.
Example (predicting text and ranking documents are different tasks). "The best prior for the documents must also be the best for retrieval." The two criteria ask different questions. The marginal and leave-one-out likelihoods score how well the smoothed model predicts the abstract's own tokens. Retrieval scores how well the probability of a query, a different and shorter text, orders the abstracts. Zhai and Lafferty (2004, abstract) found that verbose queries generally need more aggressive smoothing than keyword queries, which they take to suggest that smoothing plays two roles: making the estimated document model more accurate, and explaining the noninformative words of the query. Cranfield's full-sentence questions are verbose queries of this kind. On Cranfield the mean average precision is 0.2782 at $\mu = 114.6$ and 0.2836 at $\mu = 300$, a small difference of 0.005, while $\mu = 2{,}000$, at 0.2658, costs 0.018. The data-driven choices made without any relevance judgments land within a few thousandths of the best ranking.
What is proved, what is cited, what the experiment shows
Proved on this page: the change of variables for a prior and the non-invariance of flat priors (Theorem 1, with the uniform-to-logistic example); the condition for a proper posterior under the improper Beta(0, 0) prior (Theorem 2); the invariance of Jeffreys' prior in one and several dimensions (Theorem 3 and the paragraph after it); the Jeffreys priors of the Bernoulli, Poisson, normal location, normal scale, joint normal and categorical models (Theorem 4, including the determinant identity it uses); the coverage of the joint-Jeffreys interval for a normal mean; the Jeffreys prior on the log odds equal to the half-success likelihood; the smoothing table, Jelinek–Mercer as a document-dependent Dirichlet prior, and absolute discounting as no Dirichlet posterior mean.
Used from other pages: the Fisher informations of the standard families, the reparametrization rule $I^*(\eta) = I(g(\eta))g'(\eta)^2$ and the information matrix (Fisher Information and the Cramér–Rao Bound); the change-of-variables rule for densities (Random Variables and Distribution Functions); the normal–inverse-gamma posterior and the Dirichlet predictive (Priors, Posteriors and Conjugate Families); the Dirichlet posterior mean (Naive Bayes); the equivalence of Dirichlet and length-weighted Jelinek–Mercer smoothing (Language Models for Retrieval); the existence condition for the logistic estimate (Deviance and Inference for Generalized Linear Models).
Cited: Jeffreys' invariance proposal (Jeffreys, 1946, abstract); the definitions and warnings about noninformative, improper and weakly informative priors, and the improper Beta(0, 0) case (Gelman et al., 2013, §§2.4, 2.8, 2.9, 3.2); the default Cauchy priors for logistic regression and their motivation (Gelman et al., 2008, abstract and §2); the two roles of smoothing in retrieval and the heavier smoothing preferred by verbose queries (Zhai and Lafferty, 2004, abstract).
Computed: the posteriors for 0 and 10 of 40 under four priors (the NHANES sample; National Center for Health Statistics, 2020); the coverage of the joint-Jeffreys interval, with Student's sleep data (Student, 1908); the horse-kick posterior of Problem 1 (Bortkiewicz, 1898); on the Cranfield abstracts (Cleverdon, 1967), the predicted unseen mass, the three choices of $\mu$ (the held-out halves over 30 seeds), the per-token log-likelihoods and the mean average precision. These describe Cranfield's 1,400 short abstracts; on collections of longer documents the best $\mu$ differs, as the retrieval article's citation of the usual value of 2,000 shows. Every number is computed and checked against this page by tools/experiments/statistics/choosing-priors.py, which loads Cranfield with tools/experiments/cranfield/cranexp.py.
Worked problems
The problems were written for this article, on its running data and on small exact cases. Each number is recomputed by the article's script.
Problem 1 (Jeffreys' prior for horse kicks). For Bortkiewicz's 122 deaths in 200 corps-years, find the posterior of the Poisson rate under Jeffreys' prior and compare it with the $\text{Gamma}(1, 1)$ analysis of Priors, Posteriors and Conjugate Families, Problem 2.
Solution. By Theorem 4, part 2, the posterior is $\text{Gamma}(122.5, 200)$, with mean $122.5/200 = 0.6125$ and central 95% interval 0.5088 to 0.7256. The $\text{Gamma}(1, 1)$ prior gave mean 0.6119 and interval 0.5086 to 0.7247. The Jeffreys prior adds half a death and no exposure; the gamma prior adds one death and one corps-year. With 200 corps-years the difference is in the fourth decimal place. Jeffreys' prior is improper, but the posterior is proper because the data include at least one corps-year.
Problem 2 (Jeffreys' prior on the log odds). Compute the Jeffreys prior directly on the log odds scale $\varphi$ for a Bernoulli observation, and check that it is the Jeffreys prior for $p$ transformed by Theorem 1.
Solution. With $p = 1/(1 + e^{-\varphi})$, $\ln f = x\varphi - \ln(1 + e^\varphi)$, so the score is $x - p$ and $I(\varphi) = \operatorname{Var}(X) = p(1-p)$ (the information is the variance of the sufficient statistic in the natural parameter, as Exponential Families proves). So $\pi_J(\varphi) \propto \sqrt{p(1-p)} = e^{\varphi/2}/(1 + e^\varphi)$. Transforming $\text{Beta}(\tfrac12, \tfrac12)$ from $p$ to $\varphi$ multiplies $p^{-1/2}(1-p)^{-1/2}$ by $dp/d\varphi = p(1-p)$, giving $p^{1/2}(1-p)^{1/2}$, the same function, as Theorem 3 requires. Unlike the flat prior on $\varphi$, it is proper: $\int e^{\varphi/2}/(1 + e^\varphi)\,d\varphi = \int_0^1 p^{-1/2}(1-p)^{-1/2}\,dp = \pi$.
Problem 3 (Jelinek–Mercer smoothing as a prior). Language Models for Retrieval chose a document weight $\lambda = 0.2$ for Jelinek–Mercer smoothing on Cranfield. What Dirichlet prior strength does that imply for an average abstract, and for an abstract of 300 tokens?
Solution. By the table, $\mu_d = L_d(1 - \lambda)/\lambda = 4L_d$. For the average abstract, $4 \times 111.65 = 446.6$ pseudo-tokens (the exact average is $156{,}309/1{,}400 = 111.649$); for 300 tokens, 1,200. Every abstract gives its own counts weight 0.2 and the collection 0.8, regardless of length. The marginal likelihood above prefers a fixed strength near 115, which gives an average abstract weight 0.49 and a 300-token abstract weight $300/414.6 = 0.72$. The two methods agree only for an abstract of $\mu\lambda/(1 - \lambda) = 114.58/4 = 28.6$ tokens at the marginal-likelihood choice of $\mu$.
Problem 4 (add-half smoothing of a small class). Estimate the term probabilities of the China class of Naive Bayes, with counts $(5, 1, 1, 1, 0, 0)$ over six terms, under Jeffreys' prior, and compare with add-one smoothing.
Solution. Under $\text{Dirichlet}(\tfrac12, \ldots, \tfrac12)$ the posterior is $\text{Dirichlet}(5.5, 1.5, 1.5, 1.5, 0.5, 0.5)$ with total 11, so the posterior means are $5.5/11 = 0.5000$ for Chinese, $1.5/11$ for each city, and $0.5/11 = 0.0455$ for Tokyo and Japan. Add-one gives $6/14 = 0.4286$ and $1/14 = 0.0714$. The Jeffreys prior adds three pseudo-tokens instead of six and so stays closer to the counts. With six terms neither prior is overwhelming; with Cranfield's 4,803 terms, as computed above, both are.
Problem 5 (the Beta(0, 0) prior with one event). Under the prior $\pi(p) \propto 1/[p(1-p)]$, find the posterior for 1 hypertensive among 40 and for none. What does the second case say about using this prior as a default?
Solution. By Theorem 2, with $x = 1$ the posterior is $\text{Beta}(1, 39)$, with mean $1/40 = 0.025$, the maximum likelihood estimate, and density $39(1-p)^{38}$, largest at $p = 0$. With $x = 0$ the function $p^{-1}(1-p)^{39}$ has an infinite integral near 0, so there is no posterior. A default prior that fails for exactly the samples where a prior is most needed, those with no events, is not a usable default for proportions near 0. The uniform and Jeffreys priors give proper posteriors for every count.
Summary
- A prior uniform on one scale is not uniform on another (Theorem 1): uniform on $p$ is logistic on the log odds, and flat on the log odds is the improper Beta(0, 0) prior $1/[p(1-p)]$, whose posterior exists only for $0 < x < n$ (Theorem 2).
- Improper priors are usable when the posterior is proper; that must be checked for each model and data set.
- Jeffreys' prior $\pi_J \propto \sqrt{\det I(\boldsymbol\theta)}$ gives the same answer on every scale (Theorem 3): $\text{Beta}(\tfrac12, \tfrac12)$ for a proportion, $\lambda^{-1/2}$ for a Poisson mean, flat for a normal mean, $1/\sigma$ for a normal scale, $\text{Dirichlet}(\tfrac12, \ldots)$ for categories (Theorem 4). The joint rule for $(\mu, \sigma)$ gives $1/\sigma^2$ and intervals with coverage 0.9363 at $n = 10$.
- Weakly informative priors are proper and deliberately weaker than real knowledge; Gelman and colleagues' Cauchy(0, 2.5) for logistic coefficients is a longer-tailed version of the Jeffreys prior on the log odds. Near a boundary, such as zero events, no default is neutral.
- Add-one, add-half, Lidstone, Dirichlet and Jelinek–Mercer smoothing are Dirichlet posterior means; a uniform prior over 4,803 terms predicts 96.4% unseen mass where 42.0% is observed.
- On Cranfield the marginal likelihood chooses $\mu = 114.6$, leave-one-out $\mu = 123.3$, held-out halves a median of 113.1; retrieval prefers 200 to 300, with mean average precision 0.278 against 0.284.
What we have learned
There is no prior that adds no information. A flat prior is flat only on a chosen scale; a uniform prior over many categories is a strong prediction about unseen ones; an improper prior can leave no posterior at all. Jeffreys' rule removes the dependence on the scale; it gives good intervals for a proportion, as Credible Intervals and How They Differ from Confidence Intervals shows, and exact ones for a normal mean, but not automatically for several parameters. In practice the useful priors are either weakly informative, chosen to rule out absurd values, or learned from related data, as the Dirichlet prior on Cranfield was learned from the collection itself.
Learning the prior's strength from the data, by maximizing the marginal likelihood, treats the prior as one level of a larger model. Bayes Factors and Posterior Model Probabilities uses the same marginal likelihood to compare models, and Hierarchical Models and Shrinkage develops the idea of a prior estimated from many related units.
References
- Bortkiewicz, L. von (1898). Das Gesetz der kleinen Zahlen. B. G. Teubner, Leipzig. §12 (deaths from horse kicks in the Prussian army), as tabulated in The Poisson Distribution. record · 5 pages
- Cleverdon, C. W. (1967). The Cranfield tests on index language devices. Aslib Proceedings, 19(6), 173–194. record · 8 pages
- Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., and Rubin, D. B. (2013). Bayesian Data Analysis (3rd ed.). CRC Press. Sections 2.4, 2.8, 2.9 and 3.2. Authors' electronic edition, http://www.stat.columbia.edu/~gelman/book/ record · 7 pages
- Gelman, A., Jakulin, A., Pittau, M. G., and Su, Y.-S. (2008). A weakly informative default prior distribution for logistic and other regression models. Annals of Applied Statistics, 2(4), 1360–1383. https://doi.org/10.1214/08-AOAS191 record
- Jeffreys, H. (1946). An invariant form for the prior probability in estimation problems. Proceedings of the Royal Society of London, Series A, 186(1007), 453–461. https://doi.org/10.1098/rspa.1946.0056 record
- National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018: DEMO_J, BPX_J and BPQ_J data files. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 95 pages
- Student (1908). The probable error of a mean. Biometrika, 6(1), 1–25. record · 17 pages
- Zhai, C., and Lafferty, J. (2004). A study of smoothing methods for language models applied to information retrieval. ACM Transactions on Information Systems, 22(2), 179–214. https://doi.org/10.1145/984321.984322 record · 2 pages