Bayes Factors and Posterior Model Probabilities

How the data change the odds between two models. The marginal likelihood defined as the model's prior predictive probability and derived in closed form for the beta-binomial, gamma-Poisson and normal models from one identity; Bayes factors and posterior odds; Lindley's paradox proved for the normal mean and shown on CDC WONDER deaths, where a p-value of 0.044 meets a Bayes factor that no normal prior centered on no change can push above 2.3; Laplace's approximation derived and the Bayesian information criterion obtained from it with an error that does not vanish, measured in a seeded simulation; and the dependence of every Bayes factor on the prior for the parameters. Worked problems on the young adults' hypertension sample, the sample size at which a just-significant result favors the null, and the criterion computed by hand.

Among the adults aged 40 to 59 examined in the National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018, 457 of 743 men (61.5%) and 462 of 831 women (55.6%) have hypertension by the 2017 ACC/AHA definition (Ostchega et al., 2020). The test of Comparing Two Proportions gives $z = 2.38$ and a two-sided p-value of 0.018, so at the 5% level the proportions differ. Among adults aged 60 and over, the shares are 78.9% of 976 men and 81.1% of 936 women, with $|z| = 1.20$ and $p = 0.23$: no evidence of a difference, and no statement either that there is none. Neither number says how probable it is that the two proportions are equal, or whether the second comparison is evidence for equality. What do the data say about two competing models, each with unknown parameters, and how far does the answer depend on the prior?

The Bayes factor answers in the terms of Priors, Posteriors and Conjugate Families. Each model gets a prior distribution for its parameters, and the data's probability under the model, averaged over that prior, is the model's marginal likelihood. The Bayes factor is the ratio of two marginal likelihoods. It is the factor by which the data multiply the odds of one model against the other. For the men and women aged 40 to 59, with uniform priors, it is 1.05: the data are almost exactly as probable under "equal" as under "different", despite $p = 0.018$. For those aged 60 and over it is 10.6 in favor of equality.

First we define the marginal likelihood and prove that it turns prior odds into posterior odds, and that it measures each model's record as a forecaster. Then we derive marginal likelihoods in closed form for the conjugate families from one identity, and compute the Bayes factors above. Next we prove Lindley's paradox: for a fixed test statistic, the Bayes factor favors the null more and more strongly as the sample grows. CDC WONDER death counts show it at a scale where it matters. Then we derive Laplace's approximation to the marginal likelihood, obtain the Bayesian information criterion (BIC) from it, and measure the size of the error that the criterion leaves. Finally we show how strongly every Bayes factor depends on the prior for the parameters, and what can be done about it. The article assumes Priors, Posteriors and Conjugate Families for the conjugate posteriors and Likelihood Ratio, Wald and Score Tests for the likelihood ratio statistic.

Data. The NHANES adults are the 5,182 nonpregnant adults of Comparing Two Proportions with a blood pressure reading, treated here as two independent binomial samples per age group, without the survey's weights or design. The young adults' sample of the course is the one of Confidence Intervals for a Proportion: 10 with hypertension among 40 adults aged 18 to 39, drawn from 1,696 such adults of whom 389 (22.94%) have it. The death counts are from CDC WONDER, as in Poisson Regression and Rate Models. Every number is computed by tools/experiments/statistics/bayes-factors.py.

Posterior odds and the marginal likelihood

Two models $M_0$ and $M_1$ are proposed for data $\mathbf{x}$. Model $M_k$ has a parameter $\theta_k$ (a number or a vector of $d_k$ numbers), a pmf or pdf $f(\mathbf{x} \mid \theta_k, M_k)$, and a prior density $\pi_k(\theta_k)$. Each model also has a prior probability $P(M_k)$, with $P(M_0) + P(M_1) = 1$. The marginal likelihood of $M_k$ is

$$m_k(\mathbf{x}) = m(\mathbf{x} \mid M_k) = \int f(\mathbf{x} \mid \theta_k, M_k)\,\pi_k(\theta_k)\,d\theta_k ,$$

a sum when $\theta_k$ is discrete. It is the probability (or density) of the observed data under $M_k$ before the parameter is known: the likelihood averaged over the prior, not maximized over the parameter. Kass and Raftery (1995, §3.1) note that it is also called the integrated likelihood, and that it is the predictive probability of the data calculated before any data became available; it is also called the evidence. The Bayes factor for $M_1$ against $M_0$ is

$$B_{10} = \frac{m_1(\mathbf{x})}{m_0(\mathbf{x})}, \qquad B_{01} = 1/B_{10}.$$

Here is the idea of the first theorem: Bayes' theorem applied to the two models instead of to a parameter says that the data multiply the prior odds by a factor that does not depend on the prior odds.

Theorem 1 (posterior odds). If $m_0(\mathbf{x})$ and $m_1(\mathbf{x})$ are positive and finite,

$$\frac{P(M_1 \mid \mathbf{x})}{P(M_0 \mid \mathbf{x})} = B_{10} \times \frac{P(M_1)}{P(M_0)}, \qquad P(M_1 \mid \mathbf{x}) = \frac{P(M_1)\,m_1(\mathbf{x})}{P(M_0)\,m_0(\mathbf{x}) + P(M_1)\,m_1(\mathbf{x})}.$$

Proof. The pair (model, parameter) is drawn first and the data second. The probability of the model $M_k$ and the data $\mathbf{x}$ together is $P(M_k)$ times the probability of $\mathbf{x}$ given $M_k$, which by the law of total probability over the parameter is $m_k(\mathbf{x})$. Dividing by the total, $P(M_0)m_0(\mathbf{x}) + P(M_1)m_1(\mathbf{x})$, gives the second formula (Bayes' theorem, as in Predictive Values and Bayes' Theorem). Dividing the formula for $M_1$ by the one for $M_0$ cancels the denominator and gives the first. $\square$

With equal prior probabilities the posterior odds equal the Bayes factor, and $P(M_1 \mid \mathbf{x}) = B_{10}/(1 + B_{10})$. For the men and women aged 40 to 59 below, $B_{10} = 1.05$ gives a posterior probability of 0.51 that the proportions differ.

The Bayes factor is not the likelihood ratio statistic of Likelihood Ratio, Wald and Score Tests, which compares the two likelihoods at their maxima. A model with more parameters can always fit at least as well at its maximum. Averaged over a prior, its extra freedom costs something: the prior spreads probability over parameter values that the data then rule out. That cost is the automatic penalty for complexity that the rest of the article measures.

A Bayes factor is a forecasting record. The marginal likelihood can be built one observation at a time.

Proposition 1 (sequential prediction). For data $x_1, \ldots, x_n$ in any order,

$$m(\mathbf{x}) = \prod_{i=1}^n f(x_i \mid x_1, \ldots, x_{i-1}),$$

where $f(x_i \mid x_1, \ldots, x_{i-1}) = \int f(x_i \mid \theta, x_1, \ldots, x_{i-1})\,\pi(\theta \mid x_1, \ldots, x_{i-1})\,d\theta$ is the posterior predictive probability of the $i$-th observation given the earlier ones (for $i = 1$, the prior predictive).

Proof. The marginal likelihood is the joint pmf or pdf of $(x_1, \ldots, x_n)$ under the model with the parameter integrated out, and any joint distribution factors as $m(x_1)\,m(x_2 \mid x_1) \cdots m(x_n \mid x_1, \ldots, x_{n-1})$ (the multiplication rule). Each conditional $m(x_i \mid x_1, \ldots, x_{i-1})$ is the ratio of two marginals, and expanding both as integrals over $\theta$ and applying Bayes' theorem to the denominator gives the predictive integral above. $\square$

So $\ln B_{10} = \sum_i \ln f(x_i \mid \text{past}, M_1) - \sum_i \ln f(x_i \mid \text{past}, M_0)$: the difference between the two models' cumulative logarithmic scores as forecasters, each forecast made from the data before it. Kass and Raftery (1995, §3.2) make this reading, which does not need either model to be true. The logarithmic score is the proper scoring rule of Calibration and Proper Scoring Rules.

Example (forty forecasts). For the 40 young adults under a binomial model with a uniform prior on the proportion $p$, the posterior after $s$ cases among $i$ adults is Beta$(1 + s, 1 + i - s)$ (Priors, Posteriors and Conjugate Families), and its mean $(s + 1)/(i + 2)$ is the predictive probability that the next adult has hypertension. The first three adults drawn do not have it, so the first three forecasts of "no" are $1/2$, $2/3$ and $3/4$. The product of all 40 forecasts, in the order the adults were drawn, is $10!\,30!/41! = 2.88 \times 10^{-11}$. That is the marginal likelihood of the sequence, and the order does not matter: any order gives the same product, because the marginal likelihood is a function of the whole sample.

Reading the size. Kass and Raftery (1995, §3.2) give rough categories, pooling two of Jeffreys's: a Bayes factor $B_{10}$ from 1 to 3.2 is "not worth more than a bare mention", 3.2 to 10 "substantial", 10 to 100 "strong" and above 100 "decisive" evidence against $M_0$. On the scale of $2\ln B_{10}$, which matches the scale of deviances, their own categories are 0 to 2, 2 to 6 ("positive"), 6 to 10 ("strong") and above 10 ("very strong"). They present them as guidelines, not as a calibration. The same scale read the other way measures evidence for $M_0$, which a p-value cannot do.

Marginal likelihoods in closed form

For conjugate families the marginal likelihood needs no integration. The idea: Bayes' theorem, $\pi(\theta \mid \mathbf{x}) = f(\mathbf{x} \mid \theta)\pi(\theta)/m(\mathbf{x})$, can be solved for $m(\mathbf{x})$, and when the posterior is a known density the right side is known at every $\theta$.

Theorem 2 (the marginal likelihood from any parameter value). For every $\theta$ with $\pi(\theta \mid \mathbf{x}) > 0$,

$$m(\mathbf{x}) = \frac{f(\mathbf{x} \mid \theta)\,\pi(\theta)}{\pi(\theta \mid \mathbf{x})}.$$

Proof. Multiply Bayes' theorem by $m(\mathbf{x})$ and divide by $\pi(\theta \mid \mathbf{x})$. $\square$

The right side does not depend on $\theta$, so all the terms in $\theta$ must cancel. That is a check on any calculation. The three conjugate cases follow.

Corollary 1 (closed forms).

  1. Binomial with a beta prior. For $x$ successes in $n$ trials, $f(x \mid p) = \binom{n}{x}p^x(1-p)^{n-x}$, prior Beta$(a, b)$ with density $p^{a-1}(1-p)^{b-1}/\mathrm{B}(a, b)$, and posterior Beta$(a + x, b + n - x)$: $$m(x) = \binom{n}{x}\frac{\mathrm{B}(a + x,\, b + n - x)}{\mathrm{B}(a, b)}.$$ With the uniform prior, $a = b = 1$, this is $1/(n + 1)$ for every $x$: before the data, every count from 0 to $n$ is equally probable.

  2. Poisson with a gamma prior. For counts $x_1, \ldots, x_n$ with total $s$, $f(\mathbf{x} \mid \lambda) = \lambda^s e^{-n\lambda}/\prod_i x_i!$, prior Gamma$(a, b)$ with shape $a$ and rate $b$, density $b^a\lambda^{a-1}e^{-b\lambda}/\Gamma(a)$, and posterior Gamma$(a + s, b + n)$: $$m(\mathbf{x}) = \frac{1}{\prod_i x_i!}\cdot\frac{b^a\,\Gamma(a + s)}{\Gamma(a)\,(b + n)^{a + s}}.$$

  3. Normal mean with known variance. For a sample of $n$ from the normal distribution with mean $\mu$ and variance $\sigma^2$, the sample mean $\bar x$ is sufficient. With prior on $\mu$ normal with mean $\mu_0$ and variance $\tau_0^2$, the marginal density of $\bar x$ is normal with mean $\mu_0$ and variance $\sigma^2/n + \tau_0^2$.

Proof. In case 1, the numerator of Theorem 2 is $\binom{n}{x}p^{a + x - 1}(1-p)^{b + n - x - 1}/\mathrm{B}(a, b)$ and the denominator is $p^{a + x - 1}(1-p)^{b + n - x - 1}/\mathrm{B}(a + x, b + n - x)$. The powers of $p$ cancel and leave the formula. With $a = b = 1$, $\mathrm{B}(1 + x, 1 + n - x) = x!\,(n - x)!/(n + 1)!$, and multiplying by $\binom{n}{x} = n!/(x!\,(n - x)!)$ gives $n!/(n + 1)! = 1/(n + 1)$. In case 2, the numerator is $\lambda^{a + s - 1}e^{-(b + n)\lambda}\,b^a/(\Gamma(a)\prod_i x_i!)$ and the denominator is $(b + n)^{a + s}\lambda^{a + s - 1}e^{-(b + n)\lambda}/\Gamma(a + s)$; the powers of $\lambda$ and the exponentials cancel. In case 3, write $\bar x = \mu + (\bar x - \mu)$. Under the model, $\mu$ is normal with mean $\mu_0$ and variance $\tau_0^2$, and $\bar x - \mu$ is normal with mean 0 and variance $\sigma^2/n$, independent of $\mu$ (its conditional distribution given $\mu$ does not depend on $\mu$). A sum of independent normal variables is normal with the means and variances added (Moment Generating Functions and Slutsky's Theorem). $\square$

Case 1 is Theorem 4 of Priors, Posteriors and Conjugate Families with $m = n$. Cases 2 and 3 reduce at $n = 1$ to the predictive distributions of a single count and a single observation in its Theorems 5 and 6 (the latter with no data, so $\mu_n = \mu_0$ and $\tau_n = \tau_0$). All are obtained here without integration.

In case 3 the full data have a density that factors into a function of $\bar x$ and $\mu$ times a function of the data alone. That second factor is the same for any prior, so it cancels from every Bayes factor between models that share the normal sampling model, and the density of $\bar x$ can stand in for the marginal likelihood. The same holds for the binomial coefficient in case 1 and for $\prod_i x_i!$ in case 2, but only when both models use the same data in the same form. A Bayes factor between a binomial model for the count and a Bernoulli model for the sequence would be wrong by the factor $\binom{n}{x}$.

Back to the two age groups. For two proportions the models are

  • $M_0$: $p_1 = p_2 = p$, with $p$ uniform on $(0, 1)$;
  • $M_1$: $p_1$ and $p_2$ independent, each uniform on $(0, 1)$.

Under $M_1$ the two counts are independent and the marginal likelihood is the product of two cases of Corollary 1. Under $M_0$ the likelihood is $\binom{n_1}{x_1}\binom{n_2}{x_2}p^{x_1 + x_2}(1 - p)^{n_1 + n_2 - x_1 - x_2}$, a binomial likelihood in the pooled counts times the two coefficients. So

$$B_{10} = \frac{\mathrm{B}(x_1 + 1,\, n_1 - x_1 + 1)\,\mathrm{B}(x_2 + 1,\, n_2 - x_2 + 1)}{\mathrm{B}(x_1 + x_2 + 1,\, n_1 + n_2 - x_1 - x_2 + 1)}.$$

For the adults aged 40 to 59 the logarithms of the two marginal likelihoods (with the binomial coefficients) are $\ln m_0 = -13.38$ and $\ln m_1 = -13.34$, and $B_{10} = 1.05$. For those aged 60 and over $B_{01} = 10.6$, and with equal prior probabilities $P(M_0 \mid \mathbf{x}) = 0.91$.

Example (a small p-value is not a small probability of the null). A common reading of $p = 0.018$ is that the probability of equal proportions is about 0.018. Under these priors the posterior probability of equal proportions is 0.49. The p-value is the probability, computed under $M_0$ alone, of a difference at least as large as the one seen. The Bayes factor asks how probable the data are under $M_0$ relative to $M_1$. Under $M_1$, with uniform priors, most of the prior probability sits on differences far larger than 6 points (64% on differences above 20 points, since $P(|p_1 - p_2| > 0.2) = 0.8^2$ for independent uniform proportions). The data rule out those differences far more firmly than they rule out a difference of zero, and $M_1$ is penalized for the prior probability it spent there. The two numbers answer different questions, and the second depends on the prior, as the last sections show.

Example (evidence for the null). For the adults aged 60 and over, the p-value of 0.23 says only that the data are not surprising if the proportions are equal. The Bayes factor of 10.6 says the data are more than ten times as probable under equality as under uniform independent proportions: "strong" evidence on Kass and Raftery's scale. Evaluating the evidence in favor of a null hypothesis is the goal of the first two applications that Kass and Raftery (1995, §2) use to motivate Bayes factors.

Lindley's paradox

The p-value of 0.018 and the Bayes factor of 1.05 disagree, and the disagreement grows with the sample. Here is the idea: if the test statistic stays fixed while $n$ grows, the observed effect shrinks like $1/\sqrt{n}$. A null hypothesis predicts effects of that size well. An alternative whose prior does not shrink with $n$ spends nearly all its probability on effects that are too large, so its marginal likelihood falls like $1/\sqrt{n}$.

Theorem 3 (Lindley's paradox for a normal mean). Let $\bar x$ be the mean of $n$ observations from the normal distribution with mean $\mu$ and known variance $\sigma^2$, and let $z = \sqrt{n}(\bar x - \mu_0)/\sigma$. Compare $M_0\!: \mu = \mu_0$ with $M_1$: $\mu$ normal with mean $\mu_0$ and variance $\tau_0^2$. With $r = n\tau_0^2/\sigma^2$,

$$B_{01} = \sqrt{1 + r}\;\exp\Big(-\frac{z^2}{2}\cdot\frac{r}{1 + r}\Big).$$

Consequently: (a) for fixed $z$ and $\tau_0$, $B_{01} \to \infty$ as $n \to \infty$, growing like $\sqrt{n}\,(\tau_0/\sigma)\,e^{-z^2/2}$; (b) for fixed $n$ and $z$, $B_{01} \to \infty$ as $\tau_0 \to \infty$.

Proof. Under $M_0$, $\bar x$ is normal with mean $\mu_0$ and variance $\sigma^2/n$. Under $M_1$, by Corollary 1(3), it is normal with mean $\mu_0$ and variance $\sigma^2/n + \tau_0^2 = (\sigma^2/n)(1 + r)$. The factors of the data alone cancel, so $B_{01}$ is the ratio of the two normal densities at $\bar x$:

$$B_{01} = \frac{(2\pi\sigma^2/n)^{-1/2}\exp\{-(\bar x - \mu_0)^2 n/(2\sigma^2)\}}{(2\pi\sigma^2(1 + r)/n)^{-1/2}\exp\{-(\bar x - \mu_0)^2 n/(2\sigma^2(1 + r))\}} = \sqrt{1 + r}\,\exp\Big\{-\frac{z^2}{2}\Big(1 - \frac{1}{1 + r}\Big)\Big\},$$

which is the formula. For fixed $z$ the exponential factor stays between $e^{-z^2/2}$ and 1, while $\sqrt{1 + r} \to \infty$ as $r \to \infty$. In (a), $r = n\tau_0^2/\sigma^2 \to \infty$, and $\sqrt{1 + r}/\sqrt{n} \to \tau_0/\sigma$ while the exponential tends to $e^{-z^2/2}$. In (b), $r \to \infty$ as $\tau_0 \to \infty$. $\square$

Lindley (1957) described the conflict, as Robert (2014) recounts it: for a fixed value of the test statistic, and for almost any prior on the alternative, the Bayes factor for the null grows without bound with the sample size while the p-value stays the same. Robert (2014) reviews the paradox and its readings, and reports that Lindley himself pointed out that Jeffreys (1939) had already set out the facts, hence the name Jeffreys–Lindley paradox. Part (b) is sometimes called Bartlett's paradox, after Bartlett (1957), as Kass and Raftery (1995, §5.1) report: a prior made very diffuse in an attempt to be noninformative forces the Bayes factor toward $M_0$, whatever the data. Robert (2014) states the case $\tau_0 = \sigma$ and $z = 1.96$: with equal prior probabilities, $P(M_0 \mid \mathbf{x}) = 0.95$ at $n = 16{,}818$. The formula gives:

$n$ 10 100 1,000 10,000 100,000 1,000,000
$B_{01}$ at $z = 1.96$ 0.579 1.500 4.644 14.653 46.325 146.490
$P(M_0 \mid \mathbf{x})$ 0.367 0.600 0.823 0.936 0.979 0.993

The p-value is 0.05 in every column. The Bayes factor favors $M_1$ only in the first, and by the last it is "decisive" for the null.

How much evidence can a just-significant result carry at most? A prior centered on the null value with a free width gives a bound.

Proposition 2 (the largest Bayes factor against the null from a centered normal prior). In Theorem 3, for $|z| > 1$, the largest value of $B_{10}$ over all $\tau_0 > 0$ is

$$\max_{\tau_0} B_{10} = \frac{e^{(z^2 - 1)/2}}{|z|},$$

attained at $r = z^2 - 1$. For $|z| \le 1$ the supremum is 1, approached as $\tau_0 \to 0$.

Proof. By Theorem 3, $\ln B_{10} = g(r) = -\tfrac12\ln(1 + r) + \tfrac{z^2}{2}\cdot\tfrac{r}{1 + r}$, and $r$ ranges over $(0, \infty)$ as $\tau_0$ does. The derivative is $g'(r) = -\frac{1}{2(1 + r)} + \frac{z^2}{2(1 + r)^2} = \frac{z^2 - (1 + r)}{2(1 + r)^2}$. If $|z| > 1$ it is positive for $1 + r < z^2$ and negative after, so $g$ is largest at $1 + r = z^2$, where $g = -\ln|z| + \tfrac12(z^2 - 1)$. If $|z| \le 1$, $g' < 0$ on $(0, \infty)$, so $g < g(0) = 0$ and $g \to 0$ as $r \to 0$. $\square$

At $z = 1.96$ the bound is 2.11. Kass and Raftery (1995, §5.2) report that Edwards, Lindman and Savage (1963) computed this maximum over normal priors. No normal prior centered on the null makes a two-sided p-value of 0.05 more than about 2 to 1 evidence against it, which is at most "not worth more than a bare mention" on the scale above.

The paradox on death counts. CDC WONDER records 183,207 deaths of U.S. residents aged 45 to 54 in 2010, in a population of 45,006,716, and 183,247 in 2011, in 44,718,203: death rates of 407.1 and 409.8 per 100,000, a rate ratio of $\hat\psi = 1.0067$. Given the total of 366,454 deaths, the 2011 deaths are binomial with probability $44{,}718{,}203\,\psi/(45{,}006{,}716 + 44{,}718{,}203\,\psi)$, the conditional binomial model of Poisson Regression and Rate Models (its Theorem 5). The score test of $\psi = 1$ gives $z = 2.01$ and $p = 0.044$. The comparison was chosen to illustrate the paradox, by this rule: of the 220 comparisons of adjacent years from 1999 to 2019 in the 11 age groups, it is one of five whose $|z|$ lies between 1.9 and 2.1, and the only one of the five with a positive $z$. Its p-value is used only to illustrate the paradox, not as a finding.

Under $M_1$ give $\ln\psi$ a normal prior with mean 0 and standard deviation $s$; under $M_0$, $\psi = 1$. The script integrates the exact conditional binomial likelihood against each prior:

$s$ 0.001 0.0025 0.005 0.01 0.02 0.05 0.1 0.5
$B_{10}$ 1.134 1.667 2.258 1.948 1.170 0.495 0.250 0.050
$P(M_0 \mid \mathbf{x})$ 0.469 0.375 0.307 0.339 0.461 0.669 0.800 0.952

The standard error of $\ln\hat\psi$ is $\sqrt{1/183{,}207 + 1/183{,}247} = 0.00330$, so the log rate ratio is a normal estimate with $z = 2.01$, and Theorem 3 applies with $\sigma^2/n$ replaced by $0.00330^2$. The normal formula agrees with the exact integral to four significant figures at every $s$ in the table. The largest Bayes factor is 2.28, at $s = 0.0058$, as Proposition 2 predicts. A prior whose standard deviation for the year-to-year change is about 5% ($s = 0.05$) favors no change by 2 to 1; one with about 10% ($s = 0.1$) favors it by 4 to 1. The p-value is 0.044 whatever the prior.

Which answer is right? Both computations are correct; they answer different questions. The test asks whether the data are surprising if $\psi = 1$ exactly, and with 366,454 deaths a change of 0.67% is surprising. The Bayes factor asks whether $\psi = 1$ predicted the data better than a stated spread of alternatives. Whether a point null deserves a positive prior probability at all is a separate question. Kass and Raftery (1995, §8.1) defend sharp hypotheses: calling a theory correct means that deviations from it are "sufficiently minor to be irrelevant for all practical purposes at hand". They report Berger and Delampady's (1987) finding that, in testing a normal mean, a point null is a good approximation to an interval null when the width of the interval is less than about one half of the standard error of the sample mean. Gelman et al. (2013, §7.4) prefer to avoid positive prior probability on a single value of a continuous parameter.

Laplace's approximation

Most marginal likelihoods have no closed form. When the sample is large, the integrand $f(\mathbf{x} \mid \theta)\pi(\theta)$ is sharply peaked, and its integral is roughly the height of the peak times its width. Laplace's method makes that precise by fitting a normal curve to the peak.

The derivation. Let $\theta$ have dimension $d$, let $h(\theta) = \ln f(\mathbf{x} \mid \theta) + \ln\pi(\theta)$, let $\tilde\theta$ be its maximum (the posterior mode), and let $H = -\nabla^2 h(\tilde\theta)$ be the matrix of second derivatives with the sign changed, assumed positive definite (a nondegenerate interior maximum). The gradient vanishes at $\tilde\theta$, so a second-order Taylor expansion gives $h(\theta) \approx h(\tilde\theta) - \tfrac12(\theta - \tilde\theta)^{\mathsf T}H(\theta - \tilde\theta)$ near $\tilde\theta$. Exponentiating and integrating over $\mathbb{R}^d$ uses the normalizing constant of the multivariate normal density with covariance $H^{-1}$, $\int e^{-\frac12 u^{\mathsf T}Hu}\,du = (2\pi)^{d/2}|H|^{-1/2}$:

$$m(\mathbf{x}) \approx \hat m(\mathbf{x}) = (2\pi)^{d/2}\,|H|^{-1/2}\,f(\mathbf{x} \mid \tilde\theta)\,\pi(\tilde\theta).$$

This is Kass and Raftery's (1995) equation 4. The derivation replaces the integrand by a normal curve; the step that needs proof is that the regions where the expansion is poor contribute little, which holds when $H$ grows with $n$. Kass and Raftery (1995, §4.1.1) report that, under regularity conditions, $m = \hat m\,(1 + O(n^{-1}))$: the relative error is of order $1/n$, so the error in $\ln m$ is of order $1/n$. The same order holds for the variant that uses the maximum likelihood estimate $\hat\theta$ and the observed information $J(\hat\theta)$ in place of $\tilde\theta$ and $H$ (their equation 5); with the expected information instead, the order is $n^{-1/2}$.

Example (exact in the normal case). For the normal mean with a normal prior, $h$ is a quadratic function of $\mu$: the log-likelihood is quadratic in $\mu$ and so is the log prior. The Taylor expansion is then exact, and so is the approximation. That is the case of Priors, Posteriors and Conjugate Families in which the posterior is itself normal.

Example (the two age groups). With uniform priors the posterior mode is the maximum likelihood estimate. For the adults aged 40 to 59, the exact $\ln m_0$ is $-13.3803$ and Laplace's is $-13.3799$; the exact $\ln m_1$ is $-13.3359$ and Laplace's is $-13.3340$. The log Bayes factor is 0.0444 exactly and 0.0459 by Laplace's method.

The Bayesian information criterion

Laplace's approximation still needs the prior at the mode. Here is the idea of the criterion: the curvature $H$ grows in proportion to $n$, so $\ln|H|$ grows like $d\ln n$, and the terms that do not grow can be dropped.

Corollary 2 (BIC from Laplace). Use the maximum likelihood form of the approximation, with $\ell(\hat\theta) = \ln f(\mathbf{x} \mid \hat\theta)$ and the observed information $J(\hat\theta)$. For independent observations, $J(\hat\theta) = n\,\bar J$, where $\bar J = J(\hat\theta)/n$ is the average observed information per observation, so $\ln|J(\hat\theta)| = d\ln n + \ln|\bar J|$ and

$$\ln m(\mathbf{x}) = \ell(\hat\theta) - \frac{d}{2}\ln n + \Big[\ln\pi(\hat\theta) + \frac{d}{2}\ln(2\pi) - \frac12\ln|\bar J|\Big] + O(n^{-1}).$$

Under the model with true value $\theta_0$, $\hat\theta \xrightarrow{p} \theta_0$ and $\bar J \xrightarrow{p} I(\theta_0)$, the Fisher information per observation, so the bracket converges in probability to the constant $c = \ln\pi(\theta_0) + \frac{d}{2}\ln(2\pi) - \frac12\ln|I(\theta_0)|$. For two models,

$$\ln B_{10} = \big[\ell_1(\hat\theta_1) - \ell_0(\hat\theta_0)\big] - \frac{d_1 - d_0}{2}\ln n + (c_1 - c_0) + o_p(1).$$

Proof. Substitute $\ln|J(\hat\theta)| = d\ln n + \ln|\bar J|$ into the logarithm of Kass and Raftery's equation 5 and group the terms. Here $\hat\theta$ is the consistent root of the likelihood equation, which is the maximum likelihood estimate when that root is unique, as in both binomial models of this article. Its convergence and that of the average curvature at it are proved in one dimension in Asymptotics of the Maximum Likelihood Estimate (Theorem 1, Corollary 1 and the lemma on the curvature near the true value); the vector case follows the same argument coordinate by coordinate and is taken as given here. The function $\ln\pi$ is continuous at $\theta_0$ when the prior density is continuous and positive there, and the determinant and logarithm are continuous, so the bracket converges by the continuous mapping theorem (Proof of the Central Limit Theorem). $\square$

Dropping $c_1 - c_0$ leaves the Schwarz criterion

$$S = \ell_1(\hat\theta_1) - \ell_0(\hat\theta_0) - \frac{d_1 - d_0}{2}\ln n = \frac{W_{\mathrm{LR}}}{2} - \frac{d_1 - d_0}{2}\ln n,$$

where $W_{\mathrm{LR}}$ is the likelihood ratio statistic. Minus twice a model's own term, $\mathrm{BIC} = -2\ell(\hat\theta) + d\ln n$, is the Bayesian information criterion, and $S = -(\mathrm{BIC}_1 - \mathrm{BIC}_0)/2$. Model Selection uses it to choose among regression models. Schwarz (1978) gave a rigorous derivation for linear subfamilies of exponential families, as Kass and Raftery (1995, §4.1.3) report. The Bayesian Information Criterion and the Laplace Approximation proves Laplace's method in one dimension with its relative error of order $1/n$ (its Theorem 1) and proves BIC consistent for choosing among normal linear models.

How large is the error? Corollary 2 says it plainly: $\ln B_{10} - S$ tends to the constant $c_1 - c_0$, not to 0. Kass and Raftery (1995, §4.1.3) put it as $(S - \ln B_{10})/\ln B_{10} \to 0$ (their equation 9): the error is small relative to $\ln B_{10}$, which grows with $n$ when $M_1$ is true, but the relative error of $e^S$ as an approximation to $B_{10}$ is of order 1, "even for very large samples". The constant is zero only for particular priors. Kass and Raftery (1995, §4.1.3) report Kass and Wasserman's (1995) result that for nested models with a normal prior carrying the information of one observation (the "unit information prior"), and under further conditions (null orthogonality of the nuisance parameters and local alternatives), the error is of order $n^{-1/2}$. BIC therefore approximates the Bayes factor for one particular prior, and is accurate only to order 1 for others. The $n$ in $\ln n$ is the number of observations that contribute to the information: for binomial responses it is the total number of trials, not the number of groups (Kass and Raftery, 1995, §4.1.3).

Example (the criterion and the exact answer). For the adults aged 40 to 59 the likelihood ratio statistic is 5.651 and $\ln n = \ln 1{,}574 = 7.361$, so $S = (5.651 - 7.361)/2 = -0.855$: BIC prefers equal proportions, with $e^{0.855} = 2.35$ to 1. The exact log Bayes factor is 0.0444, slightly in favor of different proportions. The gap of 0.90 is the order-one term $c_1 - c_0$ of Corollary 2 for uniform priors, evaluated at these estimates, up to a term of order $1/n$.

The error measured. For the two-proportion models with uniform priors, $\ln\pi = 0$ for both models; $M_0$ has $d_0 = 1$ and information $1/(\bar p(1 - \bar p))$ per observation, with $\bar p$ the pooled proportion, and $M_1$ has $d_1 = 2$ and information matrix with diagonal $w_k/(p_k(1 - p_k))$ when a share $w_k$ of the observations is in group $k$. So

$$c_1 - c_0 = \ln(2\pi) - \tfrac12\ln\frac{w_1w_2}{p_1(1 - p_1)\,p_2(1 - p_2)} - \tfrac12\ln(2\pi) - \tfrac12\ln\big(\bar p(1 - \bar p)\big).$$

With $p_1 = 0.615$, $p_2 = 0.556$ and equal group sizes, the predicted limit of the BIC error $S - \ln B_{10}$ is $-0.900$. The script drew both groups from these binomial distributions, 30 seeds (1 to 30) for each size, and computed the exact, Laplace and Schwarz values of $\ln B_{10}$. The errors, as median (interquartile range) over the seeds:

Per group Laplace error BIC error
50 0.0218 (0.0214 to 0.0220) −0.873 (−0.886 to −0.861)
200 0.0055 (0.0055 to 0.0055) −0.896 (−0.900 to −0.892)
800 0.0014 (0.0014 to 0.0014) −0.898 (−0.902 to −0.896)
3,200 0.0003 (0.0003 to 0.0003) −0.899 (−0.901 to −0.898)
12,800 0.0001 (0.0001 to 0.0001) −0.900 (−0.901 to −0.899)

In this simulation the Laplace error fell by a factor of about 4 each time the sample grew fourfold, as an error of order $1/n$ does, while the BIC error settled at the predicted $-0.900$. A constant error of 0.9 in $\ln B_{10}$ is a factor of 2.5 in the Bayes factor: enough to move a comparison across a category of the scale, and never removed by more data.

Sensitivity to the prior on the parameters

Corollary 2 shows where the prior enters at large $n$: through $\ln\pi(\hat\theta)$, a term of order 1 that does not fade. That is unlike estimation, where the posterior forgets the prior as $n$ grows (Credible Intervals and How They Differ from Confidence Intervals).

Proposition 3 (changing the prior). If the priors $\pi_k$ are replaced by $\pi_k^{\text{new}}$, both continuous and positive near the estimates, then

$$B_{10}^{\text{new}} \approx B_{10}\cdot\frac{\pi_1^{\text{new}}(\hat\theta_1)/\pi_1(\hat\theta_1)}{\pi_0^{\text{new}}(\hat\theta_0)/\pi_0(\hat\theta_0)},$$

with relative error of order $1/n$ when the maximum likelihood form of Laplace's approximation is used for both.

Proof. In the maximum likelihood form, $\ln\hat m_k = \ell_k(\hat\theta_k) + \ln\pi_k(\hat\theta_k) + \frac{d_k}{2}\ln(2\pi) - \frac12\ln|J_k(\hat\theta_k)|$, and only $\ln\pi_k(\hat\theta_k)$ depends on the prior. Taking differences between the new and the old priors, for both models, gives the formula for the approximations; each approximation has relative error of order $1/n$ (Kass and Raftery, 1995, §4.1.2), so the ratio does too. $\square$

This is Kass and Raftery's equation 14, which they recommend for checking a Bayes factor over many priors at little cost (§5.2).

Example (Jeffreys priors). Replace the uniform priors in the comparison of the adults aged 40 to 59 by Jeffreys priors Beta$(\tfrac12, \tfrac12)$ (Choosing a Prior). The exact Bayes factor falls from 1.05 to 0.679. The ratio of prior densities at the estimates is 0.649, and Proposition 3 predicts 0.679 (0.6785 against the exact 0.6791). Both priors are called noninformative, and they disagree on which model the data favor.

Example (a prior with a free spread). Test the young adults' sample, 10 of 40, against $M_0$: $p = 0.2294$, the share among all 1,696 young adults. Under $M_1$ give $p$ the prior Beta$(k \cdot 0.2294,\, k(1 - 0.2294))$, centered on the null value with "prior sample size" $k$:

$k$ 0.01 0.1 1 2 10 100 1,000 10,000
$B_{01}$ 587.520 61.577 8.536 5.335 2.275 1.179 1.019 1.002

As $k \to \infty$ the prior concentrates on 0.2294, $M_1$ becomes $M_0$, and the Bayes factor tends to 1. As $k \to 0$ the beta prior puts nearly all its mass near 0 and 1, where the data are nearly impossible, and $B_{01}$ grows without bound: Bartlett's paradox again. Between them, the evidence for the true null ranges from barely worth a mention to decisive, with the data fixed.

What to do about it. Three practices follow from these examples.

  • Priors on the parameters being tested must be proper. An improper prior is defined only up to an arbitrary constant, which then multiplies the Bayes factor; Gelman et al. (2013, §7.4) note that the Bayes factor is defined only when each marginal density is proper.
  • Report the Bayes factor over a range of reasonable priors, as Kass and Raftery (1995, §5.2) recommend, and say which range was used. Proposition 3 makes that cheap.
  • Where no prior can be defended, BIC, or a Bayes factor under a unit information prior, gives a reference value (Kass and Raftery, 1995, §4.1.3), with its order-one error stated.

Posterior model probabilities and model averaging

With more than two models, Theorem 1 extends directly: $P(M_k \mid \mathbf{x}) = P(M_k)\,m_k(\mathbf{x})/\sum_j P(M_j)\,m_j(\mathbf{x})$, by the same proof with a sum over $j$ in the denominator. Model Selection computes such probabilities from BIC for the eleven best regression models of each size, which by Corollary 2 is accurate only up to the order-one terms of each model.

Posterior model probabilities can also weight predictions instead of choosing one model. The predictive distribution of a new observation $\tilde x$ is

$$f(\tilde x \mid \mathbf{x}) = \sum_k P(M_k \mid \mathbf{x})\, f(\tilde x \mid \mathbf{x}, M_k),$$

by the law of total probability over the models. Kass and Raftery (1995, §6) describe this composite prediction, which takes account of uncertainty about the model, and the algorithms proposed for it when the class of models is very large.

Gelman et al. (2013, §7.4) take a different view. They find Bayes factors helpful when the models are truly discrete, as two genetic hypotheses are, and a distraction when two models are the ends of a continuum, such as "all groups equal" and "all groups unrelated". They "generally prefer to replace a discrete set of models with an expanded continuous family." For the two age groups, that family puts a prior on the difference between the proportions and estimates it. For many groups it is the hierarchical model of Hierarchical Models and Shrinkage, which estimates how different the groups are instead of choosing between equal and unrelated.

What is proved here, what is cited, and what is computed

Proved on this page: posterior odds equal the Bayes factor times the prior odds (Theorem 1); the marginal likelihood as the product of sequential predictions (Proposition 1); the marginal likelihood from any parameter value (Theorem 2) and the closed forms for the beta-binomial, gamma-Poisson and normal models (Corollary 1); the Bayes factor of a normal mean and both limits of Lindley's paradox (Theorem 3); the largest Bayes factor over centered normal priors (Proposition 2); the decomposition of the Laplace approximation into the Schwarz criterion and a term of order 1, given the consistency of the estimate and of the average curvature (Corollary 2); the prior-change formula from the Laplace approximation (Proposition 3).

Derived here, proved elsewhere: Laplace's approximation itself, derived here by a Taylor expansion of the log posterior; it is proved in one dimension, with relative error of order $1/n$, in The Bayesian Information Criterion and the Laplace Approximation (Theorem 1), and the error terms of the $d$-dimensional version are cited.

Taken from other pages: the conjugate posteriors (Priors, Posteriors and Conjugate Families); the conditional binomial model for two Poisson counts (Poisson Regression and Rate Models); consistency of the maximum likelihood estimate and convergence of the average curvature, in one dimension (Asymptotics of the Maximum Likelihood Estimate); the continuous mapping theorem (Proof of the Central Limit Theorem).

Cited: the relative error $O(n^{-1})$ of Laplace's approximation and the $O(n^{-1/2})$ of its expected-information variant, the behavior of the Schwarz criterion (equation 9), the unit information prior result of Kass and Wasserman (1995), the prior-change approximation (equation 14), the interpretation scales, Bartlett's paradox, and Edwards, Lindman and Savage's maximum (Kass and Raftery, 1995); Lindley's statement of the paradox and its history (Lindley, 1957; Robert, 2014); the view of Bayes factors for continuous models (Gelman et al., 2013, §7.4); Schwarz's derivation (Schwarz, 1978).

Computed: every Bayes factor, probability and table on the page, by tools/experiments/statistics/bayes-factors.py, from the NHANES 2017–2018 hypertension data and the CDC WONDER deaths of 2010 and 2011. The simulation of the Laplace and BIC errors uses seeds 1 to 30 for each sample size.

Worked problems

These problems are posed for this article, on its own data and on the numbers of Theorem 3. Every number is recomputed by the article's script.

Problem 1 (the young adults against the true share). Ten of 40 young adults have hypertension. Compare $M_0$: $p = 0.2294$ with $M_1$: $p$ uniform on $(0, 1)$, using the sequence of 40 outcomes. Find $B_{01}$ and $P(M_0 \mid \mathbf{x})$ with equal prior probabilities.

Solution. Under $M_0$ the sequence has probability $0.2294^{10}(1 - 0.2294)^{30}$, with $0.2294 = 389/1{,}696$. Under $M_1$, by Proposition 1 and the example after it, the sequence has probability $10!\,30!/41! = 2.88 \times 10^{-11}$. (For the count instead of the sequence both probabilities gain the factor $\binom{40}{10}$, which cancels.) The ratio is $B_{10} = 0.177$, so $B_{01} = 5.65$ and $P(M_0 \mid \mathbf{x}) = 5.65/6.65 = 0.850$. The data favor the null value, which is the true share of the population the sample came from, by "substantial" evidence on Kass and Raftery's scale. Under the prior Beta$(k \cdot 0.2294, k \cdot 0.7706)$ with $k = 2$, which has the same prior sample size as the uniform but is centered on the null, the table of the last section gives a smaller $B_{01}$ of 5.335.

Problem 2 (when a just-significant result favors the null). In Theorem 3 take $\tau_0 = \sigma$ and $z = 1.96$. For what $n$ does $P(M_0 \mid \mathbf{x})$ reach 0.95 with equal prior probabilities, and with prior odds of 10 to 1 for $M_0$?

Solution. With equal prior probabilities, $P(M_0 \mid \mathbf{x}) = 0.95$ means $B_{01} = 0.95/0.05 = 19$. With $r = n$ the equation is $\sqrt{1 + n}\,\exp\{-1.9208\,n/(1 + n)\} = 19$, using $1.96^2/2 = 1.9208$. The left side increases in $n$ once $n > 2.84$, and solving numerically gives $n = 16{,}817.7$, so $n = 16{,}818$ is the first whole number, as Robert (2014) reports. With prior odds of 10 to 1, Theorem 1 requires $10\,B_{01} = 19$, so $B_{01} = 1.9$, and the equation gives $n = 163.3$: from $n = 164$ on, matching Robert's figure. At $n = 164$ the experiment rejects $M_0$ at the 5% level and leaves it with posterior probability above 0.95.

Problem 3 (the criterion by hand). For the adults aged 60 and over (770 of 976 men and 759 of 936 women), compute the Schwarz criterion $S$ and compare $e^{-S}$ with the exact $B_{01} = 10.6$.

Solution. The maximized log-likelihoods, with the binomial coefficients, are $\ell_0 = -7.588$ under equal proportions (pooled $\hat p = 1{,}529/1{,}912$) and $\ell_1 = -6.867$ under separate ones, so the likelihood ratio statistic is $2(-6.867 + 7.588) = 1.440$. With $n = 1{,}912$, $\ln n = 7.556$, and $S = (1.440 - 7.556)/2 = -3.058$. So BIC gives $B_{01} \approx e^{3.058} = 21.3$, about twice the exact 10.6 (exact $\ln B_{10} = -2.365$). The difference, 0.69 on the log scale, is the order-one term $c_1 - c_0$ of Corollary 2 evaluated at these estimates, up to a term of order $1/n$. Both say "strong" evidence for equal proportions; they disagree by a factor of 2.

Summary

Quantity Formula
Marginal likelihood $m_k(\mathbf{x}) = \int f(\mathbf{x} \mid \theta_k, M_k)\pi_k(\theta_k)\,d\theta_k = \prod_i f(x_i \mid \text{past}, M_k)$
Bayes factor, posterior odds $B_{10} = m_1/m_0$; posterior odds $= B_{10} \times$ prior odds
From any parameter value $m(\mathbf{x}) = f(\mathbf{x} \mid \theta)\pi(\theta)/\pi(\theta \mid \mathbf{x})$
Beta-binomial $\binom{n}{x}\mathrm{B}(a + x, b + n - x)/\mathrm{B}(a, b)$; uniform prior: $1/(n + 1)$
Normal mean, normal prior $B_{01} = \sqrt{1 + r}\,e^{-\frac{z^2}{2}\frac{r}{1 + r}}$, $r = n\tau_0^2/\sigma^2$
Lindley's paradox fixed $z$: $B_{01} \to \infty$ as $n \to \infty$; $\max B_{10} = e^{(z^2 - 1)/2}/|z|$
Laplace $\hat m = (2\pi)^{d/2}|H|^{-1/2}f(\mathbf{x} \mid \tilde\theta)\pi(\tilde\theta)$, relative error $O(n^{-1})$
Schwarz criterion $S = W_{\mathrm{LR}}/2 - \frac{d_1 - d_0}{2}\ln n$; $\ln B_{10} - S \to c_1 - c_0$, order 1
Prior change $B_{10}^{\text{new}} \approx B_{10}\cdot\frac{\pi_1^{\text{new}}/\pi_1}{\pi_0^{\text{new}}/\pi_0}$ at the estimates

What we have learned

A Bayes factor is the ratio of two marginal likelihoods, each the probability of the data averaged over a model's prior. It converts prior odds into posterior odds, it scores the two models as sequential forecasters, and it can measure evidence for a null hypothesis as well as against it. For conjugate families it has a closed form; otherwise Laplace's method approximates it with an error of order $1/n$, and the Bayesian information criterion approximates it with an error of order 1. Its weakness is the one Lindley's paradox exposes: through the prior on the parameters, the Bayes factor depends on how large the alternative's effects are supposed to be, and no amount of data removes that dependence. A p-value of 0.044 from 366,454 deaths is at most 2.3 to 1 evidence against no change under any centered normal prior, and 4 to 1 evidence for no change under a prior with a standard deviation of about 10% for the change.

The alternative that Gelman et al. prefer is to estimate how different the groups are instead of testing whether they differ. That is the subject of Hierarchical Models and Shrinkage, which puts a common distribution on many group parameters, estimates its spread from the data, and proves that the resulting shrinkage beats separate estimates.

References