Variational Inference

Approximating a posterior distribution by the closest member of a simpler family, found by optimization instead of sampling. The evidence lower bound derived twice, from Jensen's inequality and as the log evidence minus a Kullback–Leibler divergence; the optimal factor of a mean-field approximation derived and coordinate ascent proved never to lower the bound; the normal model with unknown mean and precision solved in closed form, with the exact divergence from the posterior, on 25 NHANES blood pressures; why the divergence in this direction seeks a mode and shrinks spreads, proved for correlated normals and shown on a two-mode target and on the normal (Laplace) approximation to the logistic regression posterior of the unit. Worked problems on a beta posterior where the bound is tight, a correlated pair, and coordinate ascent on a bivariate normal.

Twenty-five adults drawn from the National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018 (National Center for Health Statistics, 2020) have a mean systolic blood pressure of 128.69 mm Hg. With a normal model whose mean and variance are both unknown, and a conjugate prior, Priors, Posteriors and Conjugate Families writes down the posterior distribution exactly. Most models allow nothing of the kind. For a logistic regression of diabetes on age and obesity in 5,107 adults, Markov Chain Monte Carlo computed the posterior of just three coefficients on a grid of 117,649 points, each needing a pass over all 5,107 adults. The usual alternative is a sampler that runs for thousands of iterations. Can the posterior instead be replaced by the closest member of a simple family of distributions, found by optimization, and how much is lost when it is? It can, the loss can be measured exactly where the posterior is known, and the measurement shows a systematic error: the approximation is too narrow.

Variational inference chooses, within a family of distributions $q$ that are easy to work with, the one that minimizes the Kullback–Leibler divergence $D(q \,\|\, \pi(\cdot \mid \mathbf{x}))$ from $q$ to the posterior. The divergence itself cannot be computed, because it involves the marginal likelihood that made the posterior hard. The method maximizes instead a quantity called the evidence lower bound, which differs from minus the divergence by a constant. The most common family is the mean-field family, in which groups of unknowns are independent.

First we derive the bound, from Jensen's inequality and again as the log evidence minus the divergence, and relate it to the EM algorithm. Then we find the best factor of a mean-field approximation when the others are held fixed, which gives the coordinate ascent algorithm, and prove that it never lowers the bound. Next we solve the normal model with unknown mean and precision on the 25 adults, where the fixed point has a closed form and the divergence from the exact posterior can be computed, and compare the result with a Gibbs sampler. Finally we show why the divergence in this direction makes approximations seek a single mode and understate spreads, on a two-mode target and on the normal (Laplace) approximation to the logistic regression posterior. The article assumes Priors, Posteriors and Conjugate Families and Cross-Entropy and Kullback–Leibler Divergence.

Notation

The unit's notation is kept. The data are $\mathbf{x}$, the unknowns $\theta$ (a vector, which may include latent variables as well as parameters), the model $f(\mathbf{x} \mid \theta)$, the prior density $\pi(\theta)$, the marginal likelihood or evidence $m(\mathbf{x}) = \int f(\mathbf{x} \mid \theta)\pi(\theta)\,d\theta$, and the posterior $\pi(\theta \mid \mathbf{x}) = f(\mathbf{x} \mid \theta)\pi(\theta)/m(\mathbf{x})$. An approximating density is $q(\theta)$, and $E_q$ is an expectation with $\theta$ drawn from $q$. For two densities the divergence is

$$D(q \,\|\, p) = \int q(\theta)\ln\frac{q(\theta)}{p(\theta)}\,d\theta,$$

in nats, with the conventions of the Information Theory unit ($0\ln 0 = 0$, and the divergence is infinite if $q$ puts probability where $p$ has none). Differential Entropy and the Gaussian proves Gibbs' inequality for densities: $D(q \,\|\, p) \ge 0$, with equality exactly when $q = p$ (at every point where both are continuous). Sums replace integrals for discrete unknowns, and every statement below holds with them. The gamma distribution has shape $a$ and rate $b$: density $b^a\tau^{a-1}e^{-b\tau}/\Gamma(a)$, mean $a/b$, variance $a/b^2$. The digamma function $\psi$ is the derivative of $\ln\Gamma$.

The evidence lower bound

Here is the idea in plain words. The logarithm of the evidence is the logarithm of an average, and an average of logarithms is smaller. Writing the evidence as an average under any $q$ we like and moving the logarithm inside gives a lower bound that can be computed whenever $q$ can be sampled or integrated against. The amount by which the bound falls short is exactly the divergence from $q$ to the posterior.

Theorem 1 (the evidence lower bound). For any density $q$ on the parameter space with $E_q\lvert\ln f(\mathbf{x} \mid \theta) + \ln\pi(\theta) - \ln q(\theta)\rvert < \infty$, define

$$\mathcal{L}(q) = E_q\big[\ln f(\mathbf{x} \mid \theta) + \ln\pi(\theta) - \ln q(\theta)\big].$$

Then

$$\ln m(\mathbf{x}) = \mathcal{L}(q) + D\big(q \,\|\, \pi(\cdot \mid \mathbf{x})\big), \qquad (1)$$

so $\mathcal{L}(q) \le \ln m(\mathbf{x})$, with equality exactly when $q$ is the posterior. If also $E_q\lvert\ln f(\mathbf{x} \mid \theta)\rvert < \infty$ (which can fail when the likelihood is unbounded in $\theta$, as for mixtures), then

$$\mathcal{L}(q) = E_q\big[\ln f(\mathbf{x} \mid \theta)\big] - D(q \,\|\, \pi). \qquad (2)$$

First proof (Jensen's inequality). Let $A$ be the set where $q > 0$. Then

$$\ln m(\mathbf{x}) \ge \ln\int_A f(\mathbf{x} \mid \theta)\pi(\theta)\,d\theta = \ln E_q\Big[\frac{f(\mathbf{x} \mid \theta)\pi(\theta)}{q(\theta)}\Big] \ge E_q\Big[\ln\frac{f(\mathbf{x} \mid \theta)\pi(\theta)}{q(\theta)}\Big] = \mathcal{L}(q).$$

The first step drops the part of the integral outside $A$. The second writes the integral over $A$ as an expectation under $q$. The third is Jensen's inequality for the concave logarithm (Probability Inequalities, section "Jensen's inequality"). This proves the bound but not the identity.

Second proof (the identity). On $A$, Bayes' theorem gives $f(\mathbf{x} \mid \theta)\pi(\theta) = m(\mathbf{x})\,\pi(\theta \mid \mathbf{x})$. Take logarithms, subtract $\ln q(\theta)$ and take the expectation under $q$:

$$\mathcal{L}(q) = \ln m(\mathbf{x}) + E_q\Big[\ln\frac{\pi(\theta \mid \mathbf{x})}{q(\theta)}\Big] = \ln m(\mathbf{x}) - D\big(q \,\|\, \pi(\cdot \mid \mathbf{x})\big).$$

The assumption makes $\mathcal{L}(q)$ finite, so the divergence is finite too and the rearrangement is (1). By Gibbs' inequality the divergence is nonnegative and zero exactly when $q$ equals the posterior. For (2), split $\ln[f\pi/q] = \ln f + \ln[\pi/q]$. When $E_q\lvert\ln f(\mathbf{x} \mid \theta)\rvert < \infty$, the first part has a finite expectation, so the second does too, and its expectation is $E_q\ln[\pi/q] = -D(q \,\|\, \pi)$. $\square$

Three readings follow. By (1), maximizing $\mathcal{L}$ over a family is the same as minimizing the divergence from $q$ to the posterior over that family, because $\ln m(\mathbf{x})$ does not depend on $q$. The bound needs only the unnormalized posterior $f(\mathbf{x} \mid \theta)\pi(\theta)$, which can always be evaluated. And by (2) the best $q$ balances two pulls: toward values of $\theta$ that explain the data, and toward the prior. Blei, Kucukelbir and McAuliffe (2017, §2.2) give (1) as their equation (14) and (2) in the same section, and write that in the original literature the bound "was derived through Jensen's inequality", citing Jordan, Ghahramani, Jaakkola and Saul (1999).

Example (the bound when the family contains the posterior). If the family is rich enough to contain the posterior, its maximum is the posterior and the bound equals $\ln m(\mathbf{x})$. Worked problem 1 checks this for the hypertension sample of the unit, 10 of 40 young adults with a uniform prior: the evidence is $1/41$, and the beta density $\text{Beta}(11, 31)$ attains $\mathcal{L} = \ln(1/41)$. Variational inference only approximates when the family is restricted, and all of its error is the divergence in (1).

The same identity in the EM algorithm

The EM Algorithm splits a log-likelihood $\ell(\theta)$ with missing data $\mathbf{z}$ into an expected complete-data part and a conditional part, and proves that the second can only help. Read with (1), its decomposition is the statement that, for any distribution $q$ of the missing data, $\ell(\theta) = \mathcal{L}(q, \theta) + D(q \,\|\, k(\cdot \mid \mathbf{x}; \theta))$, where $k$ is the conditional distribution of the missing data and $\mathcal{L}(q, \theta)$ is the bound with the parameter held fixed. The E step sets $q$ to that conditional distribution, which makes the divergence zero and the bound touch $\ell$. The M step maximizes the bound over $\theta$ with $q$ fixed. EM is coordinate ascent on the evidence lower bound, with the first coordinate maximized exactly. Variational inference differs in two ways. All unknowns, parameters included, are given distributions. And the first step is restricted to a family, so the divergence after it is not zero. When that restriction is applied inside EM, with parameters held as point estimates, the method is called variational EM; Variational Inference for LDA derives it for latent Dirichlet allocation, and this article does not repeat that derivation.

Mean-field families and coordinate ascent

Which family? The mean-field family splits $\theta$ into blocks $\theta_1, \ldots, \theta_J$ and takes them to be independent:

$$q(\theta) = \prod_{j=1}^{J} q_j(\theta_j).$$

Nothing else is assumed about the factors $q_j$. Here is the idea of the next theorem. If all factors but one are held fixed, the bound as a function of the remaining factor is, apart from a constant, minus the divergence from that factor to a particular density. That density is the best choice, and it can be written down.

Theorem 2 (the optimal factor). Fix $q_k$ for every $k \ne j$, and write $E_{-j}$ for the expectation over $\theta_k$, $k \ne j$, each drawn independently from its $q_k$. Suppose the function

$$g_j(\theta_j) = E_{-j}\big[\ln f(\mathbf{x} \mid \theta) + \ln\pi(\theta)\big]$$

is finite and $Z_j = \int e^{g_j(\theta_j)}\,d\theta_j$ is finite. Then among all densities $q_j$ (with the bound finite), $\mathcal{L}$ is maximized by

$$q_j^*(\theta_j) = \frac{\exp g_j(\theta_j)}{Z_j}, \qquad \text{that is,} \qquad \ln q_j^*(\theta_j) = E_{-j}\big[\ln f(\mathbf{x} \mid \theta) + \ln\pi(\theta)\big] + \text{const}, \qquad (3)$$

and by no other density.

Proof. Under the product $q$, the entropy term splits by independence: $E_q[\ln q(\theta)] = \sum_k E_{q_k}[\ln q_k(\theta_k)]$. In the first term, take the expectation over the other blocks first (the expectation of a function of independent blocks can be taken one block at a time): $E_q[\ln f + \ln\pi] = E_{q_j}[g_j(\theta_j)]$. So

$$\mathcal{L}(q) = E_{q_j}\big[g_j(\theta_j) - \ln q_j(\theta_j)\big] - \sum_{k \ne j}E_{q_k}[\ln q_k] = -D(q_j \,\|\, q_j^*) + \ln Z_j - \sum_{k \ne j}E_{q_k}[\ln q_k],$$

because $g_j = \ln q_j^* + \ln Z_j$. Only the first term depends on $q_j$, and by Gibbs' inequality it is largest, at zero, exactly when $q_j = q_j^*$. $\square$

In words: the log of the best factor is the log of the joint density of data and unknowns, averaged over the other factors. The constant is fixed by normalizing. Bishop (2006, §10.1.1) obtains (3) as his equation (10.9) by the same rearrangement, and Blei, Kucukelbir and McAuliffe (2017, §2.4) as their equation (18). Since $\ln\pi(\theta \mid \mathbf{x})$ differs from $\ln f + \ln\pi$ by a constant, (3) can be written with the posterior, or with the full conditional of $\theta_j$, in place of the joint density. In the second form the update resembles the Gibbs sampler of Markov Chain Monte Carlo, which draws $\theta_j$ from its full conditional; here the logarithm of that conditional is averaged instead. Blei, Kucukelbir and McAuliffe make the same comparison.

Coordinate ascent. Start with any factors, then replace each in turn by its optimal factor (3), and repeat. This is coordinate ascent variational inference (Blei, Kucukelbir and McAuliffe, 2017, Algorithm 1).

Corollary (the bound never falls). Each update of coordinate ascent leaves $\mathcal{L}$ at least as large as before, and the values of $\mathcal{L}$ converge.

Proof. Each update replaces $q_j$ by the maximizer of $\mathcal{L}$ over $q_j$ with the others fixed (Theorem 2), so the value cannot decrease. The values form a nondecreasing sequence bounded above by $\ln m(\mathbf{x})$ (Theorem 1), so they converge. $\square$

What is not guaranteed is the same as for EM. The factors need not converge, and a limit need not be the best member of the family. Blei, Kucukelbir and McAuliffe (2017, §2.5) state that the method "only guarantees convergence to a local optimum, which can be sensitive to initialization", and show ten starts reaching ten different values of the bound for a mixture model. The two-mode example below has three local optima.

Coordinate ascent on the blood pressures

Back to the 25 adults. The model makes the pressures independent and normal with mean $\mu$ and variance $\sigma^2$, and it is convenient to work with the precision $\tau = 1/\sigma^2$. The prior belongs to the normal–inverse-gamma family of Priors, Posteriors and Conjugate Families, whose Theorem 7 gives the posterior; its values are our choice: $\mu_0 = 120$ mm Hg (the round value of that article's known-variance example), $\kappa_0 = 1$ (the prior is worth one observation about the mean), $\nu_0 = 4$ and $\sigma_0^2 = 400$ (worth four observations about the variance, at a standard deviation of 20). In terms of $\tau$: $\mu$ given $\tau$ is normal with mean $\mu_0$ and variance $1/(\kappa_0\tau)$, and $\tau$ has the gamma distribution with shape $a_0 = \nu_0/2 = 2$ and rate $b_0 = \nu_0\sigma_0^2/2 = 800$, the reciprocal of the inverse gamma.

The exact posterior. Theorem 7 of that article gives it. With $n = 25$, $\bar x = 128.69$ and sum of squares $S = \sum_i(x_i - \bar x)^2 = 8{,}763.1$ (sample standard deviation 19.11):

$$\kappa_n = 26, \quad \mu_n = \frac{\kappa_0\mu_0 + n\bar x}{\kappa_n} = 128.36, \quad a_n = a_0 + \frac n2 = 14.5, \quad b_n = b_0 + \frac12\Big[S + \frac{\kappa_0 n}{\kappa_n}(\bar x - \mu_0)^2\Big] = 5{,}217.9,$$

where $a_n = \nu_n/2$ and $b_n = \nu_n\sigma_n^2/2$ in that article's letters. The posterior of $\tau$ is gamma with shape $a_n$ and rate $b_n$, so $E[\tau \mid \mathbf{x}] = a_n/b_n$, which is 2.779 per thousand. Given $\tau$, $\mu$ is normal with mean $\mu_n$ and variance $1/(\kappa_n\tau)$. Marginally $\mu$ is Student $t$ with $2a_n = 29$ degrees of freedom, location $\mu_n$ and scale $\sqrt{b_n/(a_n\kappa_n)}$. Its variance is the squared scale times $29/27$ (Chi-Squared and t Distributions proves that a $t$ variable with $\nu$ degrees of freedom has variance $\nu/(\nu - 2)$), which is

$$\operatorname{Var}(\mu \mid \mathbf{x}) = \frac{b_n}{\kappa_n(a_n - 1)},$$

a standard deviation of 3.86. The central 95% posterior interval for $\mu$ is (120.75, 135.97). The log evidence, in closed form from the same conjugacy, is $-111.49$.

The posterior does not factor. The spread of $\mu$ given $\tau$ depends on $\tau$: if the variance of the pressures is large, the mean is less certain. A mean-field approximation $q(\mu, \tau) = q_\mu(\mu)\,q_\tau(\tau)$ cannot represent that, and we can now measure what it costs.

The two updates

Two identities do all the work. Completing the square in $\mu$, which follows from $\sum_i(x_i - \mu)^2 = S + n(\bar x - \mu)^2$ and identity (4) of the conjugate-priors article with $u = \kappa_0$ and $v = n$,

$$\sum_i(x_i - \mu)^2 + \kappa_0(\mu - \mu_0)^2 = \kappa_n(\mu - \mu_n)^2 + 2(b_n - b_0). \qquad (4)$$

And for $\tau$ gamma with shape $a$ and rate $b$, $E[\ln\tau] = \psi(a) - \ln b$. To see it, differentiate $\int_0^\infty \tau^{a-1}e^{-b\tau}\,d\tau = \Gamma(a)b^{-a}$ with respect to $a$, which brings down $\ln\tau$ on the left and gives $\Gamma(a)b^{-a}[\psi(a) - \ln b]$ on the right; dividing by $\Gamma(a)b^{-a}$ turns the left side into the expectation. The step of differentiating under the integral sign is covered by Exponential Families, Theorem 2, which gives the mean of a sufficient statistic as a derivative of the log-partition function: the gamma family has sufficient statistic $\ln\tau$ with natural parameter $a - 1$, and log-partition function $\ln\Gamma(a) - a\ln b$.

The log joint density of data and unknowns is, up to constants that involve neither $\mu$ nor $\tau$,

$$\ln f(\mathbf{x} \mid \mu, \tau) + \ln\pi(\mu \mid \tau) + \ln\pi(\tau) = \Big(a_0 + \frac{n + 1}{2} - 1\Big)\ln\tau - b_0\tau - \frac\tau2\Big[\sum_i(x_i - \mu)^2 + \kappa_0(\mu - \mu_0)^2\Big].$$

The power of $\tau$ collects $n/2$ from the likelihood, $1/2$ from the normal prior of $\mu$ given $\tau$, and $a_0 - 1$ from the gamma prior.

Update of $q_\mu$. By (3), average over $\tau$ with $q_\tau$ fixed, and write $e = E_{q_\tau}[\tau]$. Only the last term involves $\mu$, and by (4) it averages to $-\frac{e}{2}\kappa_n(\mu - \mu_n)^2$ plus a constant. So $\ln q_\mu^*(\mu)$ is a quadratic in $\mu$, and $q_\mu^*$ is normal with mean $\mu_n$ and precision $\lambda = \kappa_n e$.

Update of $q_\tau$. Average over $\mu$ with $q_\mu$ normal, mean $\mu_n$ and precision $\lambda$. By (4) the bracket averages to $\kappa_n/\lambda + 2(b_n - b_0)$. So

$$\ln q_\tau^*(\tau) = \Big(a_n + \tfrac12 - 1\Big)\ln\tau - \Big(b_n + \frac{\kappa_n}{2\lambda}\Big)\tau + \text{const},$$

the gamma density with shape $\alpha = a_n + \tfrac12$ and rate $\beta = b_n + \kappa_n/(2\lambda)$.

Neither form was assumed. A normal factor for $\mu$ and a gamma factor for $\tau$ are what (3) produces. The same updates appear in Bishop (2006, §10.1.3) for this model, with his parameters in place of ours.

Theorem 3 (the fixed point in closed form). For the normal model with the normal–inverse-gamma prior, coordinate ascent from any starting $q_\tau$ with $E\lvert\tau\rvert$ and $E\lvert\ln\tau\rvert$ finite (every gamma density qualifies) converges to the mean-field approximation

$$q_\mu^* = \text{normal with mean } \mu_n \text{ and precision } \lambda^* = \frac{\kappa_n a_n}{b_n}, \qquad q_\tau^* = \text{gamma with shape } \alpha = a_n + \tfrac12 \text{ and rate } \beta^* = \frac{b_n\alpha}{a_n},$$

and the error in the rate $\beta$ of $q_\tau$ is multiplied by $1/(2\alpha)$ at every round after the first. This limit maximizes $\mathcal{L}$ over all product densities $q_\mu(\mu)q_\tau(\tau)$ whose factor $q_\tau$ has $E\lvert\tau\rvert$ and $E\lvert\ln\tau\rvert$ finite. At the fixed point:

  1. $E_{q}[\mu] = \mu_n$ and $E_{q}[\tau] = a_n/b_n$: both posterior means are exact.
  2. $\operatorname{Var}_q(\mu)/\operatorname{Var}(\mu \mid \mathbf{x}) = (a_n - 1)/a_n$ and $\operatorname{Var}_q(\tau)/\operatorname{Var}(\tau \mid \mathbf{x}) = a_n/(a_n + \tfrac12)$: both spreads are too small.
  3. The divergence from the approximation to the posterior depends on the data only through $a_n$: $$D\big(q^* \,\|\, \pi(\cdot \mid \mathbf{x})\big) = \ln\Gamma(a_n) - \ln\Gamma(a_n + \tfrac12) + (a_n + \tfrac12)\ln(a_n + \tfrac12) - a_n\ln a_n - \tfrac12. \qquad (5)$$

Proof. One round updates $q_\mu$ and then $q_\tau$. If $q_\tau$ has rate $\beta$ before the round, then $e = \alpha/\beta$ (the shape is $\alpha$ after the first round, whatever it was at the start), $\lambda = \kappa_n\alpha/\beta$, and the new rate is

$$\beta' = b_n + \frac{\kappa_n}{2\lambda} = b_n + \frac{\beta}{2\alpha}.$$

This map has the single fixed point $\beta^* = b_n/(1 - 1/(2\alpha)) = 2\alpha b_n/(2\alpha - 1) = \alpha b_n/a_n$, since $2\alpha - 1 = 2a_n$, and $\beta' - \beta^* = (\beta - \beta^*)/(2\alpha)$. The factor is below 1, so $\beta$ converges to $\beta^*$ geometrically, and $\lambda^* = \kappa_n\alpha/\beta^* = \kappa_n a_n/b_n$. (Before the first round $q_\tau$ need not be a gamma density, so the contraction applies from the first round's output on.)

Global maximum. Let $q$ be any product density whose factor $q_\tau$ has $E\lvert\tau\rvert$ and $E\lvert\ln\tau\rvert$ finite (so that the function $g_\mu$ of Theorem 2 is finite), with $\mathcal{L}(q)$ finite, and run coordinate ascent from it. By the corollary $\mathcal{L}$ never falls. After the $\tau$-update of round $k$ the iterate is a normal with mean $\mu_n$ and precision $\lambda_k$ times a gamma with shape $\alpha$ and rate $\beta_k$, with $\lambda_k = \kappa_n\alpha/\beta_{k-1}$ from round 2 on; so $\beta_k \to \beta^*$ and $\lambda_k \to \lambda^*$. On that family $\mathcal{L}$ is a continuous function of $(\lambda, \beta)$ on $(0, \infty)^2$, as its expression term by term shows. So $\mathcal{L}(q) \le \lim_k\mathcal{L} = \mathcal{L}(q^*)$. Hence $q^*$ is a global maximizer, and it is the only one: a maximizer cannot be improved by either update, so by the uniqueness in Theorem 2 it equals its own updates, and the only such pair is the fixed point found above.

(1) $q_\mu^*$ has mean $\mu_n$, the posterior mean. $E_q[\tau] = \alpha/\beta^* = a_n/b_n = E[\tau \mid \mathbf{x}]$.

(2) $\operatorname{Var}_q(\mu) = 1/\lambda^* = b_n/(\kappa_n a_n)$, against $b_n/(\kappa_n(a_n - 1))$. $\operatorname{Var}_q(\tau) = \alpha/\beta^{*2} = a_n^2/(\alpha b_n^2)$, against $a_n/b_n^2$; the ratio is $a_n/\alpha$.

(3) Write the posterior as $\pi(\tau \mid \mathbf{x})\,\pi(\mu \mid \tau, \mathbf{x})$. Then $\ln[q^*/\pi(\cdot \mid \mathbf{x})] = \ln[q_\tau^*/\pi(\tau \mid \mathbf{x})] + \ln[q_\mu^*/\pi(\mu \mid \tau, \mathbf{x})]$, and taking expectations under $q^*$,

$$D = D\big(q_\tau^* \,\|\, \pi(\tau \mid \mathbf{x})\big) + E_{q_\tau^*}\Big[D\big(q_\mu^* \,\|\, \pi(\mu \mid \tau, \mathbf{x})\big)\Big].$$

The second term compares two normal densities with the same mean and precisions $\lambda^*$ and $\kappa_n\tau$. The divergence between such densities is $\tfrac12(r - 1 - \ln r)$ with $r = \kappa_n\tau/\lambda^*$ the ratio of the second precision to the first (a special case of Differential Entropy and the Gaussian, section "Two normal densities"). Its average over $q_\tau^*$, using $E[\tau] = \alpha/\beta^*$ (so $E[r] = 1$) and $E[\ln\tau] = \psi(\alpha) - \ln\beta^*$, is

$$\tfrac12\big[\ln\lambda^* - \ln\kappa_n - \psi(\alpha) + \ln\beta^*\big] = \tfrac12\big[\ln\alpha - \psi(\alpha)\big].$$

For the first term, the divergence between gamma densities with shapes and rates $(\alpha, \beta)$ and $(a, b)$ is, from the two log densities and the two expectations just used,

$$\big[\alpha\ln\beta - \ln\Gamma(\alpha) + (\alpha - 1)E\ln\tau - \beta E\tau\big] - \big[a\ln b - \ln\Gamma(a) + (a - 1)E\ln\tau - bE\tau\big] = (\alpha - a)\psi(\alpha) - \ln\Gamma(\alpha) + \ln\Gamma(a) + a\ln\frac\beta b - \alpha + \frac{b\alpha}{\beta}.$$

With $a = a_n$, $b = b_n$, $\alpha - a_n = \tfrac12$, $\beta^*/b_n = \alpha/a_n$ and $b_n\alpha/\beta^* = a_n$, this is $\tfrac12\psi(\alpha) - \ln\Gamma(\alpha) + \ln\Gamma(a_n) + a_n\ln(\alpha/a_n) - \tfrac12$. Adding the two terms, the digamma terms cancel and $\tfrac12\ln\alpha + a_n\ln\alpha = \alpha\ln\alpha$, which gives (5). $\square$

The script checks (5) against $\ln m(\mathbf{x}) - \mathcal{L}(q^*)$, with the bound computed term by term from its definition, and against a numerical double integral of the divergence. All three agree to $10^{-7}$.

The trace. We started coordinate ascent from a deliberately poor $q_\tau$, gamma with shape $\alpha = 15$ and $E_q[\tau] = 1$, a standard deviation of about 1 mm Hg for the pressures. The table shows each round: the standard deviation of $q_\mu$ after its update, $E_q[\tau]$ after the update of $q_\tau$, and the bound after each of the two updates.

Round Standard deviation of $q_\mu$ $E_q[\tau]$ (per thousand) $\mathcal{L}$ after $q_\mu$ $\mathcal{L}$ after $q_\tau$
1 0.196 2.8745 −5,229.54 −113.94
2 3.658 2.7820 −111.51 −111.51
3 3.718 2.7790 −111.51 −111.51
4 3.720 2.7789 −111.51 −111.51

The bound starts near −5,230 because a $q_\mu$ with standard deviation 0.196 is absurdly confident. It rose at every update, as the corollary requires, and the script asserts that it did. With $\alpha = 15$ the rate of convergence is $1/30 = 0.033$ per round, so each round gains about one and a half decimal digits, and after four rounds the parameters agree with the closed form to three decimals. The limit has $\beta^* = 5{,}397.8$ and $\lambda^* = 72.25$ per thousand, a standard deviation of 3.72 for $q_\mu$.

The exact posterior of the mean μ and standard deviation σ for the 25 adults (solid) and the mean-field approximation at the fixed point of coordinate ascent (dashed), each drawn at the contours that enclose 50% and 90% of its probability. Notice that the exact contours are wider at the top: when σ is large the mean is less certain, so the posterior leans like a pear. The product q(μ)q(τ) cannot lean, and its contours are narrower in μ than the exact ones.
The exact posterior of the mean μ and standard deviation σ for the 25 adults (solid) and the mean-field approximation at the fixed point of coordinate ascent (dashed), each drawn at the contours that enclose 50% and 90% of its probability. Notice that the exact contours are wider at the top: when σ is large the mean is less certain, so the posterior leans like a pear. The product q(μ)q(τ) cannot lean, and its contours are narrower in μ than the exact ones.

How good is the approximation?

Theorem 3 answers it in numbers for the 25 adults, with $a_n = 14.5$:

  • Means. Exact: 128.36 for $\mu$ and 2.779 per thousand for $\tau$.
  • Spread of the mean. The approximation's standard deviation is 3.72 against the exact 3.86, a ratio of $\sqrt{13.5/14.5} = 0.9649$. Its central 95% interval, $128.36 \pm 1.96 \times 3.72$, is (121.07, 135.65), against the exact (120.75, 135.97). The exact posterior probability of the approximation's interval is 0.940, not 0.95.
  • Spread of the standard deviation. The standard deviation of $q_\tau$ is $\sqrt{14.5/15} = 0.9832$ of the exact one. Translated into $\sigma = \tau^{-1/2}$, the central 95% interval is (15.16, 25.36) under $q$ and (15.11, 25.50) exactly.
  • Divergence. By (5), $D(q^* \,\|\, \pi(\cdot \mid \mathbf{x})) = 0.0171$ nats, of which 0.0169 is the mean's term $\tfrac12[\ln\alpha - \psi(\alpha)]$ and 0.0003 the precision's. The bound at the fixed point is $-111.51$, and the log evidence $-111.49$.

Example (a closer fit is not a better spread). The approximation is accurate in its means and still wrong in a systematic direction: both spreads are too small, for every data set, by the factors in part 2 of Theorem 3. The two errors do not average out over samples, as Monte Carlo error does. They are properties of the family and of the direction of the divergence, and the last two sections explain where they come from.

The divergence depends on the data only through $a_n = a_0 + n/2$, so it can be tabulated against the sample size alone, for this prior:

$n$ $a_n$ $D(q^* \,\|\, \text{posterior})$ (nats) Standard deviation ratio for $\mu$
5 4.5 0.0545 0.882
25 14.5 0.0171 0.965
100 52 0.0048 0.990
400 202 0.0012 0.998

The divergence falls roughly like $1/n$: it is close to $1/(4\alpha)$, which is 0.0167 at $n = 25$. As $n$ grows the posterior of $\tau$ concentrates, the dependence of $\mu$'s spread on $\tau$ stops mattering, and the posterior comes to factor approximately. With five observations the mean-field interval for $\mu$ is 12% too narrow in standard deviation.

The same posterior by sampling

Markov Chain Monte Carlo samples this posterior with a two-block Gibbs sampler. Over 30 runs (seeds 1 to 30), each with four chains of 2,000 iterations of which the second 1,000 are kept, the sampled standard deviation of $\mu$ had median 3.86 (interquartile range 3.81 to 3.89), between 3.75 and 3.95 in all runs. None of the 30 runs came as low as the variational 3.72. The two methods err differently. The sampler's error is random, centered on the exact value, and shrinks as the run lengthens. The variational error is deterministic, and no amount of computation removes it within the family; the coordinate ascent took four rounds of arithmetic.

Which way the divergence points

Variational inference minimizes $D(q \,\|\, p)$, with the approximation first, where $p$ is the posterior. Cross-Entropy and Kullback–Leibler Divergence (section "The two directions") shows on a crude family of character distributions that the two directions of the divergence pick different answers. Here is the idea for continuous targets. In $D(q \,\|\, p) = E_q[\ln q - \ln p]$, the average is over $q$. Wherever $q$ puts probability and $p$ has very little, $-\ln p$ is huge, so the minimizer avoids such regions even at the cost of ignoring part of $p$. In $D(p \,\|\, q)$ the average is over $p$, and wherever $p$ has probability that $q$ lacks, $-\ln q$ is huge, so the minimizer covers all of $p$, even at the cost of putting probability where $p$ has none. Bishop (2006, §10.1.2) calls the first behavior zero forcing, and describes the second as the approximation being "nonzero in regions where p(Z) is nonzero".

Theorem 4 (the best normal in the forward direction). Among normal densities $q$, $D(p \,\|\, q)$ is minimized by the normal with the mean and variance of $p$ (when $p$ has a finite variance).

Proof. $D(p \,\|\, q) = E_p[\ln p] - E_p[\ln q]$, and for $q$ normal with mean $m$ and variance $s^2$, $-E_p[\ln q] = \tfrac12\ln(2\pi s^2) + E_p[(\theta - m)^2]/(2s^2)$. Write $E_p[(\theta - m)^2] = v + (\mu_p - m)^2$, with $\mu_p$ and $v$ the mean and variance of $p$. The minimum over $m$ is at $m = \mu_p$, and then $\tfrac12\ln s^2 + v/(2s^2)$ has derivative $1/(2s^2)\,(1 - v/s^2)$ in $s^2$, negative below $v$ and positive above, so the minimum is at $s^2 = v$. $\square$

No such formula exists in the reverse direction, which has to be minimized numerically.

Example (a target with two modes). Take $p$ to be an equal mixture of two normal densities with variance 1 and means $-3$ and $3$. This is the shape of a posterior that cannot tell two labels apart, such as the difference between the two means of a two-component mixture, where swapping the labels gives the same likelihood (Gaussian Mixture Models treats this label switching). Its density is 0.199 at each mode and 0.0044 at 0.

  • Forward. By Theorem 4 the best normal has mean 0 and variance $1 + 9 = 10$, a standard deviation of 3.16. It covers both modes and puts its peak where $p$ is almost empty: at 0 its density is 28.5 times that of $p$. Its divergence $D(p \,\|\, q)$ is 0.462.
  • Reverse. Minimizing $D(q \,\|\, p)$ numerically over the mean and standard deviation of $q$, from starts at means $-4$, $-1$, $1$ and $4$, ends at one mode or the other: mean $\pm 2.984$, standard deviation 1.023, divergence 0.689. The approximation describes one mode and ignores the other. The probability that $p$ gives to its central 95% interval is 0.478, about half. There is a third local minimum: with the mean held at 0 by symmetry, the best standard deviation is 2.74, with divergence 0.841, and moving the mean a little off 0 raises the divergence, so a run started exactly at 0 stays there at a worse value.

The value 0.689 is close to $\ln 2 = 0.693$ for a reason. A normal density $\varphi_3$ placed exactly on the right mode has $D(\varphi_3 \,\|\, p) = E[\ln\varphi_3 - \ln(\tfrac12\varphi_3 + \tfrac12\varphi_{-3})] = \ln 2 - E[\ln(1 + \varphi_{-3}/\varphi_3)]$, with the expectation under $\varphi_3$; the last term is small and positive when the modes are far apart, so describing one mode of two costs about $\ln 2$.

Each fit is poor by the other's measure. The forward fit has reverse divergence 0.938, worse than either local minimum of the reverse problem. The reverse fit has forward divergence 7.86, because it gives the left mode almost no probability.

An equal mixture of two unit normals at −3 and 3 (solid), the normal that minimizes the divergence D(q || p) from a start on the right (dashed), and the normal that minimizes D(p || q) (dotted). Notice that the first sits on one mode and ignores the other, while the second spreads over both and is highest at 0, where the target has almost no probability.
An equal mixture of two unit normals at −3 and 3 (solid), the normal that minimizes the divergence D(q || p) from a start on the right (dashed), and the normal that minimizes D(p || q) (dotted). Notice that the first sits on one mode and ignores the other, while the second spreads over both and is highest at 0, where the target has almost no probability.

This is the behavior Bishop (2006, §10.1.2) illustrates with a mixture of two normals in two dimensions (his Figure 10.3), and the reason he gives why the forward direction does badly for mixture posteriors: a forward fit would average across modes, "and the average of two good parameter values is typically itself not a good parameter value".

A correlated posterior

Mean-field approximations understate spread even when the target has a single mode, if the unknowns are correlated. The next theorem is exact for normal targets.

Theorem 5 (mean-field approximation of a normal). Let $p$ be the $d$-dimensional normal density with mean $\boldsymbol\mu$, covariance $\Sigma$ and precision matrix $\Lambda = \Sigma^{-1}$. Among all product densities $q(\theta) = \prod_j q_j(\theta_j)$ with finite variances:

  1. $D(q \,\|\, p)$ is minimized uniquely by $q_j$ normal with mean $\mu_j$ and variance $1/\Lambda_{jj}$, and the minimum is $\tfrac12\ln\big(\prod_j\Lambda_{jj}\,\det\Sigma\big)$.
  2. $D(p \,\|\, q)$ is minimized uniquely by $q_j$ the marginal of $p$, normal with mean $\mu_j$ and variance $\Sigma_{jj}$, and the minimum is $-\tfrac12\ln\det R$, where $R$ is the correlation matrix of $p$.
  3. $1/\Lambda_{jj} \le \Sigma_{jj}$, with equality exactly when $\theta_j$ is uncorrelated with the other coordinates under $p$.

Proof. (1) Let $q$ have means $m_j$ and variances $v_j$, and let $h(q_j) = -E[\ln q_j]$ be the differential entropy of a factor. By independence the entropy of $q$ is $\sum_j h(q_j)$. Since $-\ln p(\theta) = \tfrac12(\theta - \boldsymbol\mu)^{\mathsf T}\Lambda(\theta - \boldsymbol\mu) + \tfrac12\ln\det(2\pi\Sigma)$, and under $q$ the coordinates are independent,

$$D(q \,\|\, p) = -\sum_j h(q_j) + \tfrac12(\mathbf{m} - \boldsymbol\mu)^{\mathsf T}\Lambda(\mathbf{m} - \boldsymbol\mu) + \tfrac12\sum_j\Lambda_{jj}v_j + \tfrac12\ln\det(2\pi\Sigma).$$

The expectation of the quadratic form uses $E[(\theta_j - \mu_j)(\theta_k - \mu_k)] = (m_j - \mu_j)(m_k - \mu_k)$ for $j \ne k$ and $(m_j - \mu_j)^2 + v_j$ for $j = k$. The normal has the largest differential entropy for its variance (Differential Entropy and the Gaussian, section "The Gaussian has the greatest entropy for its variance"): $h(q_j) \le \tfrac12\ln(2\pi e v_j)$, with equality only for the normal. So

$$D(q \,\|\, p) \ge \tfrac12(\mathbf{m} - \boldsymbol\mu)^{\mathsf T}\Lambda(\mathbf{m} - \boldsymbol\mu) + \tfrac12\sum_j\big(\Lambda_{jj}v_j - \ln v_j - \ln(2\pi e)\big) + \tfrac12\ln\det(2\pi\Sigma).$$

The quadratic form is positive unless $\mathbf{m} = \boldsymbol\mu$, since $\Lambda$ is positive definite. Each term $\Lambda_{jj}v_j - \ln v_j$ is minimized uniquely at $v_j = 1/\Lambda_{jj}$, where it is $1 + \ln\Lambda_{jj}$. With normal factors all the inequalities are equalities, and the minimum is $\tfrac12\sum_j(1 + \ln\Lambda_{jj} - \ln(2\pi e)) + \tfrac12 d\ln(2\pi) + \tfrac12\ln\det\Sigma = \tfrac12\ln(\prod_j\Lambda_{jj}\det\Sigma)$.

(2) $D(p \,\|\, q) = E_p[\ln p] - \sum_j E_p[\ln q_j(\theta_j)]$, and the $j$-th expectation involves only the marginal $p_j$ of $\theta_j$: $-E_{p_j}[\ln q_j] = h(p_j) + D(p_j \,\|\, q_j)$. By Gibbs' inequality each term is smallest exactly when $q_j = p_j$. The marginal of a normal is normal with variance $\Sigma_{jj}$, and the minimum is $\sum_j h(p_j) - h(p) = \tfrac12\sum_j\ln(2\pi e\Sigma_{jj}) - \tfrac12\ln((2\pi e)^d\det\Sigma) = -\tfrac12\ln\det R$, since $\det R = \det\Sigma/\prod_j\Sigma_{jj}$.

(3) As a function of $\theta_j$ with the other coordinates fixed, $\ln p$ is $-\tfrac12\Lambda_{jj}(\theta_j - \mu_j)^2 - (\theta_j - \mu_j)\sum_{k \ne j}\Lambda_{jk}(\theta_k - \mu_k)$ plus terms free of $\theta_j$. Completing the square, the conditional distribution of $\theta_j$ given the others is normal with variance $1/\Lambda_{jj}$: the mean-field factor reports the conditional spread. For the comparison with $\Sigma_{jj}$, order the coordinates with $j$ first and write $\Lambda$ in blocks, $\Lambda_{jj}$, the row $\mathbf{b}^{\mathsf T}$ of the entries $\Lambda_{jk}$ for $k \ne j$, and the matrix $M$ of the remaining entries, which is positive definite because $\Lambda$ is. The formula for the inverse of a matrix in blocks gives the $(j, j)$ entry of $\Sigma = \Lambda^{-1}$ as

$$\Sigma_{jj} = \frac{1}{\Lambda_{jj} - \mathbf{b}^{\mathsf T}M^{-1}\mathbf{b}}.$$

(To check it, multiply $\Lambda$ by the vector $(1, -M^{-1}\mathbf{b})$: the result is $(\Lambda_{jj} - \mathbf{b}^{\mathsf T}M^{-1}\mathbf{b})$ times the first unit vector, so the first column of $\Lambda^{-1}$ is that vector divided by $\Lambda_{jj} - \mathbf{b}^{\mathsf T}M^{-1}\mathbf{b}$.) Since $M^{-1}$ is positive definite, $\mathbf{b}^{\mathsf T}M^{-1}\mathbf{b} \ge 0$, with equality exactly when $\mathbf{b} = \mathbf{0}$, so $\Sigma_{jj} \ge 1/\Lambda_{jj}$. When $\mathbf{b} = \mathbf{0}$, $\Lambda$ is block diagonal with $j$ alone in one block, and so is its inverse: $\theta_j$ is uncorrelated with the rest. Conversely, if $\theta_j$ is uncorrelated with the rest, $\Sigma$ is block diagonal in that way, so is $\Lambda$, and $\mathbf{b} = \mathbf{0}$. $\square$

Bishop (2006, §10.1.2) derives part 1 for two variables from the optimal-factor equation (10.9) and part 2 for any target, and notes that the reverse approximation's variance is "controlled by the direction of smallest variance" of $p$. Part 1 says the mean-field approximation reports, for each coordinate, its spread when all the others are known.

Back to the logistic regression. Markov Chain Monte Carlo fits the logistic regression of diabetes on age in years and obesity to the 5,107 adults, with weak normal priors, and shows that the posterior is very close to its Laplace approximation, the normal density at the mode with the inverse of the curvature as covariance (Bayes Factors and Posterior Model Probabilities derives Laplace's method). The intercept and the age coefficient have posterior correlation −0.947: with age measured from 0, a steeper slope must be balanced by a lower intercept. Take that normal as the target and apply Theorem 5:

Coefficient Standard deviation of the target (Laplace normal) Mean-field standard deviation Ratio Probability of the mean-field 95% interval under the target
Intercept 0.1870 0.0416 0.222 0.337
Age (per year) 0.00281 0.00066 0.236 0.356
Obese 0.0840 0.0576 0.686 0.821

The mean-field interval for the age coefficient holds 36% of the target's probability, not 95%. The divergence at the optimum of part 1 is 1.812 nats; the forward product of marginals, which reports the right spreads but ignores the correlation, has divergence 1.513 (the determinant of the correlation matrix is 0.0485). This is a computation on the normal approximation, not a run of variational inference on the logistic posterior itself; since that posterior is nearly normal, a mean-field fit to it would be expected to come close to these numbers, but we did not run one. Two remedies are available: change the coordinates, or change the family. Measuring age from its mean in the sample lowers the correlation of the intercept and the age coefficient to −0.525, and the smallest of the three ratios rises from 0.222 to 0.59. A normal family with a full covariance matrix contains this target exactly.

Blei, Kucukelbir and McAuliffe (2017, §1) summarize the general experience: variational inference "generally underestimates the variance of the posterior density; this is a consequence of its objective function". Theorems 3 and 5 are two cases where the underestimate can be computed exactly.

Where variational inference is used

The method is used where sampling is too slow: large data sets, many latent variables, and models refit often. Variational Inference for LDA is the site's full example, with a factor for each document's topic proportions and one for each word's topic; its two updates are instances of (3), and its trace on a one-word document shows the mean-field family committing to one topic, the mode seeking of the example above. Latent Dirichlet Allocation fits the same model by collapsed Gibbs sampling, the alternative. Blei, Kucukelbir and McAuliffe (2017, §2.4 and §3) give the coordinate ascent algorithm for a mixture of normals and compare it with a sampler on image data, where it was much faster. Stochastic optimization of the bound, which scales the method to data sets that do not fit in memory, is outside this article.

What is proved, what is cited, what the experiment shows

Proved on this page: the evidence lower bound and the identity (1), by two arguments, and the form (2) (Theorem 1); the optimal mean-field factor (3) (Theorem 2) and the monotonicity of coordinate ascent (corollary); the expected logarithm of a gamma variable; the updates, fixed point, rate, exact means, variance ratios and divergence (5) of the mean-field approximation for the normal model with unknown mean and precision (Theorem 3); the best normal in the forward direction (Theorem 4); the mean-field approximations of a normal target in both directions, with their divergences and the variance inequality (Theorem 5); the $\ln 2$ approximation for one of two separated modes. The proofs are written for densities; with sums in place of integrals they hold for discrete unknowns.

Used from other pages: Gibbs' inequality for densities, the maximum entropy of the normal, and the divergence between normal densities (Differential Entropy and the Gaussian); Jensen's inequality (Probability Inequalities); the normal–inverse-gamma posterior and identity (4) (Priors, Posteriors and Conjugate Families); the variance of the $t$ distribution (Chi-Squared and t Distributions); differentiation under the integral sign for the gamma family (Exponential Families); the logistic regression posterior and its Laplace approximation (Markov Chain Monte Carlo).

Cited: the forms of the bound and the coordinate update in Bishop (2006, §10.1) and Blei, Kucukelbir and McAuliffe (2017), and the latter's statements on local optima and on variance; the Jensen derivation of the bound in Jordan et al. (1999, §6, eq. 43), to which Blei, Kucukelbir and McAuliffe attribute it.

Observed in the experiment: the trace of coordinate ascent from a poor start, rising at every update; the agreement of (5) with the bound computed from its definition and with numerical integration; on the 25 adults, the Gibbs sampler's estimates of the posterior standard deviation spread around the exact value, all 30 above the variational one; the three local minima of the reverse fit to the two-mode target, found from four starts. Every number on this page is produced by tools/experiments/statistics/variational-inference.py.

Worked problems

These problems are posed for this article on the unit's data and on small exact cases; the unit has no course assignments. The script recomputes every number.

Problem 1 (a bound that is tight). Ten of 40 young adults have hypertension. With a uniform prior on the proportion $p$, compute the evidence, show that the bound of Theorem 1 attains it at $q = \text{Beta}(11, 31)$, and find the gap for $q = \text{Beta}(10, 30)$ and $q = \text{Beta}(22, 62)$.

Solution. The evidence is $m(x) = \binom{40}{10}\int_0^1 p^{10}(1-p)^{30}\,dp = \binom{40}{10}\frac{10!\,30!}{41!} = \frac{1}{41}$, so $\ln m(x) = -3.7136$; the evidence is 0.0244. In words, with a uniform prior every count from 0 to 40 is equally likely in advance. The posterior is $\text{Beta}(11, 31)$ (Priors, Posteriors and Conjugate Families, section "The beta–binomial model"), so by Theorem 1 the bound at $q = \text{Beta}(11, 31)$ is $\ln(1/41)$ exactly, which the script confirms from the definition of $\mathcal{L}$ with $E_q[\ln p] = \psi(a) - \psi(a + b)$ and $E_q[\ln(1 - p)] = \psi(b) - \psi(a + b)$ for a beta density with parameters $a$ and $b$ (the Dirichlet case of Variational Inference for LDA with two components). For any other $q$ the gap is $D(q \,\|\, \text{Beta}(11, 31))$ by (1). For $\text{Beta}(10, 30)$, which has mean 0.25 instead of 0.262 and nearly the same spread, the gap is 0.0178 nats. For $\text{Beta}(22, 62)$, which has the same mean, $22/84 = 11/42 = 0.262$, and about half the variance, the gap is 0.0986. Overconfidence about the same center costs more than a small shift.

Problem 2 (the price of independence for a correlated pair). Let $p$ be a bivariate normal with unit variances and correlation $\rho = 0.9$. Find the mean-field approximation, the ratio of its standard deviations to the true ones, and the divergence in each direction.

Solution. The precision matrix is $\Lambda = \frac{1}{1 - \rho^2}\begin{pmatrix}1 & -\rho\\ -\rho & 1\end{pmatrix}$, so $\Lambda_{jj} = 1/(1 - \rho^2)$, and by Theorem 5 part 1 each factor is normal with the true mean and variance $1 - \rho^2 = 0.19$. The standard deviations are $\sqrt{1 - \rho^2} = 0.436$ of the true ones. The divergence is $\tfrac12\ln(\Lambda_{11}\Lambda_{22}\det\Sigma) = \tfrac12\ln\big((1 - \rho^2)^{-2}(1 - \rho^2)\big) = -\tfrac12\ln(1 - \rho^2) = 0.830$ nats. In the forward direction (part 2) the factors are the unit-variance marginals, and the divergence is $-\tfrac12\ln\det R = -\tfrac12\ln(1 - \rho^2)$, the same 0.830. In two dimensions the two directions always cost the same, for any covariance: there $\Lambda_{11} = \Sigma_{22}/\det\Sigma$ and $\Lambda_{22} = \Sigma_{11}/\det\Sigma$, so $\Lambda_{11}\Lambda_{22}\det\Sigma = \Sigma_{11}\Sigma_{22}/\det\Sigma = 1/\det R$, and the two minima of Theorem 5 coincide. In the three dimensions of the normal (Laplace) approximation to the logistic regression posterior they did not (1.812 against 1.513).

Problem 3 (coordinate ascent on a correlated pair). For the same target, run coordinate ascent on the means of the two factors from $m_1 = m_2 = 1$ (the true means are 0), and find the rate of convergence. Compare it with the Gibbs sampler on the same target.

Solution. By (3) with $q_2$ fixed, $\ln q_1^*(\theta_1) = -\tfrac12\Lambda_{11}\theta_1^2 - \Lambda_{12}\theta_1 m_2 + \text{const}$, a normal with mean $-\Lambda_{12}m_2/\Lambda_{11} = \rho m_2$. Likewise the update of $q_2$ sets $m_2 = \rho m_1$. From $(1, 1)$, the first round gives $m_1 = 0.9$ and then $m_2 = 0.81$; the second gives $m_1 = 0.729$ and $m_2 = 0.6561$. Each round multiplies $m_2$ by $\rho^2 = 0.81$, so the error falls by a factor of 0.81 per round. The Gibbs sampler on the same target draws $\theta_1$ from a normal with mean $\rho\theta_2$ and then $\theta_2$ with mean $\rho\theta_1$. Markov Chain Monte Carlo (Worked problems) shows that its successive values of $\theta_2$ have autocorrelation $\rho^2$: the same factor governs how fast the sampler forgets its start and how fast coordinate ascent finds its fixed point. Strong correlation slows both.

Summary

  • The evidence lower bound $\mathcal{L}(q) = E_q[\ln f(\mathbf{x} \mid \theta) + \ln\pi(\theta) - \ln q(\theta)]$ satisfies $\ln m(\mathbf{x}) = \mathcal{L}(q) + D(q \,\|\, \pi(\cdot \mid \mathbf{x}))$: maximizing the bound minimizes the divergence from $q$ to the posterior (Theorem 1). EM is coordinate ascent on the same bound with an exact first step.
  • In a mean-field family the best factor satisfies $\ln q_j^* = E_{-j}[\ln f + \ln\pi] + \text{const}$ (Theorem 2), and coordinate ascent never lowers the bound; it may stop at a local optimum.
  • For the normal model with unknown mean and precision the fixed point is in closed form: exact means, variances too small by the factors $(a_n - 1)/a_n$ and $a_n/(a_n + \tfrac12)$, and a divergence that depends only on $a_n$ (Theorem 3). For the 25 adults: standard deviation 3.72 against 3.86, divergence 0.0171 nats.
  • The divergence $D(q \,\|\, p)$ forces $q$ away from regions where $p$ is small: on a two-mode target it picks one mode, with divergence near $\ln 2$; the forward direction matches moments and covers both (Theorem 4).
  • For a normal target the mean-field approximation reports each coordinate's conditional variance $1/\Lambda_{jj}$ (Theorem 5). On the normal (Laplace) approximation to the logistic regression posterior that is under a quarter of the true standard deviation for the intercept and the age coefficient.

What we have learned

Variational inference turns integration into optimization. The bound it maximizes is the log evidence minus the divergence from the approximation to the posterior, so all of its error is that divergence, and the divergence can be computed whenever the posterior is known. For the blood pressures it was small, 0.0171 nats, and its effect was entirely in the spreads: means exact, intervals a few percent too narrow, and narrower with fewer data. The direction of the divergence explains why. An approximation penalized for putting probability where the posterior has little will retreat to one mode, and a factorized one will report conditional spreads in place of marginal ones, which for strongly correlated coefficients can be a quarter of the truth. Markov Chain Monte Carlo makes the opposite trade: its errors are random and vanish with more computation, at the price of the computation.

This article closes the Bayesian Inference unit. The mean-field approximation of a normal used its precision matrix and conditional variances; Multivariate Statistics develops the multivariate normal distribution and its covariance matrices in full.

References