A random sample of 40 adults aged 18 to 39 from the National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018 contains 10 with hypertension. Confidence Intervals for a Proportion turned that count into intervals that cover the true proportion in a stated share of repeated samples, and The Likelihood Function showed that the likelihood $L(p) \propto p^{10}(1 - p)^{30}$ ranks the possible values of $p$ but is not a probability distribution over them. Suppose we want exactly that: a probability, given these 40 adults, that the proportion lies below 0.20, or the probability that 5 or more of the next 10 adults have hypertension. What must be added to the likelihood to get such probabilities, and when can the answer be written in closed form?
What must be added is a prior distribution: a probability distribution for the parameter, stated before the data are seen. Bayes' theorem combines it with the likelihood into a posterior distribution, the distribution of the parameter given the data. For some pairs of model and prior the posterior belongs to the same family as the prior, with parameters updated by the data. Such priors are called conjugate. This article derives the standard conjugate pairs and what follows from them, and it is the home of those results for the Bayesian Inference unit: later articles of the unit cite them from here.
First we state Bayes' theorem for a continuous parameter and apply it to the hypertension sample. Then we find the conjugate prior of every exponential family at once, and prove for one parameter that its posterior mean is a weighted average of the prior mean and the maximum likelihood estimate. Next we work through the five standard pairs: beta–binomial, gamma–Poisson, normal with known variance, normal with unknown mean and variance, and Dirichlet–multinomial, each with its predictive distribution for new data. Finally we show what happens without conjugacy. The article assumes The Likelihood Function, Maximum Likelihood Estimation and Exponential Families, and the conditional expectation identities of Conditional Expectation.
Bayes' theorem for a parameter
Predictive Values and Bayes' Theorem applies Bayes' theorem to events: a prior probability of disease, a likelihood of a test result, and a posterior probability of disease given the result. The same rule applies when the unknown is a number that can take a continuum of values. The parameter $\theta$ is now treated as a random variable with a density $\pi(\theta)$, the prior. Given $\theta$, the data $\mathbf{x} = (x_1, \ldots, x_n)$ have the probability mass function or density $f(\mathbf{x} \mid \theta)$, written with a bar because $\theta$ is now random (the Likelihood and Estimation unit wrote $f(\mathbf{x}; \theta)$ for the same function). Read as a function of $\theta$ for the observed data, $f(\mathbf{x} \mid \theta)$ is the likelihood $L(\theta)$. The joint density of $\theta$ and $\mathbf{x}$ is $f(\mathbf{x} \mid \theta)\pi(\theta)$, and dividing by the marginal density of $\mathbf{x}$ gives the conditional density of $\theta$:
$$\pi(\theta \mid \mathbf{x}) = \frac{f(\mathbf{x} \mid \theta)\,\pi(\theta)}{m(\mathbf{x})}, \qquad m(\mathbf{x}) = \int f(\mathbf{x} \mid \theta)\,\pi(\theta)\,d\theta. \qquad (1)$$
The denominator $m(\mathbf{x})$ is the marginal likelihood, also called the prior predictive distribution or the evidence: the probability (or density) of the data averaged over the prior. It does not involve $\theta$, so the posterior is proportional to likelihood times prior, $\pi(\theta \mid \mathbf{x}) \propto L(\theta)\pi(\theta)$, and $m(\mathbf{x})$ is whatever constant makes the posterior integrate to 1. Any factor of the likelihood that does not involve $\theta$, such as a binomial coefficient, cancels in the same way. That is why The Likelihood Function could say that only ratios of likelihoods carry meaning: the posterior uses the likelihood only up to a constant. The marginal likelihood comes back when two models are compared, in Bayes Factors and Posterior Model Probabilities.
For the hypertension sample, take the uniform prior on $[0, 1]$, $\pi(p) = 1$. Then the posterior is proportional to $p^{10}(1 - p)^{30}$. A density on $[0, 1]$ proportional to $p^{a-1}(1 - p)^{b-1}$, with $a, b > 0$, is the beta distribution $\text{Beta}(a, b)$:
$$\pi(p) = \frac{p^{a-1}(1 - p)^{b-1}}{\mathrm{B}(a, b)}, \qquad \mathrm{B}(a, b) = \int_0^1 u^{a-1}(1 - u)^{b-1}\,du = \frac{\Gamma(a)\,\Gamma(b)}{\Gamma(a + b)},$$
where $\Gamma$ is the gamma function, with $\Gamma(z + 1) = z\Gamma(z)$ and $\Gamma(k) = (k - 1)!$ for whole numbers $k$. The last equality is standard (Bishop, 2006, eq. 2.13); Latent Dirichlet Allocation proves it, as the case of two categories of the Dirichlet integral, by a change of variables in a product of gamma integrals. Its mean is $a/(a + b)$, which Probability for Information Retrieval proves from two properties of the integral. So the posterior is $\text{Beta}(11, 31)$. Its mean is $11/42 = 0.2619$, its mode is $10/40 = 0.25$, the maximum likelihood estimate, and its standard deviation is 0.0670. The posterior probability that $p$ is below 0.20 is 0.1823, and the central 95% of the posterior lies between 0.1422 and 0.4030.
Example (what the posterior says about the population). In this case the true proportion is known: 389 of the 1,696 young adults in the survey have hypertension, $p = 0.2294$. The posterior probability that $p$ exceeds 0.2294 is 0.6681, so the true value sits at the 33rd percentile of the posterior. Nothing is wrong with that. A posterior describes what the sample and the prior say, and a sample whose proportion, 0.25, is above the truth gives a posterior centered above the truth. How often such statements are right across samples, and in what sense, is the subject of Credible Intervals and How They Differ from Confidence Intervals.
Two more quantities follow without any new computation. The marginal likelihood under the uniform prior is $m(x) = \binom{n}{x}\int_0^1 p^x(1-p)^{n-x}\,dp = 1/(n + 1)$ for every count $x$, the integral proved in The Likelihood Function: before the data, the uniform prior makes all 41 counts equally likely. Gelman et al. (2013, §2.1) note that this was Bayes' own calculation in 1763. And the probability that the next adult sampled has hypertension is the posterior mean of $p$, since $P(\tilde x = 1 \mid \mathbf{x}) = \int p\,\pi(p \mid \mathbf{x})\,dp$. With the uniform prior it is $(x + 1)/(n + 2) = 11/42$, Laplace's law of succession (Gelman et al., 2013, §2.1, eq. 2.6).
Conjugate families
The beta prior gave a beta posterior because the likelihood, as a function of $p$, has the same form as the beta density: a power of $p$ times a power of $1 - p$. Gelman et al. (2013, §2.4) define a class $\mathcal{P}$ of priors to be conjugate for a class of sampling distributions if every prior in $\mathcal{P}$, combined with every sampling distribution in the class, gives a posterior in $\mathcal{P}$. The useful case is the natural conjugate family, whose densities have the same functional form in $\theta$ as the likelihood. Every exponential family has one, and one theorem finds them all.
Here is the idea in one sentence: the likelihood of an exponential family is an exponential of a linear function of $\theta$ and $A(\theta)$, so a prior of the same form only adds to the two coefficients. Write the family in the natural form of Exponential Families, for one observation,
$$f(x \mid \boldsymbol\theta) = h(x)\exp\{\boldsymbol\theta^{\mathsf T}\mathbf{T}(x) - A(\boldsymbol\theta)\}, \qquad \boldsymbol\theta \in \Theta \subseteq \mathbb{R}^d,$$
with natural parameter $\boldsymbol\theta$, sufficient statistic $\mathbf{T}$ and log-partition function $A$; for one parameter ($d = 1$) we drop the bold type and $\Theta$ is an open interval. That article proves that in one dimension $\mu(\theta) = A'(\theta) = E[T(X) \mid \theta]$, the mean parameter, increases strictly with $\theta$, and that the average $\bar T = \frac1n\sum_i T(x_i)$ is the maximum likelihood estimate of $\mu$ whenever one exists (it can fail to exist, for example for a Bernoulli sample of all zeros).
Theorem 1 (the conjugate prior of an exponential family). For a vector $\boldsymbol\chi \in \mathbb{R}^d$ and a number $\nu$ such that $K(\boldsymbol\chi, \nu) = \int_\Theta \exp\{\boldsymbol\chi^{\mathsf T}\boldsymbol\theta - \nu A(\boldsymbol\theta)\}\,d\boldsymbol\theta$ is finite, let
$$\pi(\boldsymbol\theta) = \frac{\exp\{\boldsymbol\chi^{\mathsf T}\boldsymbol\theta - \nu A(\boldsymbol\theta)\}}{K(\boldsymbol\chi, \nu)}.$$
If $x_1, \ldots, x_n$ are independent given $\boldsymbol\theta$ with mass function or density $f(x \mid \boldsymbol\theta)$, the posterior is the member of the same family with
$$\boldsymbol\chi_n = \boldsymbol\chi + \sum_{i=1}^n \mathbf{T}(x_i), \qquad \nu_n = \nu + n.$$
Proof. The likelihood is $\prod_i h(x_i) \exp\{\boldsymbol\theta^{\mathsf T}\sum_i \mathbf{T}(x_i) - nA(\boldsymbol\theta)\}$. Multiplying by the prior and dropping the factors free of $\boldsymbol\theta$,
$$\pi(\boldsymbol\theta \mid \mathbf{x}) \propto \exp\Big\{\Big(\boldsymbol\chi + \sum_i \mathbf{T}(x_i)\Big)^{\mathsf T}\boldsymbol\theta - (\nu + n)A(\boldsymbol\theta)\Big\}.$$
The right side has a finite integral whenever $m(\mathbf{x})$ is finite, which holds for every sample of a discrete model (there $f(\mathbf{x} \mid \theta) \le 1$) and, for a model with densities, for every sample outside a set of probability zero, because $m$ itself integrates to 1 over the samples. Normalizing gives the member with parameters $(\boldsymbol\chi_n, \nu_n)$. $\square$
So the prior acts like $\nu$ earlier observations whose statistics total $\boldsymbol\chi$. Gelman et al. (2013, §2.4) give the same construction, and add that, apart from irregular cases, exponential families are the only families with natural conjugate priors, because they are the only ones with sufficient statistics of fixed dimension for every $n$.
The second theorem explains why conjugate posterior means look like averages of a prior guess and the data. Diaconis and Ylvisaker (1979) showed, according to their abstract, that conjugate priors are characterized by this property: the posterior expectation of the mean parameter is a linear function of the data. One direction is short enough to prove here, for one parameter and under a condition at the ends of $\Theta$.
Theorem 2 (the posterior mean of the mean parameter). Let $d = 1$, $\Theta = (\theta_-, \theta_+)$, with either end possibly infinite, and let the density $\pi(\theta) \propto \exp\{\chi\theta - \nu A(\theta)\}$ of Theorem 1, with $\nu > 0$, tend to 0 at both ends of $\Theta$. Then $E[\mu(\theta)] = \chi/\nu$ under $\pi$. Consequently, if the posterior density of Theorem 1 also tends to 0 at both ends,
$$E[\mu(\theta) \mid \mathbf{x}] = \frac{\chi + \sum_i T(x_i)}{\nu + n} = w\,\frac{\chi}{\nu} + (1 - w)\,\bar T, \qquad w = \frac{\nu}{\nu + n}. \qquad (2)$$
Proof. Differentiating, $\pi'(\theta) = (\chi - \nu\mu(\theta))\,\pi(\theta)$, since $\mu = A'$. The function $\mu$ is increasing, so $\chi - \nu\mu(\theta)$ is decreasing; choose $c \in \Theta$ where it changes sign, or any $c$ if it never does. Then the integrand $(\chi - \nu\mu)\pi$ keeps one sign on $(\theta_-, c]$ and one sign on $[c, \theta_+)$. For $\theta_- < a < c < b < \theta_+$,
$$\int_a^c (\chi - \nu\mu(\theta))\,\pi(\theta)\,d\theta = \pi(c) - \pi(a), \qquad \int_c^b (\chi - \nu\mu(\theta))\,\pi(\theta)\,d\theta = \pi(b) - \pi(c).$$
As $a \to \theta_-$ and $b \to \theta_+$ these tend to $\pi(c)$ and $-\pi(c)$, because $\pi$ tends to 0 at both ends. An integrand of fixed sign whose integrals over growing intervals converge is integrable over the whole piece, so $(\chi - \nu\mu)\pi$ is integrable over $\Theta$ with integral $\pi(c) - \pi(c) = 0$. Since $\pi$ integrates to 1, $\mu\pi$ is integrable as well, and $\chi - \nu E[\mu] = 0$. The posterior is the member with $(\chi_n, \nu_n)$, so applying the same result to it gives the first expression in (2). The second is algebra: $\chi + n\bar T = \nu\cdot\frac{\chi}{\nu} + n\bar T$, divided by $\nu + n$. $\square$
Read (2) as follows. The prior mean of $\mu$ is $\chi/\nu$, the average $\bar T$ is the maximum likelihood estimate whenever one exists, and the posterior mean lies between them, with the prior weighted as $\nu$ observations against the data's $n$. The weight $w$ on the prior falls toward 0 as $n$ grows, so with enough data the prior stops mattering for the posterior mean. The condition at the ends is what makes the boundary terms vanish; the beta and gamma cases below show that it holds for every proper member of those two families.
The table lists the families this article uses. The first three columns come from Exponential Families; the conjugate prior in the last column is $\exp\{\chi\theta - \nu A(\theta)\}$ rewritten in the usual parameter, which multiplies the density by $\lvert d\theta/d\text{(parameter)}\rvert$.
| Model for one observation | Natural parameter $\theta$ | $A(\theta)$ | $T(x)$ | Conjugate prior on the usual parameter |
|---|---|---|---|---|
| Bernoulli($p$) | $\ln\frac{p}{1-p}$ | $\ln(1 + e^\theta)$ | $x$ | $\text{Beta}(\chi,\ \nu - \chi)$ |
| Poisson($\lambda$) | $\ln\lambda$ | $e^\theta$ | $x$ | $\text{Gamma}(\chi,\ \nu)$ |
| Normal, mean $\mu$, known variance $\sigma^2$ | $\mu/\sigma^2$ | $\sigma^2\theta^2/2$ | $x$ | normal |
| Categorical over $K$ classes | log odds against class $K$ | $\ln\big(1 + \sum_{k<K} e^{\theta_k}\big)$ | indicators | Dirichlet |
For the Bernoulli row: $\exp\{\chi\theta - \nu\ln(1 + e^\theta)\} = p^\chi(1-p)^{\nu - \chi}$, and $d\theta/dp = 1/[p(1-p)]$, so the density in $p$ is proportional to $p^{\chi - 1}(1-p)^{\nu - \chi - 1}$, the $\text{Beta}(a, b)$ density with $a = \chi$ and $b = \nu - \chi$. It is a density exactly when $a, b > 0$, and then $p^\chi(1-p)^{\nu-\chi} \to 0$ as $\theta \to \pm\infty$, so Theorem 2 applies: the prior mean of $p$ is $\chi/\nu = a/(a + b)$, as it should be. For the Poisson row: $\exp\{\chi\ln\lambda - \nu\lambda\}$ times $d\theta/d\lambda = 1/\lambda$ is $\lambda^{\chi - 1}e^{-\nu\lambda}$, the gamma density with shape $\chi$ and rate $\nu$, which needs $\chi, \nu > 0$; then $\lambda^\chi e^{-\nu\lambda} \to 0$ at both ends of $\theta \in \mathbb{R}$. The normal row is Theorem 6 below. The categorical row is Theorem 1 with $d = K - 1$. With $\theta_k = \ln(p_k/p_K)$ for $k < K$, the log-partition function is $A(\boldsymbol\theta) = \ln\big(1 + \sum_{k<K}p_k/p_K\big) = -\ln p_K$, so $\exp\{\boldsymbol\chi^{\mathsf T}\boldsymbol\theta - \nu A(\boldsymbol\theta)\} = \prod_{k<K}p_k^{\chi_k}\cdot p_K^{\nu - \sum_k\chi_k}$. The matrix of derivatives of the map from $\boldsymbol\theta$ to $(p_1, \ldots, p_{K-1})$ has entries $\partial p_j/\partial\theta_k = p_j(\delta_{jk} - p_k)$, with $\delta_{jk} = 1$ if $j = k$ and 0 otherwise; it is $\text{diag}(\mathbf{p}) - \mathbf{p}\mathbf{p}^{\mathsf T}$. By the identity $\det(D + \mathbf{a}\mathbf{b}^{\mathsf T}) = \det(D)(1 + \mathbf{b}^{\mathsf T}D^{-1}\mathbf{a})$, proved in Choosing a Prior (Theorem 4, part 6), its determinant is $\prod_{k<K}p_k\cdot\big(1 - \sum_{k<K}p_k\big) = \prod_{k=1}^K p_k$. Dividing by it, as the multivariate change of variables requires, gives a density in $\mathbf{p}$ proportional to $\prod_{k<K}p_k^{\chi_k - 1}\cdot p_K^{\nu - \sum_k\chi_k - 1}$, the $\text{Dirichlet}(\chi_1, \ldots, \chi_{K-1},\ \nu - \sum_k\chi_k)$ density. With $K = 2$ this is the Bernoulli row. That the Dirichlet is conjugate is also checked directly in its section below.
The beta–binomial model
With a $\text{Beta}(a, b)$ prior and $x$ successes in $n$ independent trials, the posterior is proportional to $p^{x}(1-p)^{n-x}\cdot p^{a-1}(1-p)^{b-1}$, so
$$p \mid x \sim \text{Beta}(a + x,\ b + n - x). \qquad (3)$$
The prior parameters act as $a$ earlier successes and $b$ earlier failures, and $a + b$ is the prior sample size. By (2), or directly from the beta mean, the posterior mean is
$$E[p \mid x] = \frac{a + x}{a + b + n} = w\,\frac{a}{a + b} + (1 - w)\,\frac{x}{n}, \qquad w = \frac{a + b}{a + b + n}.$$
The posterior variance, from the beta variance $ab/[(a+b)^2(a+b+1)]$ (Gelman et al., 2013, §2.4; it is Theorem 8 below with $K = 2$), is
$$\operatorname{Var}(p \mid x) = \frac{E[p \mid x]\,\big(1 - E[p \mid x]\big)}{a + b + n + 1},$$
which shrinks like $1/n$.
The prior is not determined by the data, and different priors give different posteriors. The table compares three on the hypertension sample. The uniform prior is $\text{Beta}(1, 1)$. Jeffreys' prior, $\text{Beta}(\tfrac12, \tfrac12)$, is derived in Choosing a Prior. The third, $\text{Beta}(3, 17)$, describes an analyst who expected a prevalence near 15% (the prior mean is $3/20 = 0.15$) and held that belief as firmly as 20 earlier observations; it is an illustration, not a prior anyone published.
| Prior | Posterior | Weight $w$ on prior | Posterior mean | Posterior SD | Central 95% |
|---|---|---|---|---|---|
| $\text{Beta}(1, 1)$, uniform | $\text{Beta}(11, 31)$ | 0.0476 | 0.2619 | 0.0670 | (0.1422, 0.4030) |
| $\text{Beta}(\tfrac12, \tfrac12)$, Jeffreys | $\text{Beta}(10.5, 30.5)$ | 0.0244 | 0.2561 | 0.0673 | (0.1364, 0.3983) |
| $\text{Beta}(3, 17)$, prior mean 0.15 | $\text{Beta}(13, 47)$ | 0.3333 | 0.2167 | 0.0527 | (0.1229, 0.3283) |
For the third prior, $w = 20/60 = 1/3$ and the posterior mean is $\tfrac13(0.15) + \tfrac23(0.25) = 13/60 = 0.2167$. The two weak priors give nearly the same answer, because 2 and 1 pseudo-observations count little against 40 real ones. The informative prior moves the posterior mean down by 0.045 and narrows the interval, because it adds 20 observations' worth of belief.
Example (an informative prior does not make the analysis wrong). "A prior centered at 0.15 must make the analysis wrong when the truth is 0.2294." The posterior mean 0.2167 is closer to the truth than the maximum likelihood estimate 0.25 in this sample, because this sample happened to come out high. In other samples the same prior pulls an estimate that was already low further down. As an estimator the posterior mean is biased: its expectation over samples is $w\cdot 0.15 + (1 - w)p$, so its bias at $p = 0.2294$ is $w(0.15 - p) = -0.026$, while its variance, $(1 - w)^2p(1-p)/n$, is $4/9$ of the variance of $\hat p$. An informative prior trades variance for bias, as the shrinkage estimators of Hierarchical Models and Shrinkage do; whether the trade pays depends on how far the prior's center is from the truth, which no single sample reveals.
Updating in stages
Data that arrive in batches can be used as they arrive: the posterior after the first batch becomes the prior for the second. The order and the grouping make no difference.
Theorem 3 (sequential updating). If $\mathbf{x}_1$ and $\mathbf{x}_2$ are conditionally independent given $\theta$, the posterior obtained by updating $\pi(\theta)$ with $\mathbf{x}_1$ and then updating the result with $\mathbf{x}_2$ equals the posterior from both batches at once.
Proof. By conditional independence, $f(\mathbf{x}_1, \mathbf{x}_2 \mid \theta) = f(\mathbf{x}_1 \mid \theta)f(\mathbf{x}_2 \mid \theta)$. Updating in two steps gives a density proportional to $f(\mathbf{x}_2 \mid \theta)\,\pi(\theta \mid \mathbf{x}_1) \propto f(\mathbf{x}_2 \mid \theta)f(\mathbf{x}_1 \mid \theta)\pi(\theta)$, the same function of $\theta$ as the one-step posterior, and two densities proportional to the same function are equal. $\square$
In the beta–binomial model the theorem is visible in the parameters: each batch adds its successes to $a$ and its failures to $b$. The 40 adults were drawn one at a time, and the first 20 contained 7 with hypertension, the last 20 contained 3. Worked Problem 1 carries out both orders.
The posterior predictive distribution
The posterior answers questions about $p$. Questions about future data, such as how many of the next $m$ adults will have hypertension, are answered by the posterior predictive distribution, which averages the model for new data over the posterior:
$$f(\tilde y \mid \mathbf{x}) = \int f(\tilde y \mid \theta)\,\pi(\theta \mid \mathbf{x})\,d\theta.$$
This assumes that the new data are conditionally independent of the old given $\theta$, so that $f(\tilde y \mid \theta, \mathbf{x}) = f(\tilde y \mid \theta)$.
Theorem 4 (the beta-binomial predictive distribution). If $p \sim \text{Beta}(a, b)$ and, given $p$, $Y \sim \text{Binomial}(m, p)$, then
$$P(Y = y) = \binom{m}{y}\frac{\mathrm{B}(a + y,\ b + m - y)}{\mathrm{B}(a, b)}, \quad y = 0, \ldots, m, \qquad E[Y] = \frac{ma}{a + b}, \qquad \operatorname{Var}(Y) = \frac{m\,ab\,(a + b + m)}{(a + b)^2(a + b + 1)}.$$
Proof. $P(Y = y) = \int_0^1 \binom{m}{y}p^y(1-p)^{m-y}\,\frac{p^{a-1}(1-p)^{b-1}}{\mathrm{B}(a, b)}\,dp$, and the integral of $p^{a + y - 1}(1-p)^{b + m - y - 1}$ is $\mathrm{B}(a + y, b + m - y)$ by definition. For the moments use the tower property and the law of total variance from Conditional Expectation: $E[Y] = E[mp] = ma/(a + b)$, and $\operatorname{Var}(Y) = E[mp(1 - p)] + \operatorname{Var}(mp)$. With $\bar p = a/(a+b)$ and $\operatorname{Var}(p) = \bar p(1 - \bar p)/(a + b + 1)$,
$$E[p(1-p)] = \bar p - E[p^2] = \bar p - \operatorname{Var}(p) - \bar p^2 = \bar p(1 - \bar p)\Big(1 - \frac{1}{a + b + 1}\Big) = \frac{\bar p(1 - \bar p)(a + b)}{a + b + 1}.$$
So $\operatorname{Var}(Y) = \frac{\bar p(1 - \bar p)}{a + b + 1}\,[m(a + b) + m^2] = \frac{m\bar p(1 - \bar p)(a + b + m)}{a + b + 1}$, which is the stated form since $\bar p(1 - \bar p) = ab/(a + b)^2$. $\square$
The variance exceeds the binomial variance $m\bar p(1 - \bar p)$ by the factor $(a + b + m)/(a + b + 1)$, the price of not knowing $p$. For the next $m = 10$ young adults under the uniform-prior posterior $\text{Beta}(11, 31)$, the expected number with hypertension is $10 \times 11/42 = 2.619$ and the variance is 2.338, against 1.933 for a binomial with $p = 11/42$. The predictive probabilities of 0 to 6 are 0.0663, 0.1824, 0.2526, 0.2304, 0.1526, 0.0763 and 0.0291, and the probability of 5 or more is 0.1156. Plugging in the estimate $\hat p = 0.25$ and pretending it is exact gives 0.0781 instead. The plug-in answer is too small because it ignores the posterior's spread: values of $p$ above 0.3 are still plausible, and they make five cases in ten much more likely.
The gamma–Poisson model
The Likelihood Function models the weekly number of cancer deaths in Iowa as Poisson. The gamma distribution with shape $a > 0$ and rate $b > 0$ has density $b^a\lambda^{a-1}e^{-b\lambda}/\Gamma(a)$ on $\lambda > 0$, mean $a/b$ and variance $a/b^2$; the normalizing constant comes from $\int_0^\infty \lambda^{c-1}e^{-d\lambda}\,d\lambda = \Gamma(c)/d^c$, which is the definition of $\Gamma(c)$ after the substitution $u = d\lambda$.
Theorem 5 (gamma–Poisson updating and the negative binomial predictive). If $\lambda \sim \text{Gamma}(a, b)$ and $x_1, \ldots, x_n$ are independent Poisson($\lambda$) given $\lambda$, then $\lambda \mid \mathbf{x} \sim \text{Gamma}(a + \sum_i x_i,\ b + n)$. If $\lambda \sim \text{Gamma}(a, b)$ and $Y$ is Poisson($\lambda$) given $\lambda$, then
$$P(Y = y) = \frac{\Gamma(a + y)}{\Gamma(a)\,y!}\Big(\frac{b}{b + 1}\Big)^{a}\Big(\frac{1}{b + 1}\Big)^{y}, \qquad E[Y] = \frac{a}{b}, \qquad \operatorname{Var}(Y) = \frac{a}{b} + \frac{a}{b^2}.$$
Proof. The likelihood is $\prod_i e^{-\lambda}\lambda^{x_i}/x_i! \propto \lambda^{\sum x_i}e^{-n\lambda}$, and times the prior it is $\lambda^{a + \sum x_i - 1}e^{-(b + n)\lambda}$, the stated gamma kernel. For the predictive,
$$P(Y = y) = \int_0^\infty \frac{e^{-\lambda}\lambda^y}{y!}\,\frac{b^a\lambda^{a-1}e^{-b\lambda}}{\Gamma(a)}\,d\lambda = \frac{b^a}{\Gamma(a)\,y!}\int_0^\infty \lambda^{a + y - 1}e^{-(b+1)\lambda}\,d\lambda = \frac{b^a}{\Gamma(a)\,y!}\,\frac{\Gamma(a + y)}{(b + 1)^{a + y}},$$
which rearranges to the stated form. The moments follow from the tower property and total variance: $E[Y] = E[\lambda] = a/b$ and $\operatorname{Var}(Y) = E[\lambda] + \operatorname{Var}(\lambda) = a/b + a/b^2$. $\square$
The predictive distribution is the negative binomial with mean $\mu = a/b$ and shape $k = a$, whose variance $\mu + \mu^2/k$ is the form used in Overdispersion in Count Data (here $\mu^2/k = a/b^2$). Overdispersion in Count Data derives the same gamma mixture of Poisson distributions, with scale $1/b$, and Gelman et al. (2013, §2.6) obtain it the same way. Read in the other direction, counts that vary more than a Poisson model allows are what one sees when the rate itself varies from unit to unit.
Iowa's weekly cancer deaths. Take a prior centered at 125 deaths a week but deliberately weak, $\text{Gamma}(50, 0.4)$: its mean is $50/0.4 = 125$, its standard deviation 17.68, and it acts as 0.4 of a week of data. The 53 weeks of 2014 (by the CDC's epidemiological week calendar) recorded 6,535 cancer deaths, a maximum likelihood estimate of 123.30 per week. The posterior is $\text{Gamma}(6{,}585,\ 53.4)$, with mean $6{,}585/53.4 = 123.31$ and standard deviation 1.520; the weight on the prior mean is $0.4/53.4 = 0.0075$, so the data decide. The central 95% of the posterior is 120.35 to 126.31 deaths per week.
The predictive distribution for a single week of 2015 is negative binomial with mean 123.31 and variance $123.31 \times (1 + 1/53.4) = 125.62$, a standard deviation of 11.21. Its central interval, from the 2.5% to the 97.5% point, is 102 to 146 deaths, with probability 0.9553 (the distribution is discrete, so the probability is not exactly 0.95). With 53 weeks behind it, uncertainty about $\lambda$ adds only 1.9% to the Poisson variance, and the plug-in Poisson interval with $\lambda = 123.30$ is also 102 to 146.
Of the 52 weeks of 2015, 4 fell outside that interval (the counts ranged from 99 to 160), where 2.3 would be expected. That is within chance: if the predictive distribution were right, 4 or more of 52 weeks would fall outside with probability 0.20. The 2015 counts had sample variance 160.8 around a mean of 124.15, a ratio of 1.30; for 52 Poisson counts a ratio this large or larger has probability about 0.08 (the index of dispersion, 66.05 on 51 degrees of freedom), and over the six years 2014 to 2019 the ratio is 0.958 (Overdispersion in Count Data). Iowa's weekly cancer deaths are a count the Poisson model fits. A count whose rate moves from week to week, such as all-cause deaths with their winter peaks, is different: there a model with one fixed rate understates the spread of future weeks however precisely that rate is estimated, which is the subject of Overdispersion and Quasi-Likelihood.
The normal model with known variance
Here is the idea: a normal likelihood for a mean is the exponential of a quadratic in $\mu$, a normal prior is too, and the sum of two quadratics in $\mu$ is a quadratic in $\mu$, so the posterior is normal. One identity does the algebra. For $u, v \ge 0$ with $u + v > 0$ and any numbers $c$ and $d$,
$$u(\mu - c)^2 + v(\mu - d)^2 = (u + v)\Big(\mu - \frac{uc + vd}{u + v}\Big)^2 + \frac{uv}{u + v}(c - d)^2. \qquad (4)$$
To check it, expand both sides. The coefficients of $\mu^2$ are $u + v$ on both. The coefficients of $\mu$ are $-2(uc + vd)$ on both. The constant on the right is $\frac{(uc + vd)^2 + uv(c - d)^2}{u + v} = \frac{u^2c^2 + v^2d^2 + uvc^2 + uvd^2}{u + v} = \frac{(u + v)(uc^2 + vd^2)}{u + v} = uc^2 + vd^2$, the constant on the left.
Theorem 6 (normal–normal updating). Let $x_1, \ldots, x_n$ be independent and normal with mean $\mu$ and known variance $\sigma^2$, and let the prior of $\mu$ be normal with mean $\mu_0$ and variance $\tau_0^2$. Then the posterior of $\mu$ is normal with
$$\frac{1}{\tau_n^2} = \frac{1}{\tau_0^2} + \frac{n}{\sigma^2}, \qquad \mu_n = \tau_n^2\Big(\frac{\mu_0}{\tau_0^2} + \frac{n\bar x}{\sigma^2}\Big),$$
and the predictive distribution of a new observation $\tilde x$ is normal with mean $\mu_n$ and variance $\sigma^2 + \tau_n^2$.
Proof. Since $\sum_i(x_i - \mu)^2 = \sum_i(x_i - \bar x)^2 + n(\bar x - \mu)^2$, the likelihood is proportional to $\exp\{-n(\mu - \bar x)^2/(2\sigma^2)\}$. Times the prior density, the posterior is proportional to
$$\exp\Big\{-\frac12\Big[\frac{1}{\tau_0^2}(\mu - \mu_0)^2 + \frac{n}{\sigma^2}(\mu - \bar x)^2\Big]\Big\}.$$
By (4) with $u = 1/\tau_0^2$ and $v = n/\sigma^2$, the bracket is $(u + v)(\mu - \mu_n)^2$ plus a term free of $\mu$, so the posterior is proportional to $\exp\{-(\mu - \mu_n)^2/(2\tau_n^2)\}$, the normal density with mean $\mu_n$ and variance $\tau_n^2$. For the predictive, write $\tilde x = \mu + e$, where given the data $\mu$ has the posterior distribution and $e$ is normal with mean 0 and variance $\sigma^2$, independent of $\mu$. A sum of independent normal variables is normal, with the means and the variances added (Moment Generating and Characteristic Functions). $\square$
The inverse of a variance is called a precision, and the theorem says that precisions add: the prior contributes $1/\tau_0^2$ and the data $n/\sigma^2$. The posterior mean is the precision-weighted average of the prior mean and $\bar x$, and the prior counts as $\sigma^2/\tau_0^2$ observations. Gelman et al. (2013, §2.5) give the same result and the same reading; it is (2) for this family. The predictive variance has two parts: $\sigma^2$, the variation of a new person around the mean, which no amount of data removes, and $\tau_n^2$, the uncertainty about the mean, which shrinks like $1/n$.
Systolic blood pressure. Among the 5,247 adults of the survey with a systolic reading, the mean is 126.36 mm Hg and the standard deviation (divisor $N$) is 20.12, so take $\sigma^2 = 405.01$ as known. A random sample of 25 (NumPy generator, seed 1) has mean $\bar x = 128.69$. Take a prior normal with mean $\mu_0 = 120$ (a round value chosen for illustration) and standard deviation $\tau_0 = 10$. The data's precision is $25/405.01 = 0.0617$ and the prior's is $0.01$, so the prior counts as $405.01/100 = 4.05$ observations and receives weight $0.01/0.0717 = 0.139$. The posterior has mean 127.48, standard deviation 3.734, and central 95% interval 120.16 to 134.80, which contains the population mean 126.36. A new adult's pressure is predicted to be normal with mean 127.48 and standard deviation $\sqrt{405.01 + 13.94} = 20.47$, barely more than $\sigma = 20.12$: the mean is already known well enough that a new person's own variation dominates.
The normal model with unknown mean and variance
When $\sigma^2$ is unknown as well, the conjugate prior must describe both parameters, and the form of the likelihood dictates it. Write the inverse gamma distribution $\text{IG}(a, b)$ for the distribution of a positive variable $V$ whose reciprocal is $\text{Gamma}(a, b)$; its density is $\frac{b^a}{\Gamma(a)}\,v^{-a-1}e^{-b/v}$ (change of variables $v = 1/\lambda$, with $\lvert d\lambda/dv\rvert = v^{-2}$). The normal–inverse-gamma prior is
$$\sigma^2 \sim \text{IG}\Big(\frac{\nu_0}{2},\ \frac{\nu_0\sigma_0^2}{2}\Big), \qquad \mu \mid \sigma^2 \ \text{normal with mean } \mu_0 \text{ and variance } \sigma^2/\kappa_0.$$
Its four parameters have readings as data: the prior is worth $\kappa_0$ observations about the mean, centered at $\mu_0$, and $\nu_0$ observations about the variance, with average squared deviation $\sigma_0^2$. Gelman et al. (2013, §3.3) write the same prior with a scaled inverse-$\chi^2$ distribution, which is this inverse gamma under another name. Its density is proportional to
$$(\sigma^2)^{-(\nu_0/2 + 1)}\,(\sigma^2)^{-1/2}\exp\Big\{-\frac{\nu_0\sigma_0^2 + \kappa_0(\mu - \mu_0)^2}{2\sigma^2}\Big\}.$$
The Student t distribution with $\nu$ degrees of freedom, location $m$ and scale $c$ is the distribution of $m + cT$ with $T$ a standard $t_\nu$ variable; its density is proportional to $[1 + (y - m)^2/(\nu c^2)]^{-(\nu + 1)/2}$ (Chi-Squared and t Distributions derives the standard density).
Theorem 7 (normal–inverse-gamma updating). With the prior above and $x_1, \ldots, x_n$ independent and normal with mean $\mu$ and variance $\sigma^2$ given both, write $\bar x$ for the sample mean and $s^2$ for the sample variance with divisor $n - 1$. The posterior is normal–inverse-gamma with
$$\kappa_n = \kappa_0 + n, \quad \mu_n = \frac{\kappa_0\mu_0 + n\bar x}{\kappa_n}, \quad \nu_n = \nu_0 + n, \quad \nu_n\sigma_n^2 = \nu_0\sigma_0^2 + (n - 1)s^2 + \frac{\kappa_0 n}{\kappa_n}(\bar x - \mu_0)^2.$$
Marginally, $\sigma^2 \mid \mathbf{x} \sim \text{IG}(\nu_n/2,\ \nu_n\sigma_n^2/2)$; $\mu \mid \mathbf{x}$ is Student $t$ with $\nu_n$ degrees of freedom, location $\mu_n$ and scale $\sigma_n/\sqrt{\kappa_n}$; and a new observation $\tilde x$ is Student $t$ with $\nu_n$ degrees of freedom, location $\mu_n$ and scale $\sigma_n\sqrt{1 + 1/\kappa_n}$.
Proof. The likelihood is $(2\pi\sigma^2)^{-n/2}\exp\{-\sum_i(x_i - \mu)^2/(2\sigma^2)\}$, and $\sum_i(x_i - \mu)^2 = (n - 1)s^2 + n(\bar x - \mu)^2$. Multiplying by the prior, the exponent of $\sigma^2$ in front becomes $-(\nu_0/2 + 1) - 1/2 - n/2 = -(\nu_n/2 + 1) - 1/2$, and the numerator in the exponential becomes
$$\nu_0\sigma_0^2 + (n - 1)s^2 + \kappa_0(\mu - \mu_0)^2 + n(\mu - \bar x)^2 = \nu_n\sigma_n^2 + \kappa_n(\mu - \mu_n)^2,$$
by (4) with $u = \kappa_0$ and $v = n$. So the posterior density is proportional to $(\sigma^2)^{-(\nu_n/2+1)}(\sigma^2)^{-1/2}\exp\{-[\nu_n\sigma_n^2 + \kappa_n(\mu - \mu_n)^2]/(2\sigma^2)\}$, the prior's form with the updated parameters.
Marginal of $\sigma^2$. Integrating over $\mu$, $\int\exp\{-\kappa_n(\mu - \mu_n)^2/(2\sigma^2)\}\,d\mu = \sqrt{2\pi\sigma^2/\kappa_n}$, which cancels the factor $(\sigma^2)^{-1/2}$ and leaves $(\sigma^2)^{-(\nu_n/2 + 1)}e^{-\nu_n\sigma_n^2/(2\sigma^2)}$, the stated inverse gamma.
Marginal of $\mu$. Write $B = \nu_n\sigma_n^2 + \kappa_n(\mu - \mu_n)^2$ and integrate over $\sigma^2$, substituting $u = 1/\sigma^2$, $d\sigma^2 = u^{-2}du$:
$$\int_0^\infty (\sigma^2)^{-(\nu_n + 1)/2 - 1}e^{-B/(2\sigma^2)}\,d\sigma^2 = \int_0^\infty u^{(\nu_n + 1)/2 - 1}e^{-Bu/2}\,du = \Gamma\Big(\frac{\nu_n + 1}{2}\Big)\Big(\frac{B}{2}\Big)^{-(\nu_n + 1)/2}.$$
As a function of $\mu$ this is proportional to $B^{-(\nu_n+1)/2} \propto \big[1 + \kappa_n(\mu - \mu_n)^2/(\nu_n\sigma_n^2)\big]^{-(\nu_n+1)/2}$, the $t$ density with $\nu_n$ degrees of freedom, location $\mu_n$ and scale $\sigma_n/\sqrt{\kappa_n}$.
Predictive. Given $\sigma^2$ and the data, $\tilde x = \mu + e$ with $\mu$ normal (mean $\mu_n$, variance $\sigma^2/\kappa_n$) and $e$ independent normal (mean 0, variance $\sigma^2$), so $\tilde x$ is normal with mean $\mu_n$ and variance $\sigma^2(1 + 1/\kappa_n) = \sigma^2/\kappa_n'$ with $\kappa_n' = \kappa_n/(1 + \kappa_n)$. That is the conditional distribution of $\mu$ with $\kappa_n'$ in place of $\kappa_n$, and the marginal computation above, repeated with $\kappa_n'$, gives the $t$ with scale $\sigma_n/\sqrt{\kappa_n'} = \sigma_n\sqrt{1 + 1/\kappa_n}$. $\square$
The posterior sum of squares $\nu_n\sigma_n^2$ adds three things: the prior's sum of squares, the sample's, and a term that grows with the disagreement between $\bar x$ and $\mu_0$. A prior that puts the mean in the wrong place therefore widens the posterior for $\mu$, which a known-variance analysis cannot do.
Corollary (the flat prior gives the t interval). Under the improper prior $\pi(\mu, \sigma^2) \propto 1/\sigma^2$, with $n \ge 2$ and $s^2 > 0$, the posterior of $\mu$ is Student $t$ with $n - 1$ degrees of freedom, location $\bar x$ and scale $s/\sqrt n$.
Proof. Likelihood times $1/\sigma^2$ is proportional to $(\sigma^2)^{-(n/2 + 1)}\exp\{-[(n-1)s^2 + n(\mu - \bar x)^2]/(2\sigma^2)\}$. Writing $(\sigma^2)^{-(n/2+1)} = (\sigma^2)^{-((n-1)/2 + 1)}(\sigma^2)^{-1/2}$, this is the normal–inverse-gamma form with $\kappa_n = n$, $\mu_n = \bar x$, $\nu_n = n - 1$ and $\nu_n\sigma_n^2 = (n - 1)s^2$, so $\sigma_n^2 = s^2$. The integrals in the proof of Theorem 7 are finite for $\nu_n \ge 1$ and $\nu_n\sigma_n^2 > 0$, so the posterior is proper, and its $\mu$-marginal is the stated $t$. $\square$
The prior $1/\sigma^2$ is not a density (its integral is infinite) and is discussed in Choosing a Prior; Gelman et al. (2013, §3.2) derive the same posterior. The consequence is that the central 95% posterior interval for $\mu$ is $\bar x \pm t_{n-1, 0.975}\,s/\sqrt n$, exactly the $t$ confidence interval of Confidence Intervals for a Mean. The two intervals have the same endpoints and different meanings, a point Credible Intervals and How They Differ from Confidence Intervals takes up.
Student's sleep data. The ten differences in hours of sleep between two drugs (The Paired-Samples t-Test) have $\bar x = 1.58$ and $s^2 = 1.5129$. Under the flat prior the posterior of the mean difference is $t$ with 9 degrees of freedom, location 1.58 and scale 0.3890, and its central 95% interval is $1.58 \pm 2.2622 \times 0.3890 = (0.700, 2.460)$, the paired $t$ interval. A skeptical prior centered on no effect, $\mu_0 = 0$ with $\kappa_0 = 1$, $\nu_0 = 1$ and $\sigma_0^2 = 1$, gives $\kappa_n = 11$, $\mu_n = 10 \times 1.58/11 = 1.4364$, $\nu_n = 11$ and
$$\nu_n\sigma_n^2 = 1 + 9 \times 1.5129 + \frac{10}{11}(1.58)^2 = 1 + 13.616 + 2.2695 = 16.885,$$
so $\sigma_n^2 = 1.5350$ and the scale of the $t$ for $\mu$ is $\sqrt{1.5350/11} = 0.3736$. The central 95% interval is $1.4364 \pm 2.2010 \times 0.3736 = (0.614, 2.259)$. It is pulled toward zero by one observation's worth of prior belief, and it is slightly narrower than the flat-prior interval: the prior's own sum of squares (1, on one degree of freedom) and the two extra degrees of freedom alone would lower $\sigma_n^2$ from 1.5129 to $(1 + 13.616)/11 = 1.3287$; the disagreement term raises it to 1.5350. The scale still shrinks, because $\kappa_n = 11$ instead of 10, and 11 degrees of freedom instead of 9 lower the $t$ quantile. Under either prior the posterior probability that the second drug adds sleep, $P(\mu > 0 \mid \mathbf{x})$, is 0.9986 to four decimals. A new patient's difference is predicted by a $t$ with scale $\sqrt{1.5350 \times 12/11} = 1.2941$, a central 95% range of $-1.41$ to $4.28$ hours; a single patient can easily sleep less on the second drug although the mean effect is almost certainly positive.
The Dirichlet–multinomial model
For data in $K$ categories with probabilities $\boldsymbol\theta = (\theta_1, \ldots, \theta_K)$, the conjugate prior is the Dirichlet distribution $\text{Dirichlet}(a_1, \ldots, a_K)$, with density proportional to $\prod_k\theta_k^{a_k - 1}$ on the simplex $\{\theta_k \ge 0, \sum_k\theta_k = 1\}$ and normalizing constant
$$Z(\mathbf{a}) = \frac{\prod_k\Gamma(a_k)}{\Gamma(A)}, \qquad A = \sum_k a_k$$
(Bishop, 2006, eq. 2.38), which Latent Dirichlet Allocation proves in its paragraph "Why the integral is $B(a)$". With $K = 2$ it is the beta distribution. Naive Bayes proves that counts $y_1, \ldots, y_K$ turn a $\text{Dirichlet}(\mathbf{a})$ prior into a $\text{Dirichlet}(a_1 + y_1, \ldots, a_K + y_K)$ posterior, and that the posterior mean of $\theta_k$ is $(a_k + y_k)/(A + n)$, which with every $a_k = 1$ is add-one smoothing. Gelman et al. (2013, §3.4) state the same. Here we add the posterior spread and the predictive distribution, both of which follow from the normalizing constant.
Theorem 8 (Dirichlet moments and the Pólya predictive). Let $\boldsymbol\theta \sim \text{Dirichlet}(\mathbf{a})$. For $s \ne t$,
$$E[\theta_t] = \frac{a_t}{A}, \qquad \operatorname{Var}(\theta_t) = \frac{a_t(A - a_t)}{A^2(A + 1)}, \qquad \operatorname{Cov}(\theta_s, \theta_t) = -\frac{a_sa_t}{A^2(A + 1)}.$$
If, given $\boldsymbol\theta$, $m$ further tokens are drawn independently from $\boldsymbol\theta$, the probability of a particular sequence of them with counts $y_1, \ldots, y_K$ is
$$\frac{Z(\mathbf{a} + \mathbf{y})}{Z(\mathbf{a})} = \frac{\prod_k a_k(a_k + 1)\cdots(a_k + y_k - 1)}{A(A + 1)\cdots(A + m - 1)},$$
where an empty product (for $y_k = 0$) is 1.
Proof. For any nonnegative integers $\mathbf{y}$, $E\big[\prod_k\theta_k^{y_k}\big] = \frac{1}{Z(\mathbf{a})}\int\prod_k\theta_k^{a_k + y_k - 1} = Z(\mathbf{a} + \mathbf{y})/Z(\mathbf{a})$, since the integral is the normalizing constant for parameters $\mathbf{a} + \mathbf{y}$. With $\Gamma(z + j) = z(z + 1)\cdots(z + j - 1)\Gamma(z)$, from $\Gamma(z+1) = z\Gamma(z)$ applied $j$ times, the ratio is the product of rising factorials stated. That is the sequence probability, because given $\boldsymbol\theta$ a sequence with counts $\mathbf{y}$ has probability $\prod_k\theta_k^{y_k}$. Taking $\mathbf{y}$ to be one token of $t$ gives $E[\theta_t] = a_t/A$. Two tokens of $t$ give $E[\theta_t^2] = a_t(a_t + 1)/[A(A + 1)]$, so
$$\operatorname{Var}(\theta_t) = \frac{a_t(a_t + 1)}{A(A + 1)} - \frac{a_t^2}{A^2} = \frac{a_t[A(a_t + 1) - a_t(A + 1)]}{A^2(A + 1)} = \frac{a_t(A - a_t)}{A^2(A + 1)}.$$
One token each of $s$ and $t$ gives $E[\theta_s\theta_t] = a_sa_t/[A(A + 1)]$, and subtracting $E[\theta_s]E[\theta_t]$ gives the covariance. $\square$
The covariances are negative because the probabilities must sum to 1: a larger share for one category leaves less for the others. The predictive formula has a mechanical reading, the Pólya urn: start with $a_k$ balls of color $k$; draw a ball, record its color, and return it with one more of the same color. The probability of each draw is the current share of its color, and multiplying the shares along the sequence gives the formula. Each observed token makes its own term more likely to be seen again, which is how the predictive distribution expresses uncertainty about $\boldsymbol\theta$.
The China class of Naive Bayes. The textbook example of Naive Bayes has eight training tokens in the class China over six terms, with counts $(5, 1, 1, 1, 0, 0)$ for Chinese, Beijing, Shanghai, Macao, Tokyo and Japan. Under the uniform prior the posterior is $\text{Dirichlet}(6, 2, 2, 2, 1, 1)$ with $A = 14$. The posterior mean of $\theta_{\textit{Chinese}}$ is $6/14 = 0.4286$, the add-one estimate, and its posterior standard deviation is $\sqrt{6 \times 8/(196 \times 15)} = 0.1278$: eight tokens leave the probability of the class's most common term uncertain by about 0.13 either way. The covariance of $\theta_{\textit{Chinese}}$ and $\theta_{\textit{Tokyo}}$ is $-6/2{,}940 = -1/490$. The probability that the next two tokens of the class are both Chinese is $\frac{6 \times 7}{14 \times 15} = 0.2$, not the square of the add-one estimate, $(6/14)^2 = 0.1837$. For two Tokyo tokens the gap is relatively larger: $\frac{1 \times 2}{14 \times 15} = 0.0095$ against $(1/14)^2 = 0.0051$. The Naive Bayes classifier multiplies smoothed probabilities as if each token were independent with known $\boldsymbol\theta$; the predictive distribution says that a repeated rare word is more probable than that, because the first occurrence is evidence that the word is less rare than estimated. Latent Dirichlet Allocation builds its generative model from Dirichlet priors of exactly this kind, and its collapsed Gibbs sampler rests on the same integral, $Z(\mathbf{a} + \mathbf{y})/Z(\mathbf{a})$.
Beyond conjugacy
Conjugacy is a convenience, not a requirement. Gelman et al. (2013, §2.4) put it this way: nonconjugate priors make interpretation less transparent and computation harder, but pose no new conceptual problem. Any prior and likelihood define a posterior by (1). With one parameter it can be computed on a grid: evaluate $L(\theta)\pi(\theta)$ at many points, and sum.
Suppose the analyst of the hypertension example expressed the belief "around 15%" on the log odds scale instead, as a normal prior for $\varphi = \ln\frac{p}{1-p}$ with mean $\ln(0.15/0.85) = -1.7346$ and standard deviation 1. No standard family is the posterior. On a grid of 200,001 points for $\varphi$ from $-12$ to 8, the posterior mean of $p$ is 0.2367, the median 0.2326, and the central 95% interval 0.1272 to 0.3693. The prior's own mean for $p$ is 0.1885, not 0.15, because the transformation back from $\varphi$ is not linear. The answer is close to the $\text{Beta}(3, 17)$ analysis and somewhat closer to the data, because this prior is both less concentrated (standard deviation 1 on the log odds scale, against 0.675 for $\text{Beta}(3, 17)$, whose log odds have variance $\psi'(3) + \psi'(17)$, with $\psi'$ the trigamma function) and centered higher (its mean for $p$ is 0.1885, not 0.15). For several parameters a grid becomes impossible, and the posterior is explored by simulation, the subject of Markov Chain Monte Carlo, or approximated within a tractable family, the subject of Variational Inference.
What is proved, what is cited, what the experiment shows
Proved on this page: Bayes' theorem for a parameter, equation (1), which is the definition of a conditional density; the conjugate prior of an exponential family with a natural parameter of any dimension (Theorem 1); for one parameter, the posterior mean of the mean parameter as a weighted average of the prior mean and the average $\bar T$, under a condition at the ends of the parameter space (Theorem 2); sequential updating (Theorem 3); the beta-binomial predictive distribution with its mean and variance (Theorem 4); gamma–Poisson updating and the negative binomial predictive (Theorem 5); normal–normal updating and its predictive (Theorem 6); normal–inverse-gamma updating, the marginal posteriors of $\sigma^2$ and $\mu$, the $t$ predictive, and the flat-prior corollary that reproduces the $t$ interval (Theorem 7); the Dirichlet moments and the Pólya predictive (Theorem 8); the identity (4).
Used from other pages: the natural form of an exponential family, $\mu = A'(\theta)$ increasing, and the maximum likelihood estimate $\bar T$ when it exists (Exponential Families); the negative binomial as a gamma mixture of Poisson counts (Overdispersion in Count Data, also proved here for the predictive); the Dirichlet normalizing constant (Latent Dirichlet Allocation); the tower property and the law of total variance (Conditional Expectation); the binomial integral $1/(n + 1)$ (The Likelihood Function); the beta mean (Probability for Information Retrieval); the Dirichlet posterior and its mean (Naive Bayes); the $t$ density (Chi-Squared and t Distributions); sums of independent normal variables (Moment Generating and Characteristic Functions).
Cited: the beta and Dirichlet normalizing constants in terms of the gamma function (Bishop, 2006, eqs. 2.13 and 2.38; proved on Latent Dirichlet Allocation); that exponential families are, apart from irregular cases, the only families with natural conjugate priors (Gelman et al., 2013, §2.4); the characterization of conjugate priors by linear posterior expectation (Diaconis and Ylvisaker, 1979, from the abstract; only one direction is proved here).
Computed on data: the posteriors and predictive distributions for the NHANES hypertension sample, Iowa's weekly cancer deaths of 2014 (and the 2015 weeks against the predictive interval), a sample of 25 NHANES systolic pressures (National Center for Health Statistics, 2020; the weekly deaths are National Center for Health Statistics, 2021), Student's sleep data (Student, 1908), the China class, and Bortkiewicz's horse kicks (Bortkiewicz, 1898). The priors other than the uniform, Jeffreys and flat ones are illustrations chosen for this page, and the comparison of 2015 with the predictive interval is one year's observation, not a test. The NHANES adults are treated as a finite population, as elsewhere in the collection, and the survey weights are ignored. Every number is computed and checked against this page by tools/experiments/statistics/conjugate-priors.py.
Worked problems
The problems were written for this article, on its running data and on small exact cases. Each number is recomputed by the article's script.
Problem 1 (updating in two batches). The 40 young adults were drawn one at a time; the first 20 included 7 with hypertension and the last 20 included 3. Starting from the uniform prior, update with the first batch and then the second, and then in the reverse order. Compare the posterior means after the first batch in each order with the final one.
Solution. By (3) each batch adds its hypertensive count to $a$ and the rest to $b$. First batch first: $\text{Beta}(1, 1) \to \text{Beta}(8, 14) \to \text{Beta}(11, 31)$, with posterior means $8/22 = 0.3636$ and then $11/42 = 0.2619$. Reverse order: $\text{Beta}(1, 1) \to \text{Beta}(4, 18) \to \text{Beta}(11, 31)$, with means $4/22 = 0.1818$ and then $0.2619$. The intermediate beliefs differ widely, 0.36 against 0.18, because each batch of 20 is noisy. The final posterior is the same, as Theorem 3 requires: the order in which exchangeable observations arrive carries no information about $p$.
Problem 2 (a Poisson rate for horse kicks). Bortkiewicz's data, used in Maximum Likelihood Estimation, record 122 deaths by horse kick in 200 corps-years. With a $\text{Gamma}(1, 1)$ prior on the yearly rate $\lambda$ per corps, find the posterior, its mean and central 95% interval, and the predictive probabilities that a new corps-year has no death and exactly one death. Compare with the observed shares, 109/200 and 65/200.
Solution. By Theorem 5 the posterior is $\text{Gamma}(1 + 122,\ 1 + 200) = \text{Gamma}(123, 201)$, with mean $123/201 = 0.6119$ (the maximum likelihood estimate is 0.61; the prior adds one death in one corps-year), standard deviation $\sqrt{123}/201 = 0.0552$, and central 95% interval 0.5086 to 0.7247. The predictive distribution is negative binomial: $P(Y = 0) = (201/202)^{123} = 0.5431$ and $P(Y = 1) = 123 \times (201/202)^{123}/202 = 0.3307$. The observed shares are 0.545 and 0.325. The plug-in Poisson values with $\lambda = 0.61$ are $e^{-0.61} = 0.5434$ and $0.61e^{-0.61} = 0.3314$; with 200 corps-years the posterior is narrow enough that the predictive and plug-in answers agree to three decimals.
Problem 3 (how many observations for a target precision). In the blood pressure example ($\sigma^2 = 405.01$, prior standard deviation $\tau_0 = 10$), how large must the sample be for the posterior standard deviation of $\mu$ to be at most 2 mm Hg? Does the answer depend on the data?
Solution. By Theorem 6, $1/\tau_n^2 = 1/100 + n/405.01$, and $\tau_n \le 2$ means $1/\tau_n^2 \ge 1/4$. So $n \ge (0.25 - 0.01) \times 405.01 = 97.2$, and 98 observations are needed. The answer does not depend on the data: with known variance the posterior variance depends only on $n$, $\sigma^2$ and $\tau_0^2$. Without the prior ($\tau_0 \to \infty$) the requirement would be $n \ge 405.01/4 = 101.25$, so 102; the prior is worth about 4 observations, as computed above.
Problem 4 (a Pólya urn). Under the posterior $\text{Dirichlet}(6, 2, 2, 2, 1, 1)$ of the China class, find the probability that the next two tokens are Chinese then Tokyo, and compare with the product of the add-one estimates.
Solution. By Theorem 8 with $y_{\textit{Chinese}} = y_{\textit{Tokyo}} = 1$ and $m = 2$, the probability is $\frac{6 \times 1}{14 \times 15} = 1/35 = 0.0286$. The product of the add-one estimates is $\frac{6}{14}\times\frac{1}{14} = 0.0306$. Here the predictive probability is smaller, because $\operatorname{Cov}(\theta_{\textit{Chinese}}, \theta_{\textit{Tokyo}}) < 0$: $E[\theta_s\theta_t] = E[\theta_s]E[\theta_t] + \operatorname{Cov}(\theta_s, \theta_t) = \frac{6}{14}\cdot\frac{1}{14} - \frac{1}{490}$. A repeated term is more probable than the product suggests, and a pair of different terms less probable.
Problem 5 (a posterior more spread out than its prior). Take the prior $\text{Beta}(1, 9)$ for a proportion and observe one trial. Show that a success leaves the posterior more spread out than the prior, and that the posterior variance is nevertheless smaller than the prior variance on average over the possible outcomes.
Solution. The prior variance is $\frac{1 \times 9}{10^2 \times 11} = 9/1{,}100 = 0.00818$. After a success the posterior is $\text{Beta}(2, 9)$ with variance $\frac{2 \times 9}{11^2 \times 12} = 3/242 = 0.01240$, larger than the prior's: the prior said the proportion was near 0.1, and the success says it may be larger, so the posterior spreads toward larger values. After a failure it is $\text{Beta}(1, 10)$ with variance $\frac{10}{121 \times 12} = 0.00689$. The prior probability of a success is the prior mean, 0.1, so the expected posterior variance is $0.9 \times 0.00689 + 0.1 \times 0.01240 = 9/1{,}210 = 0.00744$, below 0.00818. This is the law of total variance, $\operatorname{Var}(p) = E[\operatorname{Var}(p \mid x)] + \operatorname{Var}(E[p \mid x])$, which guarantees only that the posterior variance is smaller on average (Gelman et al., 2013, §2.2). A single surprising observation can increase uncertainty.
Summary
- Bayes' theorem for a parameter: $\pi(\theta \mid \mathbf{x}) = f(\mathbf{x} \mid \theta)\pi(\theta)/m(\mathbf{x})$, with the marginal likelihood $m(\mathbf{x}) = \int f(\mathbf{x} \mid \theta)\pi(\theta)\,d\theta$; the posterior is proportional to likelihood times prior.
- Every exponential family $h(x)e^{\boldsymbol\theta^{\mathsf T}\mathbf{T}(x) - A(\boldsymbol\theta)}$ has the conjugate prior $\propto e^{\boldsymbol\chi^{\mathsf T}\boldsymbol\theta - \nu A(\boldsymbol\theta)}$; data add $\sum \mathbf{T}(x_i)$ to $\boldsymbol\chi$ and $n$ to $\nu$ (Theorem 1).
- For one parameter, when the prior and posterior densities vanish at the ends of $\Theta$, the posterior mean of the mean parameter is $w\cdot\chi/\nu + (1 - w)\bar T$ with $w = \nu/(\nu + n)$: the prior counts as $\nu$ observations (Theorem 2).
- Beta–binomial: $\text{Beta}(a + x, b + n - x)$; predictive beta-binomial with variance inflated by $(a + b + m)/(a + b + 1)$ (Theorem 4).
- Gamma–Poisson: $\text{Gamma}(a + \sum x_i, b + n)$; predictive negative binomial with mean $a/b$ and variance $a/b + a/b^2$ (Theorem 5).
- Normal, known variance: precisions add, the posterior mean is precision-weighted, and the predictive variance is $\sigma^2 + \tau_n^2$ (Theorem 6).
- Normal, unknown variance: normal–inverse-gamma; $\mu$ is $t$ with $\nu_n$ degrees of freedom; the flat prior $1/\sigma^2$ gives the $t$ interval exactly (Theorem 7).
- Dirichlet–multinomial: moments $a_t/A$, $a_t(A - a_t)/[A^2(A+1)]$, negative covariances, and the Pólya predictive (Theorem 8).
- Updating in batches equals updating at once (Theorem 3); without conjugacy the posterior is computed numerically.
What we have learned
A prior distribution turns a likelihood into probabilities for the parameter and for future data. For exponential families there is a prior that keeps the algebra closed, and with it every posterior is the prior with counts added: the prior behaves like data, its weight is a number of observations, and the posterior mean lies between the prior's guess and the maximum likelihood estimate (for the standard families of this page, by Theorem 2). Predictive distributions are wider than plug-in ones by exactly the posterior's uncertainty about the parameter, which matters for small samples and rare categories and fades as data accumulate.
None of this says how to read the probability that a posterior interval contains the parameter, or how often such intervals contain it when the data are drawn again and again. Credible Intervals and How They Differ from Confidence Intervals answers both questions on the hypertension sample.
References
- Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer. Section 2.1.1, "The beta distribution" (eq. 2.13), and Section 2.2.1, "The Dirichlet distribution" (eq. 2.38). record · 10 pages
- Bortkiewicz, L. von (1898). Das Gesetz der kleinen Zahlen. B. G. Teubner, Leipzig. §12 (deaths from horse kicks in the Prussian army), as tabulated in The Poisson Distribution. record · 5 pages
- Diaconis, P., and Ylvisaker, D. (1979). Conjugate priors for exponential families. Annals of Statistics, 7(2), 269–281. https://doi.org/10.1214/aos/1176344611 record
- Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., and Rubin, D. B. (2013). Bayesian Data Analysis (3rd ed.). CRC Press. Sections 2.1 to 2.6 and 3.2 to 3.4. Authors' electronic edition, http://www.stat.columbia.edu/~gelman/book/ record · 7 pages
- National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018: DEMO_J, BPX_J and BPQ_J data files. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 95 pages
- National Center for Health Statistics (2021). Weekly Counts of Deaths by State and Select Causes, 2014–2019 (dataset 3yf8-kanr). Centers for Disease Control and Prevention. https://data.cdc.gov/d/3yf8-kanr record · 13 pages
- Student (1908). The probable error of a mean. Biometrika, 6(1), 1–25. record · 17 pages