The Likelihood and Estimation unit treated a parameter as a fixed unknown number and judged procedures by how they behave over repeated samples. Bayesian inference treats the parameter as a random quantity. A prior distribution describes it before the data; Bayes' theorem combines the prior with the likelihood into a posterior distribution, the distribution of the parameter given the data; and every question, whether about the parameter, about future data, or about which of two models to prefer, is answered by a probability computed from that posterior. The rule is the one that Predictive Values and Bayes' Theorem applies to a diagnosis, extended from events to parameters that take a continuum of values.
The course pages already use Bayesian ideas without the name. Naive Bayes derives add-one smoothing as the mean of a posterior distribution, Language Models for Retrieval smooths each document toward the collection with a Dirichlet prior, Latent Dirichlet Allocation is a Bayesian model fitted by a Markov chain, and Estimating Rare Event Rates predicts events in deployment from a posterior. This unit is the theory beneath them.
What it assumes. The Likelihood and Estimation unit, especially The Likelihood Function, Maximum Likelihood Estimation and Fisher Information and the Cramér–Rao Bound; Exponential Families for the general form of conjugate priors; conditional expectation and Jensen's inequality from Probability Theory; and, for the last article, the Kullback–Leibler divergence of Information Theory. The course pages Confidence Intervals for a Proportion and Confidence Intervals for a Mean supply the intervals that credible intervals are compared with.
What is proved and what is cited. Proved across the unit: Bayes' theorem for densities; the conjugate prior of every exponential family; for one parameter, under a condition at the ends of the parameter space, the posterior mean of the mean parameter as a weighted average of the prior mean and the sample average of the sufficient statistic (the maximum likelihood estimate when one exists); the beta–binomial, gamma–Poisson, normal, normal–inverse-gamma and Dirichlet–multinomial posteriors with their predictive distributions; the shortest-length property of highest posterior density regions; the exact coverage of a credible interval on average over its prior, and the consequence that, under a proper prior with a positive density, a credible interval for a binomial proportion falls below its level on a set of parameter values of positive length; the non-invariance of flat priors and the invariance of Jeffreys' prior, with the Jeffreys priors of the standard models; the Bayes factor as the factor that turns prior odds into posterior odds, Lindley's paradox, and the largest Bayes factor a centered normal prior can give; the decomposition of Laplace's approximation into the Bayesian information criterion and a term of order 1, given the approximation itself; posterior means in the normal hierarchical model, Stein's lemma and the James–Stein theorem; that detailed balance makes a distribution stationary, and that Metropolis–Hastings and Gibbs updates satisfy it; the evidence lower bound and the optimal mean-field factors. Laplace's approximation itself is derived by a Taylor expansion, not proved. Cited rather than proved: the error terms of Laplace's approximation (Kass and Raftery, 1995, as cited in Bayes Factors and Posterior Model Probabilities); the Bernstein–von Mises theorem, which makes posteriors approximately normal in large samples; the convergence of Markov chain Monte Carlo, and the central limit theorems for Markov chains, which need more probability theory than the collection develops.
Notation. The model for the data is written with a bar, $f(\mathbf{x} \mid \theta)$, because the parameter is now random (the Likelihood unit wrote $f(\mathbf{x}; \theta)$); read as a function of $\theta$ it is the likelihood $L(\theta)$. The prior density is $\pi(\theta)$, the posterior $\pi(\theta \mid \mathbf{x})$, the marginal likelihood $m(\mathbf{x}) = \int f(\mathbf{x} \mid \theta)\pi(\theta)\,d\theta$, and a predictive distribution $f(\tilde x \mid \mathbf{x})$; for two models $M_0$ and $M_1$ the marginal likelihoods are $m_0$ and $m_1$, and the Bayes factor $B_{10} = m_1/m_0$. Families are written $\text{Beta}(a, b)$, $\text{Gamma}(a, b)$ with shape $a$ and rate $b$, $\text{Dirichlet}(a_1, \ldots, a_K)$ with $A = \sum_k a_k$, and $\text{IG}(a, b)$ for the inverse gamma; a normal distribution is described in words, or as $N(\mu, \sigma)$ with the standard deviation second, as elsewhere on the site. A normal prior on a mean has mean $\mu_0$ and variance $\tau_0^2$, and the normal–inverse-gamma prior has parameters $\mu_0, \kappa_0, \nu_0, \sigma_0^2$, updated to $\mu_n, \kappa_n, \nu_n, \sigma_n^2$. Credible intervals have level $1 - \alpha$. Natural logarithms are written $\ln$.
Data. The running sample is the one of Confidence Intervals for a Proportion: 40 adults aged 18 to 39 from the National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018, of whom 10 have hypertension, drawn from 1,696 young adults with a prevalence of 389/1,696 = 0.2294. Other articles use Student's ten sleep differences, the eight training tokens of the China class of Naive Bayes, Bortkiewicz's 200 corps-years of deaths by horse kick, the NHANES systolic pressures (a sample of 25 for the normal models), diabetes on age and obesity among the 5,107 NHANES adults of Generalized Linear Models (for Markov chain Monte Carlo and variational inference), hypertension in NHANES cells defined by age, sex and race or ethnicity, weekly counts of deaths by state from the National Center for Health Statistics, national deaths from CDC WONDER, and the 1,400 abstracts of the Cranfield collection. Every number is computed by a script in tools/experiments/statistics/.
Articles
Priors and posteriors
- Priors, Posteriors and Conjugate Families: Bayes' theorem for a parameter; the conjugate prior of every exponential family, with the posterior mean of the mean parameter proved, for one parameter and under a condition at the ends of the parameter space, to be a weighted average of the prior mean and the maximum likelihood estimate; beta–binomial, gamma–Poisson, normal–normal, normal–inverse-gamma and Dirichlet–multinomial updating with their predictive distributions; and a nonconjugate posterior on a grid; worked on hypertension, Iowa's weekly cancer deaths, blood pressure and Student's sleep data.
- Credible Intervals and How They Differ from Confidence Intervals: equal-tailed and highest posterior density intervals, the second proved shortest and the first proved to survive a change of scale; exact coverage on average over the prior, and why a credible interval for a count must undercover somewhere; the Jeffreys interval for a proportion against the Wald, Wilson, Agresti–Coull and Clopper–Pearson intervals by exact coverage; and the case where the credible and confidence intervals for a normal mean coincide.
- Choosing a Prior: why a flat prior is flat on one scale only; improper priors and when their posteriors exist; Jeffreys' prior derived from the Fisher information and proved invariant, for the standard models, with a case where the joint rule undercovers; weakly informative priors; and the smoothing methods of text retrieval as Dirichlet priors, with the strength of the prior chosen on Cranfield by marginal likelihood and by held-out prediction.
Comparing models and pooling information
- Bayes Factors and Posterior Model Probabilities: the marginal likelihood as a model's prior predictive probability, derived in closed form for the beta-binomial, gamma-Poisson and normal models; the Bayes factor as the factor that turns prior odds into posterior odds and as a score of sequential forecasts; Lindley's paradox and the largest Bayes factor any centered normal prior can give at a given $z$; Laplace's approximation and the Bayesian information criterion derived from it, with its error measured; and the dependence of Bayes factors on the priors for the parameters; worked on hypertension by sex and on U.S. deaths.
- Hierarchical Models and Shrinkage: partial pooling in the normal hierarchical model; empirical Bayes estimation of the prior from many units; Tweedie's formula and the winner's curse; the gamma–Poisson model for state cancer death rates from short windows of weekly counts; hypertension in NHANES cells; and Stein's lemma and the James–Stein theorem, proved for shrinkage toward any subspace.
Computing posteriors
- Markov Chain Monte Carlo: Markov chains with a stationary distribution, detailed balance, and proofs that Metropolis–Hastings and Gibbs sampling leave the posterior stationary; convergence and the Markov chain central limit theorem cited; warm-up, effective sample size and the split $\hat R$ diagnostic; a Gibbs sampler checked against an exact posterior, Bayesian logistic regression of diabetes against a grid and the Laplace approximation, and the efficiency of random-walk Metropolis as its step size varies.
- Variational Inference: the evidence lower bound derived from Jensen's inequality and as the log evidence minus a Kullback–Leibler divergence; the optimal mean-field factors and coordinate ascent; the normal model with unknown mean and precision solved in closed form, with the variance it underestimates; and the mode-seeking behavior of the reverse divergence, on a two-mode target and on logistic regression.
Where the unit leads
The unit uses the language of Latent Dirichlet Allocation and Variational Inference for LDA, which fit a Bayesian model of text by the two computational methods of the last two articles. Bayesian readings return across the collection: the Bayesian information criterion in Prediction Error and Model Choice, ridge and lasso as posterior modes in Ridge and Lasso Regression, and the posterior probability of a null hypothesis behind q-Values and the Share of True Nulls. The next unit of the collection is Multivariate Statistics.
References
Each article lists its own references. The textbooks on which the unit draws throughout: record
- Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer. Chapters 2 and 10. record · 10 pages
- Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., and Rubin, D. B. (2013). Bayesian Data Analysis (3rd ed.). CRC Press. Authors' electronic edition, http://www.stat.columbia.edu/~gelman/book/ record · 7 pages
- Kass, R. E., and Raftery, A. E. (1995). Bayes factors. Journal of the American Statistical Association, 90(430), 773–795. https://doi.org/10.1080/01621459.1995.10476572 (cited for the error of Laplace's approximation, through Bayes Factors and Posterior Model Probabilities). record · 3 pages