Credible Intervals and How They Differ from Confidence Intervals

What a Bayesian posterior interval for a parameter claims, and how often it contains the true value when the data are drawn again. Equal-tailed and highest posterior density intervals are defined; the highest density region is proved to be the shortest set of its probability, and only the equal-tailed interval is proved to survive a change of scale. A credible interval is proved to contain the parameter with exactly its stated probability on average over its prior, which forces its coverage below the stated level for some parameter values when the data are counts. The Jeffreys interval for a proportion is then compared, by exact coverage, with the Wald, Wilson, Agresti–Coull and Clopper–Pearson intervals on NHANES hypertension, and the flat-prior interval for a normal mean is shown to coincide with the t interval. Worked problems on a sample with no events, samples of five, the probability a different prior gives to the same interval, and the coverage of an interval built on an informative prior.

The 40 young adults of the running sample include 10 with hypertension. Under Jeffreys' prior the posterior distribution of the proportion $p$ is $\text{Beta}(10.5, 30.5)$, and the central 95% of it lies between 0.1364 and 0.3983. Confidence Intervals for a Proportion gave Wilson's 95% confidence interval for the same sample as 0.1419 to 0.4019, nearly the same numbers. The two intervals make different claims. One says that the probability that $p$ lies in it, given these data, is 0.95. The other says that a method which produces such intervals contains the true value in about 95% of samples. Which claims hold, under what conditions, and how often does the Bayesian interval contain the truth when it is judged as a confidence interval?

An interval with posterior probability $1 - \alpha$ is a $100(1 - \alpha)\%$ credible interval (Gelman et al., 2013, §2.3, call it a posterior interval). The coverage probability of an interval method, as on the course page, is the probability over repeated samples, with the parameter held fixed, that the interval contains the parameter. A credible interval has a coverage probability like any other method, and this article computes it.

First we define the two standard credible intervals, the equal-tailed and the highest posterior density interval, and prove what distinguishes them. Then we prove the one coverage guarantee a credible interval carries: exact coverage on average over its prior. Next we compute the exact coverage of the Jeffreys interval for a proportion and set it beside the four intervals of the course. Finally we find a case where a credible interval and a confidence interval coincide exactly, and say what each claims. The article assumes Priors, Posteriors and Conjugate Families and Confidence Intervals for a Proportion; Jeffreys' prior itself is derived in Choosing a Prior.

Two ways to choose a credible interval

Many intervals hold 95% of a posterior distribution. Two rules pick one.

The equal-tailed (or central) interval runs from the $\alpha/2$ quantile of the posterior to the $1 - \alpha/2$ quantile, leaving probability $\alpha/2$ in each tail. Gelman et al. (2013, §2.3) use it by default, because it is read directly from posterior quantiles or simulations. For the hypertension sample under Jeffreys' prior it is $(0.1364,\ 0.3983)$, of width 0.2619.

The highest posterior density (HPD) region is the set $\{\theta : \pi(\theta \mid \mathbf{x}) \ge k\}$, with $k$ chosen so that the set has posterior probability $1 - \alpha$: the density inside is never lower than outside (Gelman et al., 2013, §2.3). For a posterior that rises to a single mode and then falls, the set is an interval whose two ends have equal density. For $\text{Beta}(10.5, 30.5)$ it is $(0.1294,\ 0.3894)$, where the density is 0.917 at both ends, and its width is 0.2600, slightly less than the equal-tailed interval's. The posterior is skewed to the right, so the HPD interval shifts left, toward the mode, to collect the same probability in less length.

Theorem 1 (the HPD region is the shortest). Let $\pi(\theta \mid \mathbf{x})$ be a posterior density on the real line, and let $H = \{\theta : \pi(\theta \mid \mathbf{x}) \ge k\}$ with $k > 0$ have posterior probability $1 - \alpha$. Every set $S$ with posterior probability at least $1 - \alpha$ has length at least that of $H$.

Proof. Write $\pi$ for the posterior density and $\lvert\cdot\rvert$ for length. Split $S$ and $H$ into their common part and the rest: $S \setminus H$ and $H \setminus S$. Since $P(S) \ge P(H)$, removing the common part gives $P(S \setminus H) \ge P(H \setminus S)$. On $S \setminus H$, outside $H$, the density is below $k$, so $P(S \setminus H) \le k\lvert S \setminus H\rvert$. On $H \setminus S$, inside $H$, the density is at least $k$, so $P(H \setminus S) \ge k\lvert H \setminus S\rvert$. Together, $k\lvert S \setminus H\rvert \ge P(S \setminus H) \ge P(H \setminus S) \ge k\lvert H \setminus S\rvert$, and dividing by $k$ and adding $\lvert S \cap H\rvert$ to both sides gives $\lvert S\rvert \ge \lvert H\rvert$. $\square$

So the HPD interval is the shortest interval with its posterior probability. It has a drawback that the equal-tailed interval does not, which shows when the parameter is described on another scale.

Theorem 2 (equal-tailed intervals survive a change of scale; HPD intervals do not). Let $g$ be continuous and strictly increasing, and let the posterior distribution function of $\theta$ be continuous and strictly increasing on the parameter space. If $(l, u)$ is the equal-tailed interval for $\theta$, then $(g(l), g(u))$ is the equal-tailed interval for $g(\theta)$. For a strictly decreasing $g$ the same holds with the ends exchanged. No such statement holds for HPD intervals.

Proof. For increasing $g$, $P(g(\theta) \le g(l) \mid \mathbf{x}) = P(\theta \le l \mid \mathbf{x}) = \alpha/2$, so $g(l)$ is the $\alpha/2$ quantile of $g(\theta)$; likewise $g(u)$ is its $1 - \alpha/2$ quantile. For decreasing $g$, $P(g(\theta) \le g(u) \mid \mathbf{x}) = P(\theta \ge u \mid \mathbf{x}) = \alpha/2$, so $g(u)$ becomes the lower end. For the HPD interval the example below is a counterexample. $\square$

Example (the HPD interval on the log odds scale). The same posterior, $\text{Beta}(10.5, 30.5)$ for $p$, is a posterior for the log odds $\varphi = \ln\frac{p}{1 - p}$, with density proportional to $e^{10.5\varphi}/(1 + e^\varphi)^{41}$ by the change-of-variables rule. Its 95% HPD interval for $\varphi$ is $(-1.8228,\ -0.3922)$, which maps back to $p$ in $(0.1391,\ 0.4032)$. That is not the HPD interval for $p$, $(0.1294, 0.3894)$. The equal-tailed interval for $\varphi$ maps back exactly to $(0.1364, 0.3983)$. An analyst who reports the HPD interval must therefore also say on which scale the density was highest; with the equal-tailed interval the scale does not matter. The change-of-variables factor $dp/d\varphi = p(1 - p)$ is what moves the highest density, the same factor that The Likelihood Function showed changes areas under a likelihood.

What a credible interval guarantees

A confidence interval method promises coverage at every fixed value of the parameter, at least approximately. A credible interval is built from the posterior and promises something else, which can be proved in one line.

Theorem 3 (coverage on average over the prior). Let the prior $\pi$ be a proper distribution, and for every possible data set $\mathbf{x}$ let $C(\mathbf{x})$ be a set with $P(\theta \in C(\mathbf{x}) \mid \mathbf{x}) = 1 - \alpha$ under the posterior. If $\theta$ is drawn from $\pi$ and then $\mathbf{X}$ from $f(\mathbf{x} \mid \theta)$, then $P(\theta \in C(\mathbf{X})) = 1 - \alpha$. Equivalently, the coverage function $c(\theta) = P(\theta \in C(\mathbf{X}) \mid \theta)$ satisfies $\int c(\theta)\,\pi(\theta)\,d\theta = 1 - \alpha$.

Proof. By the tower property (Conditional Expectation), $P(\theta \in C(\mathbf{X})) = E\big[P(\theta \in C(\mathbf{X}) \mid \mathbf{X})\big] = E[1 - \alpha] = 1 - \alpha$, the inner probability being the posterior probability of $C(\mathbf{X})$. Conditioning on $\theta$ instead, the same probability is $E[c(\theta)] = \int c(\theta)\pi(\theta)\,d\theta$. $\square$

The theorem is exact, holds for every sample size, and needs no approximation. It says that the 95% is a promise about the parameter values that the prior considers likely, weighted as the prior weighs them, not a promise at each value. The hypothesis itself is the stronger, conditional statement: among all draws that produced a particular data set, the parameter lies in that data set's interval with probability $1 - \alpha$.

Corollary (a credible interval for a count must undercover somewhere). Let $X$ be binomial with $n$ trials and let $C(x)$, $x = 0, \ldots, n$, be credible intervals of level $1 - \alpha$, with $0 < \alpha < 1$, under a proper prior with a positive density on $(0, 1)$. Then the coverage $c(p)$ is below $1 - \alpha$ on a set of values of $p$ of positive length.

Proof. If $c(p) \ge 1 - \alpha$ for all $p$ outside a set of zero length, then $\int c\,\pi = 1 - \alpha$ forces $c(p) = 1 - \alpha$ for all $p$ outside a set of zero length. The coverage is $c(p) = \sum_{x \in S(p)}\binom{n}{x}p^x(1-p)^{n-x}$, where $S(p)$ is the set of counts whose intervals contain $p$. The finitely many interval endpoints split $(0, 1)$ into finitely many open pieces, and on each piece $S(p)$ is fixed, so $c$ is a polynomial there. A polynomial equal to $1 - \alpha$ on an interval equals $1 - \alpha$ everywhere, including at $p = 0$, where a sum of binomial probabilities over a fixed set of counts equals 1 or 0. Since $0 < 1 - \alpha < 1$, this is impossible. $\square$

So under a proper prior with a positive density, no credible interval for a proportion is a confidence interval in the strict sense of coverage at least $1 - \alpha$ at every $p$. The Clopper–Pearson interval, which does achieve that, is therefore not a credible interval for any such prior. Its average coverage under any such prior exceeds the nominal level strictly: its coverage is at least $1 - \alpha$ everywhere and, by the proof, cannot equal $1 - \alpha$ outside a set of zero length, so it is larger on a set of positive length. That matches the course page, where its coverage averaged 0.971 over $p$ at $n = 40$.

Example (checking Theorem 3 by simulation). Draw $p$ from $\text{Beta}(\tfrac12, \tfrac12)$, then a count from $\text{Binomial}(40, p)$, and record whether the plain equal-tailed interval of the posterior $\text{Beta}(x + \tfrac12, 40 - x + \tfrac12)$ (its 0.025 and 0.975 quantiles, with no change at $x = 0$ or $x = 40$) contains $p$. In 30 runs of 100,000 draws (seeds 1 to 30), the share of intervals containing $p$ had median 0.9499 (interquartile range 0.9496 to 0.9504). Among the draws that produced exactly 10 successes, the median share was 0.9480 (interquartile range 0.9459 to 0.9528, from about 1,800 such draws per run, so each run's share has a standard error near 0.005). Integrating the exact coverage function against the prior density gives 0.9500 to within $10^{-12}$, as the theorem requires.

The Jeffreys interval for a proportion

With Jeffreys' prior $\text{Beta}(\tfrac12, \tfrac12)$ and $x$ successes in $n$ trials, the posterior is $\text{Beta}(x + \tfrac12,\ n - x + \tfrac12)$ (Priors, Posteriors and Conjugate Families, equation 3). Brown, Cai and DasGupta (2001, §3.1.3) define the 95% Jeffreys interval as its equal-tailed interval, the 0.025 and 0.975 quantiles of that beta distribution, with one change at the extremes: the lower limit is set to 0 when $x = 0$ and the upper limit to 1 when $x = n$. Without that change the interval for $x = 0$ would exclude the values of $p$ below its tiny lower limit, and the coverage would tend to 0 as $p$ tends to 0, the failure of the Wald interval near the boundary. They recommend it, with Wilson's, for samples of 40 or fewer (their abstract and §1). With the change at $x = 0$ and $x = n$ the interval is no longer exactly credible for those two counts (its posterior probability there is 0.975). The proof of Theorem 3 then gives its prior-averaged coverage as $1 - \alpha + (\alpha/2)[m(0) + m(n)]$, where $m(x)$ is the prior predictive probability of the count $x$: slightly above $1 - \alpha$, and 0.9544 at $n = 40$, where $m(0) = m(40) = 0.0889$. The corollary no longer applies, although the tables below show that the interval still undercovers at many values of $p$. The Python package statsmodels computes the unmodified equal-tailed interval, without the change at $x = 0$ and $x = n$, as proportion_confint(..., method="jeffreys") (Inference for Proportions and Means).

For the sample of 40 the interval is $(0.1364, 0.3983)$. The first table repeats the course page's comparison with the Jeffreys interval added, and the HPD interval of the same posterior. The other two tables also include the modification of Brown, Cai and DasGupta (2001, §4.1.2), which raises the upper limit for $x = 0$ to the Clopper–Pearson value $1 - (\alpha/2)^{1/n}$, sets the lower limit for $x = 1$ to 0, and mirrors both at $x = n$; for 10 of 40 it gives the same interval as the unmodified Jeffreys interval.

Interval Lower Upper Width Coverage at $p = 0.2294$, $n = 40$ Expected length there
Wald 0.1158 0.3842 0.2684 0.9109 0.2557
Wilson 0.1419 0.4019 0.2601 0.9628 0.2495
Agresti–Coull 0.1402 0.4036 0.2634 0.9628 0.2540
Clopper–Pearson 0.1269 0.4120 0.2850 0.9780 0.2734
Jeffreys 0.1364 0.3983 0.2619 0.9414 0.2502
Jeffreys, HPD 0.1294 0.3894 0.2600 0.9414 0.2476

At this one value of $p$ the Jeffreys interval covers less often than Wilson's, 0.9414 against 0.9628, and is about as long. One value of $p$ says little, because every one of these coverage functions jumps as $p$ crosses an endpoint. The next two tables use the course page's summaries: first every sample size from 10 to 200 at $p = 0.2294$, then 2,000 values of $p$ at $n = 40$.

Interval Lowest coverage, $10 \le n \le 200$ Mean coverage Share of $n$ below 0.93
Wald 0.8202 (at $n = 13$) 0.9374 18.3%
Wilson 0.9107 (at $n = 13$) 0.9505 1.0%
Agresti–Coull 0.9316 (at $n = 16$) 0.9532 0%
Clopper–Pearson 0.9521 (at $n = 160$) 0.9635 0%
Jeffreys 0.9133 (at $n = 10$) 0.9491 1.6%
Jeffreys, modified 0.9311 (at $n = 62$) 0.9503 0%
Jeffreys, HPD 0.8945 (at $n = 16$) 0.9428 9.4%
Interval ($n = 40$) Mean coverage over $p$ Lowest coverage Share of $p$ below 0.93 Lowest for $0.1 \le p \le 0.9$ Mean expected length
Wald 0.8907 0.0100 52.3% 0.8599 0.2341
Wilson 0.9520 0.8434 2.3% 0.9278 0.2367
Agresti–Coull 0.9589 0.9330 0% 0.9330 0.2426
Clopper–Pearson 0.9710 0.9503 0% 0.9503 0.2584
Jeffreys 0.9503 0.8859 7.2% 0.9173 0.2352
Jeffreys, modified 0.9532 0.9173 2.8% 0.9173 0.2366
Jeffreys, HPD 0.9407 0.8425 28.1% 0.9038 0.2312
Exact coverage of nominal 95% intervals for samples of 40, against the true proportion p. Left: the Jeffreys interval (solid) and Wilson's (dashed). Notice that the Jeffreys curve oscillates around 0.95 with a mean of 0.9503, and that its deepest dips are near p = 0.06 and p = 0.94, where Wilson's is above 0.95. Right: the Jeffreys equal-tailed interval (solid) and the HPD interval of the same posterior (dotted). Notice that the HPD curve is often lower and rarely higher (lower at 36% of the values of p, higher at 15%, equal at the rest), with a mean of 0.9407 against 0.9503.
Exact coverage of nominal 95% intervals for samples of 40, against the true proportion p. Left: the Jeffreys interval (solid) and Wilson's (dashed). Notice that the Jeffreys curve oscillates around 0.95 with a mean of 0.9503, and that its deepest dips are near p = 0.06 and p = 0.94, where Wilson's is above 0.95. Right: the Jeffreys equal-tailed interval (solid) and the HPD interval of the same posterior (dotted). Notice that the HPD curve is often lower and rarely higher (lower at 36% of the values of p, higher at 15%, equal at the rest), with a mean of 0.9407 against 0.9503.

Four observations follow from the tables and the figure.

  • On average over $p$, the Jeffreys interval is the closest to 95%. Its mean coverage over $p$ at $n = 40$ is 0.9503. (Averaged over sample sizes at $p = 0.2294$ it is 0.9491, and there Wilson's 0.9505 and the modified interval's 0.9503 are closer.) Theorem 3 does not explain this. It applies to the plain equal-tailed interval, whose coverage averaged with the weights of $\text{Beta}(\tfrac12, \tfrac12)$ is exactly 0.95; setting the limits at $x = 0$ and $x = n$ to 0 and 1 raises that prior average to 0.9544 at $n = 40$. The uniform average over $p$ is a different average, and the theorem says nothing about it: in Worked Problem 4 the prior average is 0.95 and the uniform average 0.4868. That the Jeffreys interval's uniform average lands near 0.95 is what the exact calculation shows here. Brown, Cai and DasGupta (2001, §§3.2 and 3.3) also find the Jeffreys interval's average coverage very close to the nominal level, and its expected length slightly smaller than Wilson's; here the mean expected lengths are 0.2352 and 0.2367.
  • Its worst coverage is near the ends of the range. The lowest values, 0.8859 near $p = 0.06$ and the mirror image near 0.94, come from the count $x = 0$: its interval stops at 0.0605, so for $p$ just above that value the 8% of samples with no events, $(1 - 0.0605)^{40}$, all miss. That is the downward spike Brown, Cai and DasGupta (2001, §§3.2 and 4.1.2) describe, and their modification removes it: the modified interval's lowest coverage over $p$ is 0.9173, its share below 0.93 falls from 7.2% to 2.8%, and over sample sizes it never fell below 0.93.
  • The HPD interval is shorter and covers less. Its mean expected length is the smallest of the seven, 0.2312, and its coverage averages 0.9407, with 28.1% of the values of $p$ below 0.93. Brown, Cai and DasGupta (2001, §4.2.4) report the same: HPD intervals are harder to compute and do worse in coverage than the equal-tailed interval. Being shortest for each data set (Theorem 1) is a property of each posterior separately, and it says nothing about how often the intervals contain a fixed $p$ across samples. For the count $x = 0$ the HPD interval is $(0, 0.0466)$, against $(0, 0.0605)$ for the equal-tailed interval, so every $p$ between the two upper limits loses the samples with no events.
  • The Jeffreys interval lies inside the Clopper–Pearson interval. Brown, Cai and DasGupta (2001, §4.3) note that this follows from the beta forms of the two intervals, and that the Jeffreys limits approximate those of the "mid-P" version of Clopper–Pearson, which counts only half the probability of the observed count in each tail. The script confirms the containment for every count and every $n$ from 1 to 200. So the Jeffreys interval removes the excess length of the exact interval, and gives up its guarantee.

The course page's sampling check, which drew samples of 40 without replacement from the 1,696 young adults, applies here too. In 30 runs of 10,000 samples (seeds 0 to 29, as on that page), the Jeffreys interval contained 0.2294 in a median of 0.9444 of samples (interquartile range 0.9427 to 0.9455), against 0.9414 for the binomial calculation; sampling without replacement makes $\hat p$ slightly less variable, as the course page found for the other intervals.

What each interval claims

Back to the sample of 40, with the Jeffreys interval $(0.1364, 0.3983)$ and Wilson's $(0.1419, 0.4019)$.

The credible interval's claim. Given the binomial model, Jeffreys' prior, and these 40 observations, $P(0.1364 \le p \le 0.3983 \mid \mathbf{x}) = 0.95$. The statement is about $p$, and it is conditional on the data actually observed. It depends on the prior: under the uniform prior's posterior, $\text{Beta}(11, 31)$, the same interval has probability 0.9524, and under the informative $\text{Beta}(3, 17)$ prior of Priors, Posteriors and Conjugate Families it has 0.9467. With 40 observations the three priors nearly agree.

The confidence interval's claim. The method that produced $(0.1419, 0.4019)$ is built to contain the true $p$ in about 95% of samples at each value of $p$, a claim the course page checks by exact coverage. Of this particular interval nothing probabilistic is said: as Confidence Intervals for a Proportion puts it, the true proportion is a fixed number, and this one interval either contains it or does not.

Example (the misreading of a confidence interval). "There is a 95% probability that $p$ lies in Wilson's interval." Under the frequentist model $p$ is not random, so the sentence has no meaning there; under a Bayesian model it has a meaning, but its truth depends on the prior, and Wilson's interval was not built from one. For these data the sentence happens to be nearly true under all three priors above, because with 40 observations the Wilson and Jeffreys intervals nearly coincide. The two readings are close whenever the posterior is dominated by the likelihood, and they come apart in small samples, near the edges of the parameter space, and under strong priors.

Example (the misreading of a credible interval). "A 95% credible interval contains the true value 95% of the time." Theorem 3 says it does on average over the prior, and Theorem 3's corollary says it cannot at every value of $p$. For the Jeffreys interval the failures are small away from the ends: at $n = 40$ its coverage is never below 0.9173 at the grid values of $p$ between 0.1 and 0.9. For an informative prior they can be large, as Worked Problem 4 shows.

When the two coincide. Priors, Posteriors and Conjugate Families proves that for a normal sample with unknown mean and variance and the improper prior $\pi(\mu, \sigma^2) \propto 1/\sigma^2$, the posterior of $\mu$ is Student $t$ with $n - 1$ degrees of freedom, location $\bar x$ and scale $s/\sqrt n$. Its equal-tailed interval is $\bar x \pm t_{n-1, 1-\alpha/2}\,s/\sqrt n$, the $t$ confidence interval of Confidence Intervals for a Mean, to every digit. For Student's sleep data both are $(0.700, 2.460)$ hours. The reason is that $(\bar x - \mu)/(s/\sqrt n)$ is a pivot, a function of data and parameter whose distribution involves neither: it has a $t_{n-1}$ distribution given $\mu$ and $\sigma$ in repeated sampling, and the same $t_{n-1}$ distribution given the data under this prior (Gelman et al., 2013, §3.2). Since its sampling distribution does not depend on the parameters, the $t$ interval's coverage is exactly $1 - \alpha$ at every $\mu$ and $\sigma$, and so is the credible interval's, because they are the same interval. This does not contradict the corollary above, which concerns counts and proper priors; the prior here is improper, and Theorem 3 does not apply to it.

In large samples. Under regularity conditions (Gelman et al., 2013, §4.2, name a likelihood continuous in $\theta$ and a true value $\theta_0$ not on the boundary of the parameter space), the posterior distribution approaches a normal distribution with mean $\theta_0$ and variance $1/(nI(\theta_0))$, where $I$ is the Fisher information of one observation. This is the Bernstein–von Mises theorem; Gelman et al. state it in §4.2 and defer the argument to their appendix B, and it is not proved here. The Wald interval of Asymptotics of the Maximum Likelihood Estimator has the same large-sample form, so in the limit a 95% central posterior interval covers the true value 95% of the time under repeated sampling at any fixed value (Gelman et al., 2013, §4.4). The tables above show how far $n = 40$ is from that limit for a proportion: the Jeffreys interval's coverage still ranges from 0.8859 to 0.9945 as $p$ varies.

What is proved, what is cited, what the experiment shows

Proved on this page: the HPD region is the shortest set of its posterior probability (Theorem 1); equal-tailed intervals transform with monotone maps and HPD intervals do not (Theorem 2, with the log odds counterexample); exact coverage on average over a proper prior (Theorem 3); the corollary that a credible interval for a binomial count undercovers on a set of positive length.

Used from other pages: the beta posterior of a binomial proportion and the $t$ posterior under the flat prior (Priors, Posteriors and Conjugate Families); the four course intervals and the exact coverage calculation (Confidence Intervals for a Proportion); the tower property (Conditional Expectation).

Cited: the definition of the Jeffreys interval, its modification, the containment in the Clopper–Pearson interval and the connection to mid-P intervals, and the reported performance of the HPD interval (Brown, Cai and DasGupta, 2001, §§1, 3.1.3, 3.2, 3.3, 4.1.2, 4.2.4, 4.3); the Bernstein–von Mises theorem and the large-sample coverage of posterior intervals (Gelman et al., 2013, §§4.2, 4.4, appendix B).

Computed: every coverage, expected length and interval in the tables, by the exact binomial calculation of the course page over all counts; the simulation check of Theorem 3; the sampling check without replacement from the 1,696 young adults of NHANES 2017 to 2018 (National Center for Health Statistics, 2020). The sleep data are Student's (Student, 1908). The coverage numbers are exact for the binomial model; the two simulations are observations of 30 seeded runs each. Every number is computed and checked against this page by tools/experiments/statistics/credible-intervals.py, which reuses the interval functions of tools/experiments/applied-statistics/proportions.py.

Worked problems

The problems were written for this article, on the running sample and on small exact cases. Each number is recomputed by the article's script.

Problem 1 (a sample with no events). A sample of 40 young adults has no one with hypertension. Give the 95% Jeffreys interval, its HPD version, the modified Jeffreys interval and the Clopper–Pearson interval, and explain why they differ.

Solution. The Jeffreys posterior is $\text{Beta}(0.5, 40.5)$. Its 0.025 and 0.975 quantiles are 0.0000122 and 0.0605; following Brown, Cai and DasGupta the lower limit is set to 0, giving $(0,\ 0.0605)$. The posterior density, proportional to $p^{-1/2}(1 - p)^{39.5}$, decreases on the whole interval, so the HPD interval starts at 0 and ends at the 0.95 quantile: $(0,\ 0.0466)$. The modified Jeffreys interval replaces the upper limit by $1 - 0.025^{1/40} = 0.0881$, which is also the Clopper–Pearson upper limit for $x = 0$, so those two intervals coincide: $(0,\ 0.0881)$. The two constructions answer different questions. The Jeffreys upper limit is the point above which the posterior puts 2.5% of its probability. The Clopper–Pearson upper limit is the value of $p$ at which a count of 0 would itself have probability 0.025, a statement about the sampling distribution of the count; at $p = 0.0605$ a count of 0 still has probability $(1 - 0.0605)^{40} = 0.082$. The HPD interval is shortest of all, and its coverage near $p = 0.05$ suffers accordingly: at that value, $P(X = 0) = 0.95^{40} = 0.1285$ of samples give an interval that misses.

Problem 2 (samples of five). For samples of 5 from the young-adult population, $p = 0.2294$, list the 95% Jeffreys intervals for every count and compute the exact coverage. The course page found 0.7169 for the Wald interval and 0.9170 for Wilson's.

Solution. The intervals, with the limits at $x = 0$ and $x = 5$ set to 0 and 1, are:

$x$ 0 1 2 3 4 5
$P(X = x)$ 0.2718 0.4045 0.2408 0.0717 0.0107 0.0006
Jeffreys interval $(0,\ 0.379)$ $(0.023,\ 0.629)$ $(0.094,\ 0.791)$ $(0.209,\ 0.906)$ $(0.371,\ 0.977)$ $(0.621,\ 1)$
Contains 0.2294 yes yes yes yes no no

The coverage is $0.2718 + 0.4045 + 0.2408 + 0.0717 = 0.9887$. The difference from Wilson's 0.9170 is the count 3: Wilson's lower limit there is 0.231, just above 0.2294, and the Jeffreys lower limit is 0.209, just below. With five observations the attainable coverages at this $p$ are far apart, and which one an interval achieves depends on where a single endpoint falls.

Problem 3 (the same interval under three priors). Find the posterior probability of the Jeffreys interval $(0.1364, 0.3983)$ for the sample of 40 under the uniform prior and under the $\text{Beta}(3, 17)$ prior.

Solution. The posteriors are $\text{Beta}(11, 31)$ and $\text{Beta}(13, 47)$. Their distribution functions at the two ends give $P = 0.9524$ and $P = 0.9467$. The uniform posterior leaves 0.0188 below the interval and 0.0288 above it: shifted to the right of the Jeffreys posterior, it moves mass out of the lower tail and into the upper one, and loses less than it gains. The informative posterior, centered at 0.2167, leaves 0.0522 below 0.1364 and only 0.0012 above 0.3983: almost nothing is left in its upper tail, and its lower tail is more than twice the 0.025 the interval was built to leave. The interval's probability is a property of the interval and a posterior together.

Problem 4 (the coverage of an informative credible interval). Using the $\text{Beta}(3, 17)$ prior, the 95% equal-tailed credible interval for a count $x$ of 40 is the central 95% of $\text{Beta}(3 + x, 57 - x)$. Compute its exact coverage at $p = 0.15$, $0.2294$, $0.30$ and $0.40$, and its coverage averaged over the prior.

Solution. The coverages are 0.9836, 0.9210, 0.8887 and 0.7888. The prior centers the intervals near 0.15, so they cover well there and progressively worse as the true $p$ moves away: at $p = 0.40$ about one interval in five misses. Averaged uniformly over 2,000 values of $p$ the coverage is only 0.4868. The coverage is at least 0.95 only for $p$ between about 0.05 and 0.25; it is below 0.07 for every $p$ above 0.72 and exactly 0 above 0.8224, because even 40 successes in 40 give an interval ending there; and it is 0 for $p$ below 0.0106, the lower end of the interval for $x = 0$. Averaged with the prior's own weights, $\text{Beta}(3, 17)$, it is 0.9501 on the grid, and exactly 0.95 by Theorem 3. An informative prior keeps its promise when it is right about where $p$ is likely to be, and only then.

Summary

  • A $100(1 - \alpha)\%$ credible interval has posterior probability $1 - \alpha$. The equal-tailed interval leaves $\alpha/2$ in each tail; the HPD region keeps the highest densities.
  • The HPD region is the shortest set of its probability (Theorem 1). The equal-tailed interval transforms with any monotone change of scale; the HPD interval depends on the scale (Theorem 2).
  • A credible interval covers with exactly its stated probability on average over its prior (Theorem 3). For binomial data it therefore undercovers on a set of values of $p$ of positive length.
  • The Jeffreys interval for a proportion, the central 95% of $\text{Beta}(x + \tfrac12, n - x + \tfrac12)$ with the ends at 0 and 1 for $x = 0$ and $x = n$, has mean coverage 0.9503 over $p$ at $n = 40$ and mean expected length 0.2352, against Wilson's 0.9520 and 0.2367; its dips near $p = 0.06$ and $0.94$ are removed by the modification of Brown, Cai and DasGupta.
  • HPD intervals are shorter and cover less: mean coverage 0.9407 at $n = 40$.
  • Under the prior $1/\sigma^2$ the credible interval for a normal mean is the $t$ interval; in large smooth models the two kinds of interval agree in the limit (Bernstein–von Mises, cited).

What we have learned

A credible interval answers the question a confidence interval is often misread as answering, the probability that the parameter lies in this interval, at the price of a prior on which the answer depends. Judged as a confidence interval, the Jeffreys interval for a proportion does about as well as Wilson's: its coverage averages very close to 95% over $p$, as the exact calculation shows, and it falls short only near the ends of the range. An interval built on an informative prior covers well where the prior is right and badly where it is wrong.

That leaves the question of which prior to use when nothing particular is known, and why Jeffreys' is a natural choice for a proportion. Choosing a Prior derives it from the Fisher information, shows why a flat prior on one scale is not flat on another, and reads the smoothing methods of text retrieval as priors.

References