Among the 5,176 adults examined in the National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018 (National Center for Health Statistics, 2020), systolic blood pressure rises with age, but not in a straight line. Fit polynomials in age of degree 0 to 8 and score each by a criterion. Akaike's information criterion (AIC) chooses degree 5. The Bayesian information criterion (BIC) chooses the straight line, although the $F$ test of the line against the quintic rejects the line with $p = 0.0002$. Which criterion is right, and what does each one promise? The two are built for different jobs. BIC finds the true model when one of the candidates is true and $n$ is large enough; AIC predicts asymptotically as well as the best candidate when none is true (Shao, 1997, under his conditions), and at small $n$ BIC can predict better.
The Bayesian information criterion of Schwarz (1978) scores a model with $k$ estimated parameters and maximized log-likelihood $\hat\ell$ by $$\mathrm{BIC} = -2\hat\ell + k\ln n,$$ and the model with the smallest score is chosen. Bayes Factors and Posterior Model Probabilities derives it (its Corollary 2): Laplace's approximation to the marginal likelihood of a model gives $\ln m(\mathbf{x}) = \hat\ell - \tfrac{k}{2}\ln n + O_p(1)$, so $-\mathrm{BIC}/2$ is the log marginal likelihood up to a term that does not grow with $n$. That derivation is not repeated here. This article proves what it left open, and makes the answer above precise.
First we prove Laplace's method in one dimension with its error term, which Bayes Factors states as derived but not proved, and check it on the binomial. Then we prove that BIC is consistent: in a finite family of normal linear models that contains a true one, it chooses the smallest true model with probability tending to 1. Next we compute exactly how often a penalty of any size overfits or underfits between two nested models, extending the AIC computation of Akaike's Information Criterion, Derived to BIC, and prove that over a nested sequence AIC's choices tend to those of a random walk. Then we state the trade-off between consistency and efficiency as the literature proves it, and measure it in two simulations built from the NHANES adults. Finally we return to the adults themselves. The article assumes The Geometry of Least Squares and the distribution theory of Sampling Distributions from the Normal.
Notation and the running data
The notation is the unit's (Prediction Error and Model Choice). Responses are $\mathbf{y} = \boldsymbol\mu + \boldsymbol\varepsilon$, with true mean vector $\boldsymbol\mu$ and errors $\varepsilon_i$ independent, normal, with mean 0 and variance $\sigma^2$. A candidate model $j$ has an $n \times p_j$ design matrix $X_j$ of rank $p_j$ (the intercept counted in $p_j$), hat matrix $H_j = X_j(X_j^{\mathsf T}X_j)^{-1}X_j^{\mathsf T}$ and residual sum of squares $\mathrm{RSS}_j = \lVert(I - H_j)\mathbf{y}\rVert^2$. With the variance also estimated it has $k_j = p_j + 1$ parameters. The maximized normal log-likelihood is $\hat\ell_j = -\tfrac{n}{2}\ln(2\pi\,\mathrm{RSS}_j/n) - \tfrac{n}{2}$, so, dropping the constant $n(1 + \ln 2\pi)$ that every model shares, $$\mathrm{AIC}_j = n\ln\frac{\mathrm{RSS}_j}{n} + 2k_j, \qquad \mathrm{BIC}_j = n\ln\frac{\mathrm{RSS}_j}{n} + k_j\ln n,$$ as in Model Selection. Model $j$ is correct when $\boldsymbol\mu$ lies in the column space $\mathcal{C}(X_j)$, that is, when $(I - H_j)\boldsymbol\mu = \mathbf{0}$. Logarithms are natural.
Data. The adults are the unit's population: the 5,176 of the NHANES file with at least one systolic reading and a body mass index (the pressure is the mean of the available readings, in mm Hg), aged 18 to 80, treated as a finite population, without the survey's weights. Age is rescaled to $t = (\text{age} - 49)/31$, which runs from $-1$ to 1, and a polynomial of degree $d$ is fitted in the Legendre basis $P_0(t) = 1$, $P_1(t) = t$, $P_2(t) = (3t^2 - 1)/2, \ldots$, so it has $p = d + 1$ coefficients and $k = d + 2$ parameters. The basis changes no fitted value (The Geometry of Least Squares, Theorem 2, part 4); it only keeps the arithmetic stable. Every number in this article is computed by tools/experiments/statistics/bayesian-information-criterion.py.
Laplace's method in one dimension
A marginal likelihood is an integral $\int \pi(\theta)\,e^{\ell(\theta)}\,d\theta$, and for $n$ observations the log-likelihood $\ell$ grows like $n$. Here is the idea in one sentence: when the exponent is $n$ times a fixed function, almost all of the integral comes from a shrinking neighborhood of the maximum, where the function is a downward parabola and the integral is a normal one. The theorem below makes that exact up to a factor $1 + O(1/n)$.
Theorem 1 (Laplace's method). Let $h$ and $\pi$ be real functions on an open interval $(\alpha, \beta)$, finite or infinite, with $\pi \ge 0$, and let $\hat\theta \in (\alpha, \beta)$. Suppose:
- $h$ has four continuous derivatives on a neighborhood of $\hat\theta$, with $h'(\hat\theta) = 0$ and $a = -h''(\hat\theta) > 0$;
- $\pi$ has two continuous derivatives on a neighborhood of $\hat\theta$, and $\pi(\hat\theta) > 0$;
- the maximum is separated: for every $\delta > 0$ there is $\eta > 0$ with $h(\theta) \le h(\hat\theta) - \eta$ whenever $\lvert\theta - \hat\theta\rvert \ge \delta$;
- $\int_\alpha^\beta \pi(\theta)\,e^{h(\theta) - h(\hat\theta)}\,d\theta < \infty$.
Then, as $n \to \infty$, $$I_n = \int_\alpha^\beta \pi(\theta)\,e^{nh(\theta)}\,d\theta = e^{nh(\hat\theta)}\,\pi(\hat\theta)\sqrt{\frac{2\pi}{na}}\,\Big(1 + O\big(\tfrac1n\big)\Big).$$
Here is the plan of the proof. Cut the integral at a small distance $\delta$ from $\hat\theta$. Outside, the integrand is smaller than its peak by a factor $e^{-n\eta}$, which is negligible. Inside, rescale by $\sqrt n$, expand $h$ and $\pi$ in Taylor series, and observe that the first correction terms are odd functions, which integrate to zero; everything left is of order $1/n$.
Proof. Translate $\theta$ and subtract a constant from $h$ so that $\hat\theta = 0$ and $h(0) = 0$; neither changes the claim. Write $\pi_0 = \pi(0)$ and $\pi_1 = \pi'(0)$, and $h_3 = h'''(0)$.
Choosing $\delta$. Pick $\delta_0 > 0$ so that $[-\delta_0, \delta_0]$ lies inside both neighborhoods of conditions 1 and 2, and let $C_3$, $C_4$ and $B_2$ bound $\lvert h'''\rvert$, $\lvert h''''\rvert$ and $\lvert\pi''\rvert$ on it. Now fix $\delta = \min(\delta_0, 3a/(2C_3))$ (just $\delta_0$ if $C_3 = 0$). By Taylor's theorem with the Lagrange remainder, for $\lvert\theta\rvert \le \delta$, $$h(\theta) = -\tfrac{a}{2}\theta^2 + R(\theta), \quad \lvert R(\theta)\rvert \le \tfrac{C_3}{6}\lvert\theta\rvert^3 \le \tfrac{C_3\delta}{6}\theta^2 \le \tfrac{a}{4}\theta^2. \tag{1}$$
The outer part. Let $\eta$ be the separation for this $\delta$ (condition 3). For $\lvert\theta\rvert \ge \delta$, $nh(\theta) = (n - 1)h(\theta) + h(\theta) \le -(n - 1)\eta + h(\theta)$, so $$0 \le \int_{\lvert\theta\rvert \ge \delta}\pi(\theta)e^{nh(\theta)}\,d\theta \le e^{-(n - 1)\eta}\int_\alpha^\beta \pi(\theta)e^{h(\theta)}\,d\theta,$$ which is finite by condition 4 and smaller than any power of $1/n$, so it is $n^{-1/2} \cdot O(n^{-1})$ with room to spare.
The inner part. Substitute $\theta = u/\sqrt n$: $$A_n = \int_{-\delta}^{\delta}\pi(\theta)e^{nh(\theta)}\,d\theta = \frac{1}{\sqrt n}\int_{\lvert u\rvert < \delta\sqrt n}\pi\big(\tfrac{u}{\sqrt n}\big)\,e^{-au^2/2}\,e^{g(u)}\,du, \qquad g(u) = nR\big(\tfrac{u}{\sqrt n}\big).$$ By (1), $\lvert g(u)\rvert \le au^2/4$ on the range of integration. A Taylor expansion of $R$ one order further gives $$g(u) = \frac{h_3u^3}{6\sqrt n} + r(u), \qquad \lvert r(u)\rvert \le \frac{C_4u^4}{24n},$$ and one of $\pi$ gives $\pi(u/\sqrt n) = \pi_0 + \pi_1u/\sqrt n + \rho(u)$ with $\lvert\rho(u)\rvert \le B_2u^2/(2n)$. Finally $e^{g} = 1 + g + q$ with $\lvert q\rvert \le \tfrac12 g^2e^{\lvert g\rvert}$, from the series of the exponential. Multiply out: $$\pi\big(\tfrac{u}{\sqrt n}\big)e^{g(u)} = \pi_0 + \Big[\frac{\pi_1u}{\sqrt n} + \frac{\pi_0h_3u^3}{6\sqrt n}\Big] + E(u),$$ where $E(u)$ collects $\pi_0r + \pi_0q + (\pi_1u/\sqrt n)(g + q) + \rho\,e^{g}$. Each of these four terms is bounded by a constant times $(1 + u^8)e^{\lvert g(u)\rvert}/n$ for $n \ge 1$:
- $\pi_0\lvert r\rvert \le \pi_0C_4u^4/(24n)$;
- $\lvert g\rvert \le (\lvert h_3\rvert\lvert u\rvert^3 + C_4u^4)/\sqrt n$, so $\pi_0\lvert q\rvert \le \tfrac12\pi_0g^2e^{\lvert g\rvert} \le \pi_0(\lvert h_3\rvert\lvert u\rvert^3 + C_4u^4)^2e^{\lvert g\rvert}/(2n)$;
- $\lvert\pi_1u/\sqrt n\rvert(\lvert g\rvert + \lvert q\rvert) \le \lvert\pi_1\rvert\lvert u\rvert(\lvert h_3\rvert\lvert u\rvert^3 + C_4u^4)(1 + \lvert g\rvert)e^{\lvert g\rvert}/n$;
- $\lvert\rho\rvert e^{\lvert g\rvert} \le B_2u^2e^{\lvert g\rvert}/(2n)$.
(Every polynomial in $\lvert u\rvert$ of degree at most 8 is at most a constant times $1 + u^8$, and $1 + \lvert g\rvert \le 1 + au^2/4$.) Since $e^{\lvert g\rvert} \le e^{au^2/4}$, the error term contributes at most $$\frac{c}{n}\int_{-\infty}^{\infty}(1 + u^8)(1 + au^2/4)e^{-au^2/4}\,du = O\big(\tfrac1n\big)$$ to the integral. The bracketed terms are odd in $u$ and the range is symmetric, so they integrate to 0. The leading term gives $\pi_0\int_{\lvert u\rvert < \delta\sqrt n}e^{-au^2/2}\,du = \pi_0\sqrt{2\pi/a} - \pi_0\int_{\lvert u\rvert \ge \delta\sqrt n}e^{-au^2/2}\,du$, and the last integral is at most $2e^{-a\delta^2n/4}\int_0^\infty e^{-au^2/4}\,du$, again smaller than any power of $1/n$. Collecting, $$I_n = \frac{1}{\sqrt n}\Big[\pi_0\sqrt{\tfrac{2\pi}{a}} + O\big(\tfrac1n\big)\Big] = \pi_0\sqrt{\frac{2\pi}{na}}\Big(1 + O\big(\tfrac1n\big)\Big),$$ dividing by $\pi_0\sqrt{2\pi/a} > 0$. Undoing the shift multiplies both sides by $e^{nh(\hat\theta)}$. $\square$
Why the error is $1/n$ and not $1/\sqrt n$. The first corrections, $\pi_1u/\sqrt n$ from the slope of the prior and $h_3u^3/(6\sqrt n)$ from the skewness of the peak, are of order $n^{-1/2}$ but odd, and a symmetric normal curve cancels them. That is why the approximation is good even for modest $n$.
The next term. Keeping one more term of each expansion ($h$ to sixth order, $\pi$ to fourth) and integrating against the normal curve, whose even moments are $\int u^2 = 1/a$, $\int u^4 = 3/a^2$ and $\int u^6 = 15/a^3$ in units of $\sqrt{2\pi/a}$, collects the $1/n$ terms: $$I_n = e^{nh(\hat\theta)}\sqrt{\frac{2\pi}{na}}\Big[\pi_0 + \frac1n\Big(\frac{\pi''(\hat\theta)}{2a} + \frac{\pi_1h_3}{2a^2} + \frac{\pi_0h_4}{8a^2} + \frac{5\pi_0h_3^2}{24a^3}\Big) + \cdots\Big],$$ with $h_4 = h''''(\hat\theta)$. The four terms come from $\pi''u^2/2$, from $\pi_1u \cdot h_3u^3/6$, from $h_4u^4/24$ and from $\tfrac12(h_3u^3/6)^2$; for instance $\tfrac{1}{72}h_3^2 \cdot 15/a^3 = \tfrac{5}{24}h_3^2/a^3$. This identifies the coefficient; bounding the remainder at that order needs the longer expansions and is not done here. The check below confirms the coefficient numerically.
In statistics. The marginal likelihood is $m(\mathbf{x}) = \int\pi(\theta)\,e^{nh_n(\theta)}\,d\theta$ with $h_n = \ell/n$, the average log-likelihood, which changes with $n$. The proof uses $h$ only through $a$, $C_3$, $C_4$, $\delta_0$, $\eta$ and the integral in condition 4, and $\pi$ through $\pi_0$, $\pi_1$, $B_2$. So the conclusion holds uniformly over any family of functions $h_n$ that satisfy conditions 1 to 4 with common bounds, and it applies to a random $h_n$ on the event that it does. Kass and Raftery (1995, §4.1.1) report that under the regularity conditions of Kass, Tierney and Kadane (1990) the $d$-dimensional version has relative error $O(n^{-1})$, and they cite de Bruijn (1970, §4.4) and Tierney and Kadane (1986) for the method. The $d$-dimensional statement is the one Bayes Factors uses for BIC.
A check on the binomial
With $x$ successes in $n$ independent trials and a uniform prior on the success probability $\theta$, the probability of the observed sequence, averaged over the prior, is $$m = \int_0^1\theta^x(1 - \theta)^{n - x}\,d\theta = \mathrm{B}(x + 1, n - x + 1),$$ the beta function. Take $x = 0.3n$. Then the integrand is $e^{nh(\theta)}$ with $h(\theta) = 0.3\ln\theta + 0.7\ln(1 - \theta)$, the same function for every $n$, and $\pi \equiv 1$. The maximum is at $\hat\theta = 0.3$, where $a = 1/(0.3 \times 0.7) = 4.762$, $h_3 = 2/0.3^2 - 2/0.7^2 = 18.141$ and $h_4 = -6/0.3^3 - 6/0.7^3 = -239.715$. All four conditions hold: $h$ is smooth on $(0, 1)$, increasing then decreasing with $h \to -\infty$ at both ends, and the integral in condition 4 is finite. The predicted coefficient is $h_4/(8a^2) + 5h_3^2/(24a^3) = -0.6865$, so the exact value should be the Laplace value times about $1 - 0.6865/n$.
| $n$ | Exact $\ln m$ | Laplace $\ln m$ | $n \times$ (exact/Laplace $- 1$) |
|---|---|---|---|
| 10 | −7.1854 | −7.1213 | −0.6206 |
| 40 | −26.1573 | −26.1404 | −0.6686 |
| 160 | −100.1415 | −100.1373 | −0.6819 |
| 640 | −394.0463 | −394.0453 | −0.6854 |
| 2,560 | −1,567.5981 | −1,567.5979 | −0.6862 |
The relative error is −6.21 per cent at $n = 10$ and falls by a factor of four each time $n$ grows fourfold, as an error of order $1/n$ must; multiplied by $n$, it settles on the predicted coefficient. The error in $\ln m$ is the logarithm of $1 + O(1/n)$, so it is also of order $1/n$. That is the sense in which Laplace's approximation is accurate. BIC is a cruder object: Bayes Factors (Corollary 2 and its simulation) shows that dropping the prior and curvature terms leaves an error that tends to a constant, not to zero. Why, then, is BIC used to choose models at all? Because, as the next section proves, a constant error in the log marginal likelihood cannot stop it from finding the true model.
What BIC is for: consistency
Here is the idea in plain words. A model that is too small leaves part of the true mean in its residuals, so its residual sum of squares is too large by an amount proportional to $n$. A model that is too large fits only noise with its extra columns, and the improvement in the residual sum of squares stays of order 1 however large $n$ grows. BIC's penalty $\ln n$ per parameter grows, but more slowly than $n$: it eventually outweighs the order-1 gain of any superfluous column, and it never outweighs the order-$n$ loss of a missing one.
A selection rule is consistent if, when the family contains correct models, the probability that it chooses the smallest correct one tends to 1 as $n \to \infty$ (Shao, 1997, definition (2.1), in the case where that model minimizes the loss). The theorem covers a whole class of penalties.
Theorem 2 (consistency of BIC). Let the candidates be a fixed finite family of models $j = 1, \ldots, J$ in the normal linear model, with designs that may change with $n$. Suppose:
- exactly one correct model, $j_0$, has the smallest number of coefficients among the correct models (every other correct model has $p_j > p_{j_0}$);
- the true means are bounded on average, $\lVert\boldsymbol\mu\rVert^2/n \le B$ for all $n$;
- every incorrect model misses by a margin that does not vanish: with $\Delta_j = \lVert(I - H_j)\boldsymbol\mu\rVert^2/n$, there is $\Delta > 0$ with $\Delta_j \ge \Delta$ for every incorrect $j$ and all large $n$;
- the numbers of coefficients $p_j$ do not depend on $n$ (and $p_j < n$), and the penalty $c_n$ satisfies $c_n \to \infty$ and $c_n/n \to 0$.
Let $T_j = n\ln(\mathrm{RSS}_j/n) + c_nk_j$, and let $\hat\jmath$ minimize $T_j$ (ties broken in favor of the smaller model). Then $P(\hat\jmath = j_0) \to 1$. BIC is the case $c_n = \ln n$.
Proof. Since $\hat\jmath \ne j_0$ only if $T_j \le T_{j_0}$ for some $j \ne j_0$, and there are finitely many $j$, it is enough to show $P(T_j \le T_{j_0}) \to 0$ for each $j \ne j_0$.
A fact about correct models. If model $j$ is correct, $(I - H_j)\boldsymbol\mu = \mathbf{0}$, so $\mathrm{RSS}_j = \boldsymbol\varepsilon^{\mathsf T}(I - H_j)\boldsymbol\varepsilon$, which is $\sigma^2$ times a chi-squared variable with $n - p_j$ degrees of freedom (Sampling Distributions from the Normal, Theorem 2, with the projection $I - H_j$ of rank $n - p_j$). It has mean $\sigma^2(n - p_j)$ and variance $2\sigma^4(n - p_j)$ (The Chi-Squared and t Distributions), so by Chebyshev's inequality (Probability Inequalities) $$P\Big(\Big\lvert\frac{\mathrm{RSS}_j}{n} - \sigma^2\frac{n - p_j}{n}\Big\rvert \ge \epsilon\Big) \le \frac{2\sigma^4(n - p_j)}{n^2\epsilon^2} \to 0,$$ and $\mathrm{RSS}_j/n \to \sigma^2$ in probability. In particular this holds for $j_0$.
Overfitting: $j \ne j_0$ correct. Then $k_j - k_{j_0} \ge 1$, and $T_j \le T_{j_0}$ means $n\ln(\mathrm{RSS}_{j_0}/\mathrm{RSS}_j) \ge c_n(k_j - k_{j_0}) \ge c_n$. Both models are correct, so $$\mathrm{RSS}_{j_0} - \mathrm{RSS}_j = \boldsymbol\varepsilon^{\mathsf T}(I - H_{j_0})\boldsymbol\varepsilon - \boldsymbol\varepsilon^{\mathsf T}(I - H_j)\boldsymbol\varepsilon = \boldsymbol\varepsilon^{\mathsf T}H_j\boldsymbol\varepsilon - \boldsymbol\varepsilon^{\mathsf T}H_{j_0}\boldsymbol\varepsilon \le \boldsymbol\varepsilon^{\mathsf T}H_j\boldsymbol\varepsilon,$$ since $\boldsymbol\varepsilon^{\mathsf T}H_{j_0}\boldsymbol\varepsilon = \lVert H_{j_0}\boldsymbol\varepsilon\rVert^2 \ge 0$. With $\ln(1 + x) \le x$, $$n\ln\frac{\mathrm{RSS}_{j_0}}{\mathrm{RSS}_j} \le n\,\frac{\mathrm{RSS}_{j_0} - \mathrm{RSS}_j}{\mathrm{RSS}_j} \le \frac{\boldsymbol\varepsilon^{\mathsf T}H_j\boldsymbol\varepsilon}{\mathrm{RSS}_j/n}.$$ So $T_j \le T_{j_0}$ requires either $\mathrm{RSS}_j/n < \sigma^2/2$ or $\boldsymbol\varepsilon^{\mathsf T}H_j\boldsymbol\varepsilon \ge c_n\sigma^2/2$. The first has probability tending to 0 by the fact above. For the second, $E[\boldsymbol\varepsilon^{\mathsf T}H_j\boldsymbol\varepsilon] = \sigma^2\operatorname{tr}H_j = \sigma^2p_j$ (Covariance Matrices and Quadratic Forms, Theorem 5), so Markov's inequality bounds its probability by $2p_j/c_n \to 0$.
Underfitting: $j$ incorrect. Expand $\mathrm{RSS}_j = \lVert(I - H_j)(\boldsymbol\mu + \boldsymbol\varepsilon)\rVert^2$: $$\frac{\mathrm{RSS}_j}{n} = \Delta_j + \frac{2\boldsymbol\mu^{\mathsf T}(I - H_j)\boldsymbol\varepsilon}{n} + \frac{\boldsymbol\varepsilon^{\mathsf T}(I - H_j)\boldsymbol\varepsilon}{n}.$$ The middle term has mean 0 and variance $4\sigma^2\lVert(I - H_j)\boldsymbol\mu\rVert^2/n^2 = 4\sigma^2\Delta_j/n \le 4\sigma^2B/n$, because $\Delta_j \le \lVert\boldsymbol\mu\rVert^2/n \le B$ (a projection does not lengthen a vector); by Chebyshev it tends to 0 in probability. The last term tends to $\sigma^2$ in probability by the argument of the fact above. So with probability tending to 1, $\mathrm{RSS}_j/n \ge \sigma^2 + \Delta/2$ and $\mathrm{RSS}_{j_0}/n \le \sigma^2 + \Delta/4$, and then $$n\ln\frac{\mathrm{RSS}_j}{\mathrm{RSS}_{j_0}} \ge n\gamma, \qquad \gamma = \ln\frac{\sigma^2 + \Delta/2}{\sigma^2 + \Delta/4} > 0.$$ Now $T_j \le T_{j_0}$ means $n\ln(\mathrm{RSS}_j/\mathrm{RSS}_{j_0}) \le c_n(k_{j_0} - k_j) \le c_nK$, with $K$ the largest $k_j$. Since $c_n/n \to 0$, $n\gamma > c_nK$ for all large $n$, so $P(T_j \le T_{j_0}) \to 0$. $\square$
The proof never used that the models are nested, and it used normality only to get $\mathrm{RSS}_j/n \to \sigma^2$, through the chi-squared variance. It did use that the family is finite and fixed. Shao (1997, Theorem 2(ii)) proves the same conclusion, for the class of criteria $\mathrm{RSS}_j + \lambda_n\hat\sigma^2p_j$ with $\lambda_n \to \infty$ and $\lambda_n/n \to 0$, under weaker conditions that allow the number of candidates and their dimensions to grow with $n$, provided a correct model of fixed dimension exists; Nishii (1984) had obtained, for normal linear regression, the asymptotic distribution of the model selected by AIC, FPE, $C_p$ and BIC (as his abstract summarizes). Condition 3 is what makes the underfitting step work: a true effect whose contribution per observation shrinks as fast as $(\ln n)/n$ can be missed by BIC forever. Theorem 3 below shows why, for one coefficient: if its noncentrality is $\lambda_n = c\ln n$, then $F$ is approximately $(Z + \sqrt{\lambda_n})^2$ with $Z$ standard normal, while BIC's threshold $\tau(\ln n)$ is about $\ln n$, so the chance that BIC drops the term is about $P(\lvert Z + \sqrt{c\ln n}\rvert \le \sqrt{\ln n})$, which tends to 1 when $c < 1$ and does not tend to 0 when $c = 1$. The simulation below shows how slowly the theorem can take effect.
Example (the curvature of blood pressure). Treat the population quadratic as the truth: the least-squares fit of pressure on $P_0, P_1, P_2$ over all 5,176 adults, $\mu(t) = 125.763 + 16.179\,P_1(t) + 0.938\,P_2(t)$, with normal errors of standard deviation $\sigma = 17.39$ mm Hg (the fit's residual standard deviation). For a design whose ages follow the population's, the line misses this truth by $\Delta_{\text{line}}/\sigma^2 = 0.00071$ per observation: the curvature is real but faint. The $F$ statistic for the quadratic term then has noncentrality about $0.00071\,n$, while BIC asks it, roughly, to exceed $\ln n$ (Problem 3). As a rough guide to the scale, $0.00071\,n$ first exceeds $\ln n$ at $n = 13{,}481$. Below that size BIC usually chooses the line, and it is not wrong to: it is weighing a real effect too small to estimate well.
Why AIC is not consistent
AIC is the case $c_n = 2$, which violates $c_n \to \infty$. The underfitting half of the proof still works, because $2/n \to 0$; the overfitting half fails, because a superfluous column improves $n\ln\mathrm{RSS}$ by an amount of order 1 that does not fade, and 2 is a fixed price. Akaike's Information Criterion, Derived proves the limit: with $m$ superfluous coefficients, AIC prefers the larger model with probability tending to $P(\chi^2_m > 2m)$, which is 0.1573 for $m = 1$, and its Theorem 3 (section "AIC chooses too large a model") computes the exact probability at every $n$ through the $F$ distribution. The theorem below extends that computation to any fixed or growing penalty, BIC's included, and to the opposite error, dropping a column that belongs; for the penalty 2 with the smaller model correct, it is the AIC article's Theorem 3.
Theorem 3 (exact chances of over- and underfitting for any penalty). Let models 0 and 1 be nested, $\mathcal{C}(X_0) \subset \mathcal{C}(X_1)$, with $p_0 < p_1 < n$ and $m = p_1 - p_0$, and let model 0 be correct. Let $$F = \frac{(\mathrm{RSS}_0 - \mathrm{RSS}_1)/m}{\mathrm{RSS}_1/(n - p_1)}.$$ A criterion $n\ln(\mathrm{RSS}/n) + c\,k$ prefers model 1 exactly when $F > \tau(c) = \dfrac{(n - p_1)(e^{cm/n} - 1)}{m}$, and $$P(\text{AIC prefers model 1}) = P\big(F_{m,\,n - p_1} > \tau(2)\big), \qquad P(\text{BIC prefers model 1}) = P\big(F_{m,\,n - p_1} > \tau(\ln n)\big),$$ where $F_{m,\,n - p_1}$ has the central $F$ distribution. These probabilities do not depend on the design. If instead model 1 is correct and model 0 is not, the same events have $F$ noncentral, with noncentrality $\lambda = \lVert(I - H_0)\boldsymbol\mu\rVert^2/\sigma^2$.
Proof. Model 1 is preferred when $n\ln(\mathrm{RSS}_1/n) + c(k_0 + m) < n\ln(\mathrm{RSS}_0/n) + ck_0$, that is, when $\mathrm{RSS}_0/\mathrm{RSS}_1 > e^{cm/n}$. Subtract 1 and multiply by $(n - p_1)/m$: this is $F > \tau(c)$. By The Geometry of Least Squares (Theorem 4), $\mathrm{RSS}_0 - \mathrm{RSS}_1 = \lVert(H_1 - H_0)\mathbf{y}\rVert^2$ with $H_1 - H_0$ a projection of rank $m$, and $\mathrm{RSS}_1 = \lVert(I - H_1)\mathbf{y}\rVert^2$ with $I - H_1$ a projection of rank $n - p_1$; the two projections multiply to zero. By Sampling Distributions from the Normal (Theorem 2) the two sums of squares are independent, $\mathrm{RSS}_1/\sigma^2$ is central chi-squared because $(I - H_1)\boldsymbol\mu = \mathbf{0}$, and $(\mathrm{RSS}_0 - \mathrm{RSS}_1)/\sigma^2$ is chi-squared with noncentrality $\lVert(H_1 - H_0)\boldsymbol\mu\rVert^2/\sigma^2$. When model 0 is correct that is 0, and $F$ is central $F$; when model 1 is correct, $(H_1 - H_0)\boldsymbol\mu = \boldsymbol\mu - H_0\boldsymbol\mu$, giving the stated $\lambda$. This is the $F$ statistic of The General Linear Hypothesis, Theorem 2. $\square$
For AIC the threshold $\tau(2)$ tends to 2, and the AIC article shows that the probability tends to $P(\chi^2_m > 2m)$: 0.1573 for $m = 1$, 0.1353 for $m = 2$ ($\chi^2_2 > 4$) and 0.1116 for $m = 3$ ($\chi^2_3 > 6$). For BIC, $\tau(\ln n) \sim \ln n \to \infty$ and the probability tends to 0, which is the overfitting half of Theorem 2 again.
The exact chance of overfitting
The true model is a quadratic ($p_0 = 3$) and the cubic is offered ($p_1 = 4$, $m = 1$):
| $n$ | AIC threshold $\tau(2)$ | $P$(AIC overfits) | BIC threshold $\tau(\ln n)$ | $P$(BIC overfits) |
|---|---|---|---|---|
| 20 | 1.683 | 0.2130 | 2.59 | 0.1274 |
| 50 | 1.877 | 0.1773 | 3.74 | 0.0592 |
| 100 | 1.939 | 0.1670 | 4.52 | 0.0360 |
| 1,000 | 1.994 | 0.1582 | 6.90 | 0.0087 |
| 10,000 | 1.999 | 0.1574 | 9.21 | 0.0024 |
| 100,000 | 2.000 | 0.1573 | 11.51 | 0.0007 |
AIC's chance of adding a useless coefficient is above its limit at small $n$ and never falls below 0.157. BIC's falls steadily, but slowly: like the tail of $\chi^2_1$ beyond $\ln n$, roughly in proportion to $1/\sqrt{n\ln n}$.
The exact chance of underfitting
Now let the population quadratic of the example be the truth and offer the line ($p_0 = 2$) against the quadratic ($p_1 = 3$). The noncentrality depends on the design. For a design made of $r$ copies of the 5,176 adults' ages it is exactly $r$ times its value on the population:
| $n$ | Noncentrality $\lambda$ | $P$(AIC chooses the line) | $P$(BIC chooses the line) |
|---|---|---|---|
| 5,176 | 3.65 | 0.3092 | 0.8446 |
| 20,704 | 14.60 | 0.0080 | 0.2518 |
| 82,816 | 58.42 | 0.0000 | 0.0000 |
At the size of the survey, BIC drops the real curvature five times in six; at four times the size, one time in four; at sixteen times, essentially never. AIC drops it less often at every size. Consistency is a statement about the limit, and the limit can be far away.
AIC over a nested sequence
With more than one larger model on offer, AIC can overfit by one coefficient, or two, or more. The limiting probabilities have a simple description.
Theorem 4 (AIC's limiting choices). Let models $j_0 \subset j_0 + 1 \subset \cdots \subset j_0 + R$ be nested, with fixed numbers of coefficients and $m_l = p_{j_0 + l} - p_{j_0 + l - 1}$, let model $j_0$ be correct, and let the smaller candidates, if any, satisfy conditions 2 and 3 of Theorem 2. Let $G_1, \ldots, G_R$ be independent, $G_l$ chi-squared with $m_l$ degrees of freedom, and define the random walk $$W_0 = 0, \qquad W_r = \sum_{l=1}^{r}(2m_l - G_l).$$ Then $P(\text{AIC chooses } j_0 + r) \to P(\operatorname{argmin}_{0 \le s \le R}W_s = r)$ for each $r$, and the probability of choosing a model smaller than $j_0$ tends to 0.
Proof. The smaller models are handled by the underfitting step of Theorem 2, which needs only $c_n/n \to 0$. For the larger ones, let $G_l = \lVert(H_{j_0 + l} - H_{j_0 + l - 1})\boldsymbol\varepsilon\rVert^2/\sigma^2$. The matrices $H_{j_0 + l} - H_{j_0 + l - 1}$ are projections of ranks $m_l$ (Theorem 4 of The Geometry of Least Squares). Applied to every nested pair, that theorem gives $H_{l'}H_l = H_lH_{l'} = H_l$ for $l \le l'$ (indices counted from $j_0$), so for $l < l'$, $(H_{l'} - H_{l'-1})(H_l - H_{l-1}) = H_l - H_{l-1} - H_l + H_{l-1} = 0$: every pair multiplies to zero, not only consecutive ones. They also annihilate $\boldsymbol\mu$, so by Sampling Distributions from the Normal (Theorem 2) the $G_l$ are independent chi-squared variables with $m_l$ degrees of freedom, for every $n$. With $S_r = G_1 + \cdots + G_r$, the nested decomposition gives $\mathrm{RSS}_{j_0} = \mathrm{RSS}_{j_0 + r} + \sigma^2S_r$, so $$\mathrm{AIC}_{j_0 + r} - \mathrm{AIC}_{j_0} = 2\sum_{l \le r}m_l - n\ln\Big(1 + \frac{S_r}{n s_{n,r}}\Big), \qquad s_{n,r} = \frac{\mathrm{RSS}_{j_0 + r}}{n\sigma^2}.$$ For $u \ge 0$, $u - u^2/(2n) \le n\ln(1 + u/n) \le u$. Apply it with $u = S_r/s_{n,r}$: since $s_{n,r} \to 1$ in probability (the fact in the proof of Theorem 2) and $S_r$ has a fixed distribution, $n\ln(1 + S_r/(ns_{n,r})) - S_r \to 0$ in probability. So the vector of differences equals $(W_0, \ldots, W_R)$, built from the same $G_l$, plus a vector that tends to 0 in probability. The walk's values are distinct with probability 1 (each step has a continuous distribution), so for every $\epsilon > 0$ there is $\eta > 0$ with $P(\text{the two smallest } W_s \text{ differ by less than } 2\eta) < \epsilon$; and once every error is below $\eta$ in absolute value, the minimizer of the differences is the minimizer of $W$. Hence the two probabilities differ by at most $\epsilon$ plus a quantity tending to 0. $\square$
For the blood-pressure polynomials with the quadratic true and degrees 3 to 8 offered, $R = 6$ and every $m_l = 1$. A Monte Carlo of the random walk (30 seeds of 200,000 walks each) gives AIC's limiting chance of choosing the true degree as 0.729 (interquartile range across seeds 0.728 to 0.730), and its expected number of superfluous coefficients as 0.63. Woodroofe (1982) used generalizations of the arc sine laws to study selection among nested models; his abstract states that, as a corollary, "a popular technique may be expected to include about one superfluous parameter, even if the sample size is large." With six extra degrees on offer, the expected number here is 0.63. Akaike's Information Criterion, Derived compares the same random-walk limit, for a line as the truth, with AIC's overfitting rates at finite sizes.
Consistency and efficiency: AIC against BIC
The two criteria are best compared on what each guarantees. Shao (1997) makes both properties precise for linear models fitted by least squares. Write $L_n(j) = \lVert\boldsymbol\mu - H_j\mathbf{y}\rVert^2/n$ for the loss of model $j$, the average squared error of its fitted means. A selection rule $\hat\jmath$ is:
- consistent if $P(\hat\jmath = j_n^L) \to 1$, where $j_n^L$ minimizes $L_n$ over the candidates (his definition (2.1)); when a correct model of fixed dimension exists, this is the smallest correct model, as in Theorem 2;
- asymptotically loss efficient if $L_n(\hat\jmath)/L_n(j_n^L) \to 1$ in probability (his (2.3)): the chosen model predicts asymptotically as well as the best candidate, whether or not it is the best.
Consistency implies efficiency. The converse fails, and the two criteria part company exactly there. Shao's results, for criteria of the form $\mathrm{RSS}_j + \lambda_n\hat\sigma^2p_j$ (which includes Mallows' $C_p$ for $\lambda_n = 2$):
- With $\lambda_n = 2$ (the class of AIC and $C_p$) and a consistent estimate $\hat\sigma^2$, the rule is asymptotically loss efficient when the family contains at most one correct model (his Theorem 1(i)). In his Example 1, a fixed set of candidate regressions, it is consistent only if the full model is the only correct one. In his polynomial-approximation example, where no candidate is exactly correct and the best dimension grows with $n$, it is asymptotically loss efficient, while rules with $\lambda_n \to \infty$ "may not" be.
- With $\lambda_n \to \infty$ and $\lambda_n/n \to 0$ (the class of BIC), the rule is consistent whenever a correct model of fixed dimension exists (Theorem 2(ii)), but it can fail to be efficient when there is none.
Shao summarizes that the procedures in one class "are asymptotically valid if there exist fixed-dimension correct models", those in another "if no fixed-dimension correct model exists", and the third class compromises. He also reports earlier results: Nishii (1984) and Rao and Wu (1989) showed BIC and GIC consistent and AIC and $C_p$ not in his Example 1, while Shibata (1981) and Li (1987) showed AIC, $C_p$ and leave-one-out cross-validation "asymptotically correct in some sense" in the polynomial-approximation case.
Can one criterion have both properties? Yang (2005) shows that it cannot, in a minimax sense. His abstract recalls, as background, that "BIC is consistent in selecting the true model, and AIC is minimax-rate optimal for estimating the regression function," and his result is that any consistent model selection criterion must behave suboptimally for estimating the regression function in the minimax rate, even with a penalty that adapts to the data. Hastie, Tibshirani and Friedman (2009, §7.7) put the practical upshot plainly: "there is no clear choice between AIC and BIC"; BIC is consistent, AIC tends to choose models that are too complex as $n$ grows, and for finite samples BIC often chooses models that are too simple.
The two simulations below measure each half of that statement on data built from the NHANES adults. In both, each data set draws $n$ ages with replacement from the 5,176 adults, adds normal errors to a known mean function, and fits polynomials by least squares. There are 200 data sets per sample size for each of 30 seeds (seeds 1 to 30 for the first simulation, 1,001 to 1,030 for the second), and each table entry is the median over the seeds of a share or average over the 200.
Simulation: the truth among the candidates
The truth is the population quadratic of the example ($\sigma = 17.39$), and the candidates are degrees 0 to 8. The table gives how often each criterion chose degree 2 (right), a lower degree (under) or a higher one (over).
| $n$ | AIC right | AIC under | AIC over | BIC right | BIC under | BIC over |
|---|---|---|---|---|---|---|
| 100 | 0.118 | 0.700 | 0.178 | 0.038 | 0.955 | 0.008 |
| 400 | 0.150 | 0.668 | 0.185 | 0.025 | 0.970 | 0.000 |
| 1,600 | 0.258 | 0.545 | 0.198 | 0.043 | 0.955 | 0.000 |
| 6,400 | 0.555 | 0.193 | 0.255 | 0.203 | 0.798 | 0.000 |
| 25,600 | 0.730 | 0.000 | 0.268 | 0.860 | 0.138 | 0.000 |
| 102,400 | 0.738 | 0.000 | 0.263 | 1.000 | 0.000 | 0.000 |
At the largest size, AIC chose the true degree in a median 0.738 of the data sets (interquartile range 0.711 to 0.755 across seeds), close to the random-walk limit of 0.729 from Theorem 4, and BIC chose it in all of them (interquartile range 1.000 to 1.000). Both theorems are visible: BIC's chance of being right climbed toward 1, AIC's settled near its limit and stayed there. So did the warning of the example. Up to $n = 1{,}600$, BIC chose a degree below 2 in about 95% of the data sets, and at $n = 6{,}400$, larger than the survey, still in 79.8%. In this range AIC was right more often, because its underfitting fell first.
Simulation: the truth outside the candidates
Now the truth is the population's own mean pressure at each of the 63 single years of age, a jagged function that no polynomial of degree at most 10 reproduces. The errors are normal with the pooled within-age standard deviation, 17.36 mm Hg, and the candidates are degrees 0 to 10. No candidate is correct, so consistency has nothing to aim at, and the question is efficiency: the ratio $L_n(\hat\jmath)/\min_jL_n(j)$ of the chosen model's loss to the best candidate's, averaged over the 200 data sets.
| $n$ | Best degree | AIC degree | BIC degree | AIC loss ratio (IQR) | BIC loss ratio (IQR) |
|---|---|---|---|---|---|
| 100 | 1.0 | 1.0 | 1.0 | 1.890 (1.801–2.000) | 1.114 (1.085–1.156) |
| 400 | 2.0 | 1.0 | 1.0 | 1.334 (1.313–1.364) | 1.056 (1.047–1.064) |
| 1,600 | 5.0 | 5.0 | 1.0 | 1.149 (1.141–1.157) | 1.159 (1.158–1.162) |
| 6,400 | 8.0 | 8.0 | 1.0 | 1.032 (1.032–1.034) | 1.248 (1.238–1.255) |
| 25,600 | 10.0 | 10.0 | 5.0 | 1.005 (1.004–1.005) | 1.060 (1.059–1.061) |
(The degree columns are medians over data sets, then over seeds.) As $n$ grew, the best degree grew with it, AIC's choice tracked it, and its loss ratio fell toward 1, as Shao's efficiency result leads one to expect. BIC lagged behind by several degrees and its ratio did not fall steadily. The small samples tell the other half. At $n = 100$ and 400 the best model was the line or the quadratic, and the median choice of both criteria was the line, yet AIC's average loss ratio was 1.890 and 1.334 against BIC's 1.114 and 1.056. Where the two choose the same degree their losses are equal, so AIC's excess came from the data sets in which it chose a higher degree than BIC. (AIC's choice is never smaller than BIC's when $n > e^2$: writing $f(k) = n\ln(\mathrm{RSS}/n)$ for the model with $k$ parameters, and $k_A$, $k_B$ for the two choices, $f(k_A) + 2k_A \le f(k_B) + 2k_B$ and $f(k_B) + k_B\ln n \le f(k_A) + k_A\ln n$; adding them gives $(\ln n - 2)(k_B - k_A) \le 0$.) In this experiment neither criterion dominated: BIC was the safer choice for small samples and AIC the better one once there was enough data to estimate curvature.
The criteria on the NHANES adults
Back to the 5,176 adults. With $\ln 5{,}176 = 8.552$, BIC charges each coefficient more than four times what AIC charges ($8.552/2 = 4.28$).
| Degree | $\mathrm{RSS}/n$ | AIC − min | BIC − min |
|---|---|---|---|
| 0 | 396.07 | 1,406.03 | 1,385.81 |
| 1 | 302.53 | 13.67 | 0.00 |
| 2 | 302.32 | 12.02 | 4.90 |
| 3 | 302.05 | 9.33 | 8.76 |
| 4 | 301.93 | 9.36 | 15.34 |
| 5 | 301.27 | 0.00 | 12.53 |
| 6 | 301.20 | 0.89 | 19.97 |
| 7 | 301.20 | 2.87 | 28.51 |
| 8 | 301.05 | 2.15 | 34.34 |
Going from the line to the quintic lowers $\mathrm{RSS}/n$ by 1.26 mm Hg$^2$ against an error variance near 302, a reduction of 0.4%. The $F$ test of the four extra coefficients gives $F = 5.42$ on 4 and 5,170 degrees of freedom, $p = 0.0002$. AIC accepts them; BIC, which needs each to pay $\ln n$, refuses. Both answers are coherent once we know what each criterion is for. If the question is whether the dependence of blood pressure on age departs from a straight line, the test and AIC say yes, and the departure is real but small. If the question is which model, among those that might be exactly true, is most probable, BIC prefers the line, by $\exp\{(12.53 - 0)/2\} = e^{6.3}$ to one against the quintic under its own approximation, with the order-one error that Bayes Factors measures. Neither is the population's true mean function, which is the jagged curve of the second simulation. For predicting new adults, the cross-validation of What Cross-Validation Estimates also chooses degree 5, as AIC does.
What is proved here, what is cited, and what the experiment shows
Proved on this page: Laplace's method in one dimension with relative error $O(1/n)$, and its uniformity in the bounds it uses (Theorem 1); the consistency of every criterion $n\ln(\mathrm{RSS}/n) + c_nk$ with $c_n \to \infty$, $c_n/n \to 0$, BIC among them, for a finite family of normal linear models with a unique smallest correct model (Theorem 2); the exact probabilities with which any penalty, BIC's included, prefers the larger or the smaller of two nested normal models, through the central or noncentral $F$ distribution (Theorem 3, whose AIC case with the smaller model correct is Theorem 3 of Akaike's Information Criterion, Derived); the limit of AIC's choices over a nested sequence as the minimizer of a chi-squared random walk (Theorem 4).
Derived but not bounded: the $1/n$ coefficient of Laplace's method.
Taken from other pages: BIC from Laplace's approximation and its order-one error (Bayes Factors, Corollary 2); AIC's derivation and its limiting overfitting probability $P(\chi^2_m > 2m)$ (Akaike's Information Criterion, Derived); nested projections and the hat matrix (The Geometry of Least Squares); independence and chi-squared distributions of projections (Sampling Distributions from the Normal); the mean of a quadratic form (Covariance Matrices and Quadratic Forms); Markov's and Chebyshev's inequalities (Probability Inequalities).
Cited: the $d$-dimensional Laplace approximation and its conditions (Kass and Raftery, 1995, §4.1.1, citing Kass, Tierney and Kadane, 1990); Shao's (1997) definitions and Theorems 1 and 2, and the earlier results he reports; Nishii's (1984) and Woodroofe's (1982) results, at the strength of their abstracts; Yang's (2005) impossibility result, at the strength of its abstract; the conclusion of Hastie et al. (2009, §7.7).
Computed: every number on the page, by tools/experiments/statistics/bayesian-information-criterion.py. The simulations are observations of those experiments, with seeds stated: they show the theorems at work at particular sizes and say nothing about rates in general.
Worked problems
These problems are posed for this article, on its data and on small exact cases. Every number is recomputed by the article's script.
Problem 1 (an exact overfitting probability). Thirty observations follow a straight line with normal errors. AIC and BIC each compare the line with the quadratic. How often does each choose the quadratic?
Solution. By Theorem 3 with $n = 30$, $p_0 = 2$, $p_1 = 3$, $m = 1$, the line is correct and $F$ has the central $F$ distribution with 1 and 27 degrees of freedom. AIC chooses the quadratic when $F > \tau(2) = 27(e^{2/30} - 1)$. Here $e^{1/15} = 1.068939$, so $\tau(2) = 1.8614$, and $P(F_{1,27} > 1.8614) = 0.1837$. BIC chooses it when $F > \tau(\ln 30) = 27(e^{\ln 30/30} - 1) = 27(30^{1/30} - 1)$. Here $30^{1/30} = 1.120050$, so $\tau = 3.2413$, and $P(F_{1,27} > 3.2413) = 0.0830$. So AIC adds the useless quadratic term in about 18% of samples of this size and BIC in about 8%. Neither probability depends on the values of the inputs.
Problem 2 (Laplace by hand). Three successes in ten trials, uniform prior. Compute the averaged probability of the sequence exactly and by Theorem 1, and compare the ratio with the $1/n$ correction.
Solution. Exactly, $m = \mathrm{B}(4, 8) = 3!\,7!/11! = 1/1{,}320 = 0.0007576$. For Laplace, $h(\theta) = 0.3\ln\theta + 0.7\ln(1 - \theta)$ is maximized at $\hat\theta = 0.3$ with $a = 1/0.21$, so $$\hat m = 0.3^3\,0.7^7\sqrt{\frac{2\pi \times 0.21}{10}} = 0.0022236 \times 0.36324 = 0.0008077.$$ The ratio is $m/\hat m = 0.9379$. The two-term expansion predicts $1 + (-0.6865)/10 = 0.9313$, closer than the leading term alone. With $n = 10$ the neglected terms of order $1/n^2$ still show.
Problem 3 (what BIC demands of one coefficient). For one added coefficient, express BIC's threshold in terms of the $t$ statistic, and compare it with a 5% test, at $n = 442$ (the diabetes data of Efron, Hastie, Johnstone and Tibshirani, 2004, used in Model Selection) and at $n = 5{,}176$.
Solution. For $m = 1$ the $F$ statistic is the square of the $t$ statistic of the added coefficient (The General Linear Hypothesis), so by Theorem 3 BIC adds the coefficient when $\lvert t\rvert > \sqrt{\tau(\ln n)}$, with $\tau(\ln n) = (n - p_1)(n^{1/n} - 1)$. For a model with $p_1 = 7$ coefficients, $\sqrt{\tau} = 2.457$ at $n = 442$ and 2.924 at $n = 5{,}176$, almost exactly $\sqrt{\ln n}$, which is 2.468 and 2.924 (because $(n - p_1)(n^{1/n} - 1) \approx \ln n$ when $p_1$ and $\ln n$ are small next to $n$). A two-sided 5% test asks for $\lvert t\rvert > 1.96$ or so. In normal terms, BIC's bar corresponds to a two-sided level of $P(\lvert Z\rvert > \sqrt{\ln n})$, which is 0.0136 at $n = 442$ and 0.0035 at $n = 5{,}176$: the level of the implied test falls as the sample grows, which is how BIC achieves consistency, and it is why BIC passed over the quintic of the NHANES adults despite $p = 0.0002$ for the four coefficients together.
Summary
| Result | Statement |
|---|---|
| Laplace's method | $\int\pi e^{nh} = e^{nh(\hat\theta)}\pi(\hat\theta)\sqrt{2\pi/(na)}\,(1 + O(1/n))$, $a = -h''(\hat\theta)$ |
| BIC (normal linear model) | $n\ln(\mathrm{RSS}/n) + k\ln n$; consistent for the smallest correct model (Theorem 2) |
| General consistent penalties | $c_n \to \infty$, $c_n/n \to 0$ |
| Preference for the larger nested model | $F > (n - p_1)(e^{cm/n} - 1)/m$; central $F$ if the smaller is correct |
| AIC's limit, $m$ extra | $P(\chi^2_m > 2m)$; 0.1573 for $m = 1$ |
| AIC over a nested sequence | argmin of $W_r = \sum_{l \le r}(2m_l - \chi^2_{m_l})$ |
| Consistency vs efficiency | BIC-type: consistent with a fixed correct model; AIC-type: efficient without one (Shao, 1997); not both (Yang, 2005) |
What we have learned
Laplace's method turns a marginal likelihood into the height of its peak times a normal width, with a relative error of order $1/n$ because the first corrections are odd and cancel. BIC keeps only the part of that approximation that grows with $n$, so it is a poor estimate of a marginal likelihood, but it is a consistent selector: its penalty $\ln n$ outgrows the order-one gain of any superfluous column and stays below the order-$n$ loss of any missing one. AIC's fixed penalty of 2 lets superfluous columns in with a probability that never vanishes, exactly computable from the $F$ distribution and, over a nested sequence, given by a random walk. In exchange, AIC asymptotically predicts as well as the best candidate when no candidate is true (Shao, 1997, under his conditions), which BIC need not do; the simulation with the truth outside the candidates shows the reverse at small samples. On the blood-pressure data the two answer different questions, and both answers are right.
AIC, $C_p$ and leave-one-out cross-validation belong to the same class. What Cross-Validation Estimates asks what a cross-validation estimate actually measures, proves the leave-one-out shortcut for a wide class of smoothers, and shows why its standard error is too small.
References
- de Bruijn, N. G. (1970). Asymptotic Methods in Analysis. North-Holland. §4.4. Cited as Kass and Raftery (1995) cite it. record
- Efron, B., Hastie, T., Johnstone, I., and Tibshirani, R. (2004). Least angle regression. The Annals of Statistics, 32(2), 407–499. https://doi.org/10.1214/009053604000000067 record · 18 pages
- Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2nd ed.). Springer. §7.7. record · 31 pages
- Kass, R. E., and Raftery, A. E. (1995). Bayes factors. Journal of the American Statistical Association, 90(430), 773–795. https://doi.org/10.1080/01621459.1995.10476572 record · 3 pages
- Kass, R. E., Tierney, L., and Kadane, J. B. (1990). The validity of posterior expansions based on Laplace's method. In S. Geisser, J. S. Hodges, S. J. Press and A. Zellner (eds.), Bayesian and Likelihood Methods in Statistics and Econometrics: Essays in Honor of George A. Barnard (pp. 473–488). North-Holland. Cited as Kass and Raftery (1995) cite it. record
- Li, K.-C. (1987). Asymptotic optimality for $C_p$, $C_L$, cross-validation and generalized cross-validation: discrete index set. The Annals of Statistics, 15(3), 958–975. https://doi.org/10.1214/aos/1176350486 Cited as Shao (1997) reports it. record
- National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018: Demographic Variables and Sample Weights (DEMO_J), Blood Pressure (BPX_J) and Body Measures (BMX_J) data files. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 95 pages
- Nishii, R. (1984). Asymptotic properties of criteria for selection of variables in multiple regression. The Annals of Statistics, 12(2), 758–765. https://doi.org/10.1214/aos/1176346522 record
- Rao, C. R., and Wu, Y. (1989). A strongly consistent procedure for model selection in a regression problem. Biometrika, 76(2), 369–374. https://doi.org/10.1093/biomet/76.2.369 Cited as Shao (1997) reports it. record
- Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6(2), 461–464. https://doi.org/10.1214/aos/1176344136 record · 5 pages
- Shao, J. (1997). An asymptotic theory for linear model selection. Statistica Sinica, 7(2), 221–264. record · 2 pages
- Shibata, R. (1981). An optimal selection of regression variables. Biometrika, 68(1), 45–54. https://doi.org/10.1093/biomet/68.1.45 Cited as Shao (1997) reports it. record
- Tierney, L., and Kadane, J. B. (1986). Accurate approximations for posterior moments and marginal densities. Journal of the American Statistical Association, 81(393), 82–86. https://doi.org/10.1080/01621459.1986.10478240 Cited as Kass and Raftery (1995) cite it. record
- Woodroofe, M. (1982). On model selection and the arc sine laws. The Annals of Statistics, 10(4), 1182–1194. https://doi.org/10.1214/aos/1176345983 record
- Yang, Y. (2005). Can the strengths of AIC and BIC be shared? A conflict between model indentification and regression estimation. Biometrika, 92(4), 937–950. https://doi.org/10.1093/biomet/92.4.937 record