Twenty-five adults from the National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018, aged 19 to 80, have their systolic pressures fitted by polynomials in age with normal errors. Akaike's information criterion (AIC) prefers the cubic; its small-sample correction, AICc, prefers the straight line; and in the population the 25 were drawn from, a straight line describes the mean pressure at each age almost as well as any of these polynomials (the variance about the line is 302.53, about the best sixth-degree polynomial 301.20). Where does the criterion's penalty of two per parameter come from, what does the criterion estimate, and why does it lean toward models that are too large?
AIC for a model fitted by maximum likelihood is $-2$ times the maximized log-likelihood plus twice the number of estimated parameters (Akaike, 1974). Model Selection introduced it as the likelihood version of Mallows' $C_p$, and Optimism and Effective Degrees of Freedom proved that $C_p$ corrects the optimism of the training error. This article derives AIC the same way, with the negative log-likelihood of new data as the loss: the maximized log-likelihood is too large, as an estimate of how well the fitted model will predict, by an amount whose expectation is about one per parameter.
The route is as follows. First the target, an expected Kullback–Leibler divergence, and why the maximized log-likelihood estimates it with a bias. Then the normal linear model, where the bias can be computed exactly, giving AICc (Theorem 1). Then the general case with one parameter, where the bias variable converges in distribution to a limit whose mean is twice a ratio of two information quantities, which is 2 for a correct model and in general different for a wrong one (Theorem 2). Then what AIC does not estimate, and its tendency to choose too large a model, proved for nested normal models (Theorem 3) and measured by simulation. The article assumes Cross-Entropy and Kullback–Leibler Divergence, Asymptotics of the Maximum Likelihood Estimator and The Normal Linear Model.
The target: an expected divergence
Data $\mathbf{y}$ are generated by a true density $g$. A candidate model is a family of densities $f(\mathbf{y}; \boldsymbol\theta)$ with $k$ free parameters, fitted by maximum likelihood to give $\hat{\boldsymbol\theta}$ with maximized log-likelihood $\hat\ell = \ln f(\mathbf{y}; \hat{\boldsymbol\theta})$ (natural logarithms throughout). How good is the fitted density $f(\cdot\,; \hat{\boldsymbol\theta})$ as a description of $g$?
The Kullback–Leibler divergence answers with one number:
$$D\big(g \,\|\, f_{\hat{\boldsymbol\theta}}\big) = E_g\big[\ln g(\mathbf{y}^0)\big] - E_g\big[\ln f(\mathbf{y}^0; \hat{\boldsymbol\theta})\big],$$
where $\mathbf{y}^0$ is a new data set from $g$, independent of the $\mathbf{y}$ that produced $\hat{\boldsymbol\theta}$, and the expectations are over $\mathbf{y}^0$ with $\hat{\boldsymbol\theta}$ held fixed. The divergence is nonnegative and zero only when the fitted density is $g$ (Gibbs' inequality, proved for mass functions in Cross-Entropy and Kullback–Leibler Divergence and for densities in Differential Entropy and the Gaussian). Its first term involves only $g$ and is the same for every candidate. So comparing candidates by divergence is comparing them by the second term, the expected log-likelihood of new data. Multiplied by $-2$ and averaged over the training data, that is the target
$$T = E_{\mathbf{y}}\Big[E_{\mathbf{y}^0}\big[-2\ln f(\mathbf{y}^0; \hat{\boldsymbol\theta}(\mathbf{y}))\big]\Big] = 2E_{\mathbf{y}}\big[D(g \,\|\, f_{\hat{\boldsymbol\theta}})\big] - 2E_g[\ln g(\mathbf{y}^0)].$$
A smaller $T$ means a smaller expected divergence. This is the quantity Akaike's criterion is designed to estimate (Hastie, Tibshirani and Friedman, 2009, eq. 7.27, write the same relation per observation; Ding, Tarokh and Yang, 2018, describe AIC as motivated by the candidate closest to the truth in Kullback–Leibler divergence).
The obvious estimate of $T$ replaces the new data by the old: $-2\hat\ell$. It is too small on average for the reason training error is too small. The estimate $\hat{\boldsymbol\theta}$ was chosen to make $\ln f(\mathbf{y}; \boldsymbol\theta)$ as large as possible for these data, so it fits them better than it fits new data. The bias to be corrected is
$$B = T - E\big[-2\hat\ell\big].$$
AIC estimates $B$ by $2k$. Where that comes from, and when it is right, is the subject of the next two sections.
The normal linear model, exactly
Here is the idea. In the normal linear model with the inputs fixed, the fitted coefficients and the residual variance are independent, with known chi-squared laws. The bias then reduces to the mean of a reciprocal chi-squared variable, which is known exactly.
The model is $\mathbf{y} = X\boldsymbol\beta + \boldsymbol\varepsilon$ with fixed $X$ of size $n \times p$ and rank $p$, and independent normal errors of mean 0 and variance $\sigma^2$. The $k = p + 1$ parameters are $\boldsymbol\beta$ and $\sigma^2$, and the density is that of $N_n(X\boldsymbol\beta, \sigma^2 I)$, so
$$-2\ln f(\mathbf{y}; \boldsymbol\beta, \sigma^2) = n\ln(2\pi\sigma^2) + \frac{\lVert\mathbf{y} - X\boldsymbol\beta\rVert^2}{\sigma^2}.$$
The maximum likelihood estimates are the least-squares coefficients $\hat{\boldsymbol\beta}$ and $\hat\sigma^2 = \mathrm{RSS}/n$, with divisor $n$, so $-2\hat\ell = n\ln(2\pi\hat\sigma^2) + n$ and
$$\mathrm{AIC} = n\ln(2\pi\hat\sigma^2) + n + 2k,$$
the form of Model Selection with the constant $n(1 + \ln 2\pi)$ kept. The new data $\mathbf{y}^0$ are a fresh response vector at the same inputs, as for the in-sample error of the optimism article.
Theorem 1 (the exact bias; AICc). Suppose the truth is $\mathbf{y} \sim N_n(\boldsymbol\mu, \sigma_0^2 I)$ with $\boldsymbol\mu = X\boldsymbol\beta_0$ for some $\boldsymbol\beta_0$, so that the candidate contains the truth, and suppose $n > k + 1$. Then
$$B = \frac{2kn}{n - k - 1}, \qquad\text{so}\qquad \mathrm{AICc} = -2\hat\ell + \frac{2kn}{n - k - 1} = \mathrm{AIC} + \frac{2k(k+1)}{n - k - 1}$$
has $E[\mathrm{AICc}] = T$ exactly.
Proof. Fix $\mathbf{y}$, and so $\hat{\boldsymbol\beta}$ and $\hat\sigma^2$. Averaging over $\mathbf{y}^0$,
$$E_{\mathbf{y}^0}\big[\lVert\mathbf{y}^0 - X\hat{\boldsymbol\beta}\rVert^2\big] = E\lVert\mathbf{y}^0 - \boldsymbol\mu\rVert^2 + \lVert\boldsymbol\mu - X\hat{\boldsymbol\beta}\rVert^2 = n\sigma_0^2 + \lVert X(\hat{\boldsymbol\beta} - \boldsymbol\beta_0)\rVert^2,$$
because $\mathbf{y}^0 - \boldsymbol\mu$ has mean zero and is independent of the fixed vector $\boldsymbol\mu - X\hat{\boldsymbol\beta}$. Hence
$$E_{\mathbf{y}^0}\big[-2\ln f(\mathbf{y}^0; \hat{\boldsymbol\beta}, \hat\sigma^2)\big] - (-2\hat\ell) = \frac{n\sigma_0^2 + \lVert X(\hat{\boldsymbol\beta} - \boldsymbol\beta_0)\rVert^2}{\hat\sigma^2} - n.$$
The logarithm terms cancel exactly, since both are $n\ln(2\pi\hat\sigma^2)$. Now $X(\hat{\boldsymbol\beta} - \boldsymbol\beta_0) = H\boldsymbol\varepsilon$ and $n\hat\sigma^2 = \lVert(I - H)\boldsymbol\varepsilon\rVert^2$, with $H$ the hat matrix. By Theorem 2 of Sampling Distributions from the Normal applied to the projections $H$ and $I - H$ (ranks $p$ and $n - p$), $Q_1 = \lVert H\boldsymbol\varepsilon\rVert^2/\sigma_0^2$ and $Q_2 = n\hat\sigma^2/\sigma_0^2$ are independent chi-squared variables with $p$ and $n - p$ degrees of freedom; this is also Theorem 1 of The Normal Linear Model. The bias is the expectation of
$$\frac{n\sigma_0^2 + \sigma_0^2Q_1}{\sigma_0^2Q_2/n} - n = \frac{n(n + Q_1)}{Q_2} - n.$$
By independence, $E[(n + Q_1)/Q_2] = (n + p)\,E[1/Q_2]$, and $E[1/Q_2] = 1/(n - p - 2)$ by Lemma 1 of Hierarchical Models and Shrinkage, which needs $n - p > 2$, that is $n > k + 1$. So
$$B = \frac{n(n + p)}{n - p - 2} - n = \frac{n(2p + 2)}{n - p - 2} = \frac{2kn}{n - k - 1},$$
using $k = p + 1$. Finally $\frac{2kn}{n-k-1} - 2k = \frac{2k(k+1)}{n-k-1}$. $\square$
As $n$ grows with $k$ fixed, $2kn/(n - k - 1) \to 2k$: the AIC penalty is the large-sample limit of the exact one. For 25 observations and a straight line ($k = 3$) the exact bias is $150/21 = 7.14$ rather than 6, and for a sixth-degree polynomial ($k = 8$) it is 25.00 rather than 16. Hurvich and Tsai (1989) derived this correction for regression and autoregressive models and named it AICc; their abstract says it is "of particular use when the sample size is small, or when the number of fitted parameters is a moderate to large fraction of the sample size". Sugiura (1978) had proposed a finite correction of AIC earlier, according to his abstract. The form above agrees with the one Ding et al. (2018) give, $\mathrm{AIC} + 2(d + 1)(d + 2)/(n - d - 2)$, when their $d$ is read as the number of regression coefficients, $k - 1$, as in Hurvich and Tsai's regression setting.
The bias measured on blood pressure
The truth for the experiments of this article is a straight line. In the 5,176 NHANES adults of the Linear Models unit (National Center for Health Statistics, 2020), the least-squares line of systolic pressure on age has intercept 100.2 and slope 0.522 mm Hg per year, and the variance of the residuals about it is 302.53 (divisor $N$; standard deviation 17.39). Simulated responses are that line plus independent normal errors with that variance. The candidates are polynomials of degree 0 to 6 in age (columns $1, z, \ldots, z^6$ with $z = (\text{age} - 49)/18$), so $k$ runs from 2 to 8. Degree 0 omits the true slope, and every other degree contains the truth.
For each of 30 seeds, 25 adults were drawn to fix the ages, and 4,000 response vectors were simulated. For each fit the inner expectation over $\mathbf{y}^0$ was computed exactly, as in the first display of the proof, so only the average over $\mathbf{y}$ is simulated. A second run drew the responses instead from the real pressures of the population's adults of each age, as in the optimism article, where the variance rises with age from 124 at 18 to 494 at 80 and the distribution is skewed; the normal candidates are then wrong about the errors, and slightly wrong about the mean, since the population's mean pressure at each age is not exactly a polynomial (the variance about the best line is 302.53, about the age means 297.68). Theorem 1 does not apply to this run; it shows how far the bias moves when the model's assumptions fail in the way real data fail them. Medians over the 30 seeds:
| Degree | $k$ | AIC penalty $2k$ | Theorem 1, $\frac{2kn}{n-k-1}$ | Bias, normal truth | Bias, NHANES responses |
|---|---|---|---|---|---|
| 0 | 2 | 4 | 4.55 | 3.87 | 5.33 |
| 1 | 3 | 6 | 7.14 | 7.17 | 8.99 |
| 2 | 4 | 8 | 10.00 | 10.04 | 11.98 |
| 3 | 5 | 10 | 13.16 | 13.20 | 15.20 |
| 4 | 6 | 12 | 16.67 | 16.69 | 18.60 |
| 5 | 7 | 14 | 20.59 | 20.66 | 22.74 |
| 6 | 8 | 16 | 25.00 | 25.06 | 27.35 |
With normal responses, the measured bias agrees with Theorem 1 for every degree from 1 to 6; for degree 6 its median was 25.06 (interquartile range 24.87 to 25.25) against the exact 25.00. Degree 0, which misses the slope, has a smaller bias than the formula, 3.87 against 4.55: the theorem does not cover it, and its divergence is dominated by the missing slope, which the criterion sees through $\hat\ell$ anyway. With the real responses the bias exceeds the normal-truth value by 1.82 to 2.29 for degrees 1 to 6 and by 1.46 for degree 0 (for degree 6, median 27.35, interquartile range 25.28 to 28.74). The next section gives a heuristic for the size of the excess: when the model is wrong about the distribution, the penalty is no longer a count of parameters.
The general case: a ratio of informations
Here is the idea. Expand the log-likelihood around the parameter value that the model would choose with unlimited data. The maximized log-likelihood overshoots that value's log-likelihood by a quadratic form in the estimation error, and the expected log-likelihood of new data falls short of it by the same quadratic form. The bias is the sum of the two, and in large samples each has the mean of a chi-squared variable scaled by a ratio of two information quantities.
The theorem is stated and proved for one parameter, with independent observations $X_1, \ldots, X_n$ from a density or mass function $g$, and a candidate family $f(x; \theta)$ that need not contain $g$. Write $u(x; \theta) = \partial\ln f(x; \theta)/\partial\theta$ for the score of one observation, $\ell(\theta) = \sum_i\ln f(X_i; \theta)$, and
$$\eta(\theta) = E_g[\ln f(X; \theta)],$$
the expected log-likelihood of one new observation. The pseudo-true value $\theta^*$ is the maximizer of $\eta$, the member of the family closest to $g$ in divergence, since $D(g\,\|\,f_\theta) = E_g[\ln g(X)] - \eta(\theta)$. The target is $T = -2n\,E[\eta(\hat\theta)]$, the one-dimensional case of the definition above. Define the two information quantities at $\theta^*$,
$$\mathcal{J} = -E_g\!\left[\frac{\partial u(X; \theta^*)}{\partial\theta}\right], \qquad \mathcal{K} = E_g\big[u(X; \theta^*)^2\big].$$
When $g = f(\cdot\,; \theta_0)$ belongs to the family, $\theta^* = \theta_0$ and both equal the Fisher information $I(\theta_0)$, by the information identity of Fisher Information and the Cramér–Rao Bound. When the family is wrong they can differ, though they need not: a Poisson model fitted to counts that are not Poisson but have variance equal to their mean has $\mathcal{K}/\mathcal{J} = 1$ (Problem 3 below).
Conditions. The conditions C1 to C5 of Asymptotics of the Maximum Likelihood Estimator, with $g$ in place of $f(\cdot\,; \theta_0)$ and $\theta^*$ in place of $\theta_0$, and with C4 replaced by
- C4′. $E_g[u(X; \theta^*)] = 0$, $E_g[|\partial u(X; \theta^*)/\partial\theta|] < \infty$, $0 < \mathcal{J} < \infty$ and $\mathcal{K} < \infty$.
- C6. The data lie in the model's support, $P_g(X \in S) = 1$, and $E_g[|\ln f(X; \theta^*)|] < \infty$, so that $\eta(\theta^*)$ is finite.
The first part of C4′ says that $\theta^*$ is a stationary point of $\eta$; it holds when $\theta^*$ is an interior maximum and differentiation under the expectation is allowed. The proofs of Theorems 1 and 2 and of the curvature lemma on that page use C4 in exactly three places: the mean of the score is 0, the mean of its derivative is negative and finite, and the variance of the score is finite. Under C4′ they go through word for word, with $\mathcal{J}$ where the mean of $-\partial u/\partial\theta$ appears and $\mathcal{K}$ where the variance of $u$ appears. They give a sequence of roots $\hat\theta_n$ of the likelihood equation with $\hat\theta_n \xrightarrow{p} \theta^*$, $-\ell''(\theta_n')/n \xrightarrow{p} \mathcal{J}$ for any $\theta_n' \xrightarrow{p} \theta^*$, and
$$\sqrt{n}\,(\hat\theta_n - \theta^*) \xrightarrow{d} V, \qquad V \text{ normal with mean 0 and variance } \mathcal{K}/\mathcal{J}^2,$$
since $U(\theta^*)/\sqrt n \xrightarrow{d}$ normal with variance $\mathcal{K}$ and is divided by a quantity converging to $\mathcal{J}$. The variance $\mathcal{K}/\mathcal{J}^2$ is the sandwich form that Overdispersion and Quasi-Likelihood uses for wrong models.
Theorem 2 (the bias in large samples). Under these conditions, let
$$W_n = 2\big[\ell(\hat\theta_n) - \ell(\theta^*)\big] + 2n\big[\eta(\theta^*) - \eta(\hat\theta_n)\big].$$
Then $E[W_n] = T - E[-2\ell(\hat\theta_n)]$ whenever these expectations exist, which is the bias $B$ when $\hat\theta_n$ is the maximum likelihood estimate, and
$$W_n \xrightarrow{d} 2\,\frac{\mathcal{K}}{\mathcal{J}}\,Z^2, \qquad Z \text{ standard normal},$$
a limit with mean $2\mathcal{K}/\mathcal{J}$. When the model contains the truth the limit is $2Z^2$, twice a chi-squared variable with one degree of freedom, with mean 2.
Proof. The identity for the mean: $E[\ell(\theta^*)] = n\eta(\theta^*)$, because $\theta^*$ is a fixed number, so
$$E[W_n] = E[2\ell(\hat\theta_n)] - 2n\eta(\theta^*) + 2n\eta(\theta^*) - 2nE[\eta(\hat\theta_n)] = E[2\ell(\hat\theta_n)] + T = T - E[-2\ell(\hat\theta_n)].$$
The limit. Write $\delta_n = \hat\theta_n - \theta^*$, and work on the event that $\hat\theta_n$ solves the likelihood equation and $|\delta_n| < c$, whose probability tends to 1.
The first term. Taylor's theorem around $\hat\theta_n$, where $\ell'(\hat\theta_n) = 0$, gives a point $\bar\theta_n$ between $\theta^*$ and $\hat\theta_n$ with $\ell(\theta^*) = \ell(\hat\theta_n) + \tfrac12\ell''(\bar\theta_n)\delta_n^2$. So
$$2\big[\ell(\hat\theta_n) - \ell(\theta^*)\big] = \frac{-\ell''(\bar\theta_n)}{n}\,\big(\sqrt n\,\delta_n\big)^2,$$
and the first factor converges in probability to $\mathcal{J}$, by the curvature lemma, since $\bar\theta_n \xrightarrow{p} \theta^*$.
The second term. For a fixed $\delta$ with $|\delta| < c$, Taylor's theorem for each $x$, as in equation (1) of the asymptotics article, gives
$$\ln f(x; \theta^* + \delta) - \ln f(x; \theta^*) = \delta\,u(x; \theta^*) + \frac{\delta^2}{2}\frac{\partial u(x; \theta^*)}{\partial\theta} + \frac{\delta^3}{6}R(x),\qquad |R(x)| \le M(x).$$
Each term on the right has a finite expectation under $g$, by C4′ and C5. Taking expectations, $\eta(\theta^* + \delta) - \eta(\theta^*) = -\tfrac12\mathcal{J}\delta^2 + r(\delta)$ with $|r(\delta)| \le m|\delta|^3/6$, where $m = E_g[M(X)]$. With $\delta = \delta_n$,
$$2n\big[\eta(\theta^*) - \eta(\hat\theta_n)\big] = \mathcal{J}\big(\sqrt n\,\delta_n\big)^2 - 2n\,r(\delta_n), \qquad \big|2n\,r(\delta_n)\big| \le \frac{m}{3}\,\frac{\big|\sqrt n\,\delta_n\big|^3}{\sqrt n}.$$
The bound converges to 0 in probability, because $\sqrt n\,\delta_n$ converges in distribution and is divided by $\sqrt n$.
Together. $W_n = \big(\mathcal{J} + o_p(1)\big)(\sqrt n\,\delta_n)^2 + \mathcal{J}(\sqrt n\,\delta_n)^2 + o_p(1)$, where $o_p(1)$ stands for terms converging to 0 in probability. Since $\sqrt n\,\delta_n \xrightarrow{d} V$, the continuous mapping theorem and Slutsky's theorem (Convergence in Distribution and the Proof of the Central Limit Theorem) give $W_n \xrightarrow{d} 2\mathcal{J}V^2$. With $V = \sqrt{\mathcal{K}}\,Z/\mathcal{J}$, that is $2(\mathcal{K}/\mathcal{J})Z^2$, whose mean is $2\mathcal{K}/\mathcal{J}$. For a correct model $\mathcal{K} = \mathcal{J}$. $\square$
The theorem is about the consistent root $\hat\theta_n$ of the asymptotics article, which is the root nearest $\theta^*$ and need not be the maximum likelihood estimate that AIC uses. If the MLE equals that root with probability tending to 1, then the $W_n$ built from the MLE differs from this one with probability tending to 0 and has the same limit. This happens, for instance, when the likelihood equation has at most one root for every sample. With probability tending to 1 that root then exists and is a local maximum (Corollary 1 of the asymptotics article, read with $g$ and $\theta^*$ as above). A differentiable function on an interval whose only stationary point is a local maximum rises to it and falls after it, since its derivative cannot change sign elsewhere, so the root is then the global maximizer of $\ell$, the MLE.
Two steps separate the theorem from the statement "the bias tends to 2". First, convergence in distribution does not by itself make the means converge. That needs the sequence $W_n$ to be uniformly integrable, a condition on the tails of the estimator that we do not verify here; the derivations in the literature assume conditions that supply it. Theorem 1 shows that it holds, with an exact answer, in the normal linear model. Second, the theorem has one parameter. With $k$ parameters the same steps, with the matrices $\mathcal{J}$ and $\mathcal{K}$ in place of the numbers and the asymptotic normality of a vector estimator under a wrong model, lead to a limit with mean $2\operatorname{tr}(\mathcal{J}^{-1}\mathcal{K})$, which is $2k$ when the model is correct, since then $\mathcal{J} = \mathcal{K}$ and the trace is that of the $k \times k$ identity. We do not carry out that version: the asymptotic normality of a vector of maximum likelihood estimates is stated without proof on the asymptotics article, and its form under a wrong model is not proved on this site. The criterion with the penalty $2\operatorname{tr}(\hat{\mathcal{J}}^{-1}\hat{\mathcal{K}})$ is Takeuchi's information criterion (Takeuchi, 1976). Ding, Diao, Zhou and Tarokh (2021, §II.C) review its derivation and give the penalty in this form: per observation on the scale of $-\ln$, $n^{-1}\operatorname{tr}(V^{-1}J)$, with $V$ the negative expected Hessian and $J$ the expected outer product of the score, which is $2\operatorname{tr}(\mathcal{J}^{-1}\mathcal{K})$ on the scale of $-2\hat\ell$, and they note that it becomes AIC when the model is well specified. Ding et al. (2018) note that it was derived in a way that allows model misspecification but is rarely used because of its computational complexity.
Back to the blood pressures. A heuristic explains most of the excess in the NHANES column of the table. Apply Theorem 2 to the variance parameter of a normal model alone, as if the mean were known. One observation with error $e$ contributes $-\tfrac12\ln(2\pi\sigma^2) - e^2/(2\sigma^2)$ to the log-likelihood, so $u = -1/(2\sigma^2) + e^2/(2\sigma^4)$ and $\partial u/\partial\sigma^2 = 1/(2\sigma^4) - e^2/\sigma^6$. The pseudo-true value is $\sigma^{*2} = E_g[e^2]$, where $E_g[u] = 0$. There $\mathcal{J} = 1/(2\sigma^{*4})$ and $\mathcal{K} = \operatorname{Var}_g(e^2)/(4\sigma^{*8})$, so
$$\frac{\mathcal{K}}{\mathcal{J}} = \frac{\kappa - 1}{2}, \qquad \kappa = \frac{E_g[e^4]}{(E_g[e^2])^2},$$
the kurtosis of the errors. For normal errors $\kappa = 3$ and the ratio is 1. The NHANES pressures about their age means have pooled kurtosis 5.13, which suggests a penalty of $\kappa - 1 = 4.13$ for this parameter instead of 2, an excess of 2.13. Pooling counts the differences in variance between ages as if they were variation within one observation. With the ages fixed, as in the experiment, $\mathcal{K}$ averages $\operatorname{Var}(e_i^2)$ over the ages, and the population gives a penalty of 3.94, an excess of 1.94. In the experiment the excess over the normal-truth bias was between 1.82 and 2.29 for degrees 1 to 6, and 1.46 for degree 0, which also misses the slope. This is consistent with the heuristic but is not a test of it: the experiment has several parameters, fixed ages, unequal variances, a slightly wrong mean and $n = 25$, while Theorem 2 has one parameter and identically distributed observations. Since the candidates share the same error model, the excess shifts all of them by similar amounts.
Example (a misreading: "AIC is the probability that the model is true"). The criterion estimates $T$, an expected divergence up to a constant shared by the candidates. Only differences between candidates fitted to the same data mean anything, and a difference says which fitted model is expected to be closer to the truth, not which model is true. In the table above, the true model (degree 1) and every larger polynomial contain the truth; they differ in $T$ only because the larger ones estimate more parameters. A posterior probability of a model is a different quantity, the subject of Bayes Factors and Posterior Model Probabilities and The Bayesian Information Criterion.
What the criterion does not estimate
The target $T$ is defined for a model fixed before the data are seen, at the observed inputs. Three uses fall outside it.
The error of the model AIC chose. Choosing the candidate with the smallest AIC is a search, and the winner's AIC is optimistic about the winner, for the reason of the section "The price of choosing" of the optimism article. In the simulation of the next section at $n = 25$, the target of the model AIC chose exceeded its AIC by a median of 8.27 per data set (interquartile range over the 30 seeds 7.87 to 8.89). For a fixed straight line the expected excess would be $7.14 - 6 = 1.14$, from Theorem 1. This is optimism about the in-sample target $T$. Model Selection reports, from Hastie et al. (2009, §7.11.1), that AIC overestimated the test error of its chosen models in their simulations; that is a different target, the error at new inputs, and the two findings do not conflict. The criterion ranks candidates; it does not measure the chosen one.
Error at new inputs. $T$ predicts new responses at the observed inputs, the in-sample setting of the optimism article. Prediction at new inputs drawn from the population is the extra-sample error that What Cross-Validation Estimates treats.
A different loss. The divergence scores whole densities. A model can have the smaller divergence and the larger squared error of its predicted means, for example when it describes the error variance better. If squared error of predictions is what matters, $C_p$ or cross-validation on that loss is the matching estimate.
AIC chooses too large a model
Here is the idea. If a smaller model already contains the truth, adding $m$ useless parameters raises the maximized log-likelihood by a random amount whose doubled value is approximately chi-squared with $m$ degrees of freedom. AIC charges $2m$ for them. A chi-squared variable exceeds twice its degrees of freedom with a probability that does not vanish as $n$ grows, so neither does the chance of adding the parameters.
For nested normal linear models the probability is exact. Let a smaller model with $p_0$ columns sit inside a larger one with $p_1 = p_0 + m$ columns, with residual sums of squares $\mathrm{RSS}_0 \ge \mathrm{RSS}_1$. The difference of their AIC values is $n\ln(\mathrm{RSS}_0/\mathrm{RSS}_1) - 2m$, since $k$ differs by $m$.
Theorem 3 (the probability of adding $m$ useless parameters). Suppose the smaller model contains the truth, with normal errors. Let
$$F = \frac{(\mathrm{RSS}_0 - \mathrm{RSS}_1)/m}{\mathrm{RSS}_1/(n - p_1)}.$$
- AIC prefers the larger model exactly when $F > t_n = \frac{n - p_1}{m}\big(e^{2m/n} - 1\big)$, and $F$ has the $F$ distribution with $m$ and $n - p_1$ degrees of freedom. So the probability is $P(F_{m,\,n-p_1} > t_n)$.
- As $n \to \infty$ with $m$ and $p_1$ fixed, the probability tends to $P(\chi^2_m > 2m)$, which is positive.
Proof. (1) $n\ln(\mathrm{RSS}_0/\mathrm{RSS}_1) > 2m$ exactly when $\mathrm{RSS}_0/\mathrm{RSS}_1 > e^{2m/n}$, that is when $(\mathrm{RSS}_0 - \mathrm{RSS}_1)/\mathrm{RSS}_1 > e^{2m/n} - 1$; multiply both sides by $(n - p_1)/m$. The smaller model is the larger one with $m$ coefficients set to zero, a linear hypothesis that is true, so by Theorem 2 of The General Linear Hypothesis $F$ has the central $F$ distribution with $m$ and $n - p_1$ degrees of freedom.
(2) $F = (\chi^2_m/m)/(\chi^2_{n-p_1}/(n - p_1))$ with independent numerator and denominator. The denominator converges in probability to 1 by the weak law of large numbers (The Laws of Large Numbers), so by Slutsky's theorem $F \xrightarrow{d} \chi^2_m/m$. Since $e^{x} - 1 = x + O(x^2)$, $t_n \to 2$. Then $F - t_n + 2 \xrightarrow{d} \chi^2_m/m$ by Slutsky's theorem again, and since the limit has a continuous distribution function, $P(F > t_n) = P(F - t_n + 2 > 2) \to P(\chi^2_m/m > 2) = P(\chi^2_m > 2m)$. $\square$
The limits are $P(\chi^2_1 > 2) = 0.157$, $P(\chi^2_2 > 4) = 0.135$, $P(\chi^2_3 > 6) = 0.112$ and $P(\chi^2_5 > 10) = 0.075$. Against a single useless parameter, AIC behaves like a likelihood ratio test at level 0.157. A procedure that picks the true model with probability tending to 1 is called consistent; Theorem 3 shows that AIC is not. For likelihoods other than the normal, the same limit follows from Wilks's theorem for $m$ restrictions, which Likelihood Ratio, Wald and Score Tests states and cites; Ding et al. (2018) give this explanation of the inconsistency. Shibata (1976) obtained the asymptotic distribution of the order chosen by AIC in autoregressive models, as his abstract describes. Whether inconsistency matters, and how AIC and the Bayesian information criterion trade consistency against efficiency in prediction, is set out in The Bayesian Information Criterion and the Laplace Approximation, section "Consistency and efficiency: AIC against BIC".
The simulation. With the line as the truth and the seven candidates of degree 0 to 6, data sets of $n$ adults were drawn with replacement from the population's ages, normal responses simulated, and the degree chosen by AIC and by AICc. For each $n$, 30 seeds of 1,000 data sets each. The share choosing a degree above 1 (overfitting), as a median over seeds with its interquartile range for AIC, and the exact probability from Theorem 3 that AIC or AICc prefers degree 2 to degree 1 when only those two are compared:
| $n$ | AIC overfits (median, IQR) | AICc overfits (median, IQR) | Exact, degree 2 vs 1, AIC | Exact, degree 2 vs 1, AICc |
|---|---|---|---|---|
| 25 | 0.362 (0.344 to 0.379) | 0.169 (0.155 to 0.174) | 0.190 | 0.117 |
| 100 | 0.284 (0.276 to 0.292) | 0.242 (0.234 to 0.248) | 0.165 | 0.148 |
| 400 | 0.279 (0.266 to 0.285) | 0.269 (0.258 to 0.274) | 0.159 | 0.155 |
| 1,600 | 0.265 (0.259 to 0.277) | 0.265 (0.257 to 0.272) | 0.158 | 0.157 |
Three observations about this experiment.
- The share does not go to zero. From 100 to 1,600 adults, AIC chose too large a polynomial in 26.5% to 28.4% of data sets (medians 28.4%, 27.9% and 26.5% at 100, 400 and 1,600 adults). The limit for five nested extra terms is that of a chi-squared random walk: with independent chi-squared variables $Z_1^2, \ldots, Z_5^2$ of one degree of freedom, AIC adds $j$ terms when $\sum_{i \le j}(Z_i^2 - 2)$ is largest at $j$. A Monte Carlo of 2,000,000 such walks (seed 30,000,000) gives the right degree with probability 0.735 and too large a degree with probability 0.265. That AIC's choice converges to this walk's, with increments $\lVert(H_j - H_{j-1})\boldsymbol\varepsilon\rVert^2/\sigma^2$ that are exactly independent chi-squared variables for every $n$, is Theorem 4 of The Bayesian Information Criterion and the Laplace Approximation, section "AIC over a nested sequence"; the theorem covers AIC with the variance estimated.
- The share is larger than the pairwise probability, because AIC has five chances to overfit and takes the best of them, the price of choosing again.
- AICc helps only while $k$ is a sizable fraction of $n$. At 25 adults it halved the overfitting; by 400 the two medians differ by only 0.01, and by 1,600 they are equal, as the vanishing correction $2k(k+1)/(n - k - 1)$ leads one to expect. Underfitting was confined to the smallest sizes: AIC chose degree 0, a flat line, in 2,226 of the 30,000 data sets at $n = 25$, in 1 at $n = 100$, and never at the larger sizes.
The running sample. The 25 adults of the opening (seed 27,000,001; the youngest 19, the oldest 80) with their own measured pressures:
| Degree | $k$ | $-2\hat\ell$ | AIC | AICc |
|---|---|---|---|---|
| 0 | 2 | 219.31 | 223.31 | 223.86 |
| 1 | 3 | 211.81 | 217.81 | 218.96 |
| 2 | 4 | 209.55 | 217.55 | 219.55 |
| 3 | 5 | 206.55 | 216.55 | 219.71 |
| 4 | 6 | 205.42 | 217.42 | 222.08 |
| 5 | 7 | 204.66 | 218.66 | 225.25 |
| 6 | 8 | 204.64 | 220.64 | 229.64 |
AIC is smallest for the cubic, and AICc for the line. The cubic's AIC is only 1.3 below the line's, and at $n = 25$ its two extra parameters cost 2.0 more under AICc than under AIC, which reverses the order. On the diabetes data of Model Selection (Efron, Hastie, Johnstone and Tibshirani, 2004), with $n = 442$, the correction hardly matters: for the best six inputs AIC is 4,790.60 and AICc 4,790.94 with the constant kept as defined above (3,536.26 and 3,536.59 in the constant-free form of Model Selection), a correction of 0.33 that grows only to 0.73 for all ten inputs, and both criteria choose six inputs.
What is proved, what is cited, what the experiment shows
Proved on this page: the target as an expected divergence up to a shared constant; the exact bias of the maximized log-likelihood in the normal linear model whose columns contain the truth, and the unbiasedness of AICc (Theorem 1); for one parameter, the convergence in distribution of the bias variable to $2(\mathcal{K}/\mathcal{J})Z^2$, under the conditions of the asymptotics article with C4 replaced by C4′ (Theorem 2); the exact probability that AIC adds $m$ useless columns to a normal linear model, and its positive limit (Theorem 3).
Cited or stated: the asymptotic theorems for a consistent root and its normality, taken from Asymptotics of the Maximum Likelihood Estimator and extended to the pseudo-true value by reading its proofs; the convergence of the means, which needs uniform integrability; the $k$-parameter form $2\operatorname{tr}(\mathcal{J}^{-1}\mathcal{K})$ and Takeuchi's criterion (through Ding, Diao, Zhou and Tarokh, 2021, and Ding et al., 2018); Wilks's theorem for several restrictions (through the tests article); the random-walk limit of AIC's choice over a nested sequence (proved as Theorem 4 of the BIC article); the history of AIC, AICc and the asymptotic distribution of the selected order (Akaike, 1974; Sugiura, 1978; Hurvich and Tsai, 1989; Shibata, 1976), each from its abstract.
What the experiments show: with normal responses the measured bias matched Theorem 1 at every degree that contains the truth; with the real, heteroscedastic and skewed NHANES responses the bias exceeded it by 1.82 to 2.29 for degrees 1 to 6, close to a kurtosis heuristic. AIC chose too large a polynomial in 26.5% to 28.4% of data sets at every size from 100 to 1,600 adults, close to the limit of 0.265 for five extra terms, and AICc reduced overfitting at 25 adults only. The chosen model's AIC understated its own target by a median of 8.27 at $n = 25$. These are observations of these experiments.
Worked problems
The problems were written for this article; each numerical answer is checked by the article's script, and Problem 4 is an algebraic derivation.
Problem 1 (the AICc penalty). For a straight line with normal errors fitted to $n = 25$ observations, find the exact bias of $-2\hat\ell$ and the AICc correction.
Solution. The parameters are the intercept, the slope and the variance, so $k = 3$. By Theorem 1 the bias is $2kn/(n - k - 1) = 2 \cdot 3 \cdot 25/21 = 50/7 = 7.14$. AIC charges $2k = 6$, so AICc adds $50/7 - 6 = 8/7 = 1.14$, which is $2k(k+1)/(n - k - 1) = 24/21$.
Problem 2 (the chance of a useless quadratic term). With $n = 25$ and the line true, find the exact probability that AIC prefers the quadratic to the line, and compare it with the limit.
Solution. Here $m = 1$ and $p_1 = 3$. By Theorem 3, AIC prefers the quadratic when $F > t_{25} = 22\,(e^{2/25} - 1) = 1.832$, where $F$ has the $F$ distribution with 1 and 22 degrees of freedom. Its upper tail beyond 1.832 is 0.190, above the limit $P(\chi^2_1 > 2) = 0.157$. In small samples AIC is even more willing to add a term, which is what AICc corrects.
Problem 3 (Takeuchi's penalty for a Poisson model). Iowa's weekly deaths from all causes over the 313 weeks of 2014 to 2019 (National Center for Health Statistics, 2021) have mean 570.52 and variance 2,189.64 (divisor $n$; the asymptotics article reports 2,196.66 with divisor $n - 1$). A Poisson model is fitted. Find $\mathcal{J}$, $\mathcal{K}$ and the penalty $2\mathcal{K}/\mathcal{J}$, with the expectations estimated by sample averages, and compare it with AIC's 2.
Solution. For the Poisson, $\ln f(x; \lambda) = -\lambda + x\ln\lambda - \ln x!$, so $u = x/\lambda - 1$ and $\partial u/\partial\lambda = -x/\lambda^2$. The pseudo-true value solves $E_g[u] = E_g[X]/\lambda - 1 = 0$, so $\lambda^* = E_g[X]$, the true mean, whatever $g$ is. Then $\mathcal{J} = E_g[X]/\lambda^{*2} = 1/\lambda^*$ and $\mathcal{K} = E_g[(X/\lambda^* - 1)^2] = \operatorname{Var}_g(X)/\lambda^{*2}$, so $\mathcal{K}/\mathcal{J} = \operatorname{Var}_g(X)/\lambda^*$, the ratio of the variance to the mean. With the sample values it is $2{,}189.64/570.52 = 3.84$, and the penalty is $2 \times 3.84 = 7.68$ instead of 2. Overdispersed counts make the Poisson model's maximized log-likelihood far more optimistic than AIC assumes. (The weeks are also correlated, which the theorem's independence condition rules out; the penalty computed here accounts for the overdispersion only, not for the correlation.)
Problem 4 (AIC and $C_p$ with a known variance). Show that for a normal linear model with $\sigma^2$ known, AIC ranks models as $C_p$ does.
Solution. With $\sigma^2$ known only the $p$ coefficients are estimated, so $k = p$ and $-2\hat\ell = n\ln(2\pi\sigma^2) + \mathrm{RSS}/\sigma^2$. Then $\mathrm{AIC} = n\ln(2\pi\sigma^2) + \mathrm{RSS}/\sigma^2 + 2p$. Multiply by $\sigma^2/n$ and drop the constant: $\mathrm{RSS}/n + 2p\sigma^2/n$, which is $C_p$ of the optimism article with the true variance. A positive multiple and a constant do not change the ranking (Hastie et al., 2009, §7.5, state the equivalence).
Summary
| Quantity | Formula | Where |
|---|---|---|
| Target | $T = E_{\mathbf{y}}E_{\mathbf{y}^0}[-2\ln f(\mathbf{y}^0; \hat{\boldsymbol\theta})] = 2E[D(g\,\Vert\,f_{\hat{\boldsymbol\theta}})] + \text{const}$ | definition |
| AIC | $-2\hat\ell + 2k$ | Akaike (1974) |
| Exact bias, normal linear model containing the truth | $2kn/(n - k - 1)$; AICc $= \mathrm{AIC} + 2k(k+1)/(n - k - 1)$ | Theorem 1 |
| Large-sample bias, one parameter | limit $2(\mathcal{K}/\mathcal{J})Z^2$, mean $2\mathcal{K}/\mathcal{J}$; 2 if the model is correct | Theorem 2 |
| Takeuchi's penalty | $2\operatorname{tr}(\mathcal{J}^{-1}\mathcal{K})$ | stated |
| Adding $m$ useless parameters | $P(F_{m,\,n-p_1} > t_n) \to P(\chi^2_m > 2m)$; 0.157 for $m = 1$ | Theorem 3 |
What we have learned
As an estimate of the expected log-likelihood of new data, and so of the expected Kullback–Leibler divergence of the fitted model from the truth, the maximized log-likelihood is biased upward. The penalty of two per parameter is the mean of the large-sample limit of that bias, and equals the limit of the bias itself under a uniform-integrability condition assumed here. In the normal linear model the bias is exactly $2kn/(n - k - 1)$, which gives AICc, and in general it is twice a ratio of informations that equals the number of parameters when the model is right and need not otherwise. Because a useless parameter raises twice the log-likelihood by a chi-squared amount that exceeds 2 with probability 0.157, AIC keeps choosing too large a model however much data there is, and the model it chooses is more optimistic than its own criterion says.
A heavier penalty that grows with $n$ removes the inconsistency, at a cost in prediction. Its derivation is Bayesian and starts from the marginal likelihood: The Bayesian Information Criterion and the Laplace Approximation takes it up and compares the two criteria.
References
- Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6), 716–723. https://doi.org/10.1109/TAC.1974.1100705 record · 2 pages
- Ding, J., Diao, E., Zhou, J., and Tarokh, V. (2021). On statistical efficiency in learning. IEEE Transactions on Information Theory, 67(4), 2488–2506. https://doi.org/10.1109/TIT.2020.3047620 (arXiv:2012.13307) record
- Ding, J., Tarokh, V., and Yang, Y. (2018). Model selection techniques: an overview. IEEE Signal Processing Magazine, 35(6), 16–34. https://doi.org/10.1109/MSP.2018.2867638 (arXiv:1810.09583) record · 2 pages
- Efron, B., Hastie, T., Johnstone, I., and Tibshirani, R. (2004). Least angle regression. The Annals of Statistics, 32(2), 407–499. https://doi.org/10.1214/009053604000000067 record · 18 pages
- Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2nd ed.). Springer. Sections 7.5 and 7.11. record · 31 pages
- Hurvich, C. M., and Tsai, C.-L. (1989). Regression and time series model selection in small samples. Biometrika, 76(2), 297–307. https://doi.org/10.1093/biomet/76.2.297 record
- National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018: Demographic Variables and Sample Weights (DEMO_J), Blood Pressure (BPX_J) and Body Measures (BMX_J) data files. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 95 pages
- National Center for Health Statistics (2021). Weekly Counts of Deaths by State and Select Causes, 2014–2019. Centers for Disease Control and Prevention, data.cdc.gov dataset 3yf8-kanr. https://data.cdc.gov/d/3yf8-kanr (accessed November 2026). record · 13 pages
- Shibata, R. (1976). Selection of the order of an autoregressive model by Akaike's information criterion. Biometrika, 63(1), 117–126. https://doi.org/10.1093/biomet/63.1.117 record
- Sugiura, N. (1978). Further analysis of the data by Akaike's information criterion and the finite corrections. Communications in Statistics: Theory and Methods, 7(1), 13–26. https://doi.org/10.1080/03610927808827599 record
- Takeuchi, K. (1976). Distribution of informational statistics and a criterion of model fitting. Suri-Kagaku (Mathematical Sciences), 153, 12–18 (in Japanese). Not seen; cited as Ding, Diao, Zhou and Tarokh (2021, reference 3) give it, who also state that it was not written in English. record