Deviance and Inference for Generalized Linear Models

How a fitted generalized linear model is tested and compared. The deviance defined from the saturated model, with its unit deviances for the normal, Poisson and binomial families derived; deviance differences proved to be likelihood ratio statistics, and the same for grouped and individual records; the deviance of 0/1 data proved to depend on the fitted values alone, so that it measures no lack of fit; Pearson's statistic as its quadratic approximation; Pearson and deviance residuals; Wald, score and likelihood ratio tests in the model, with the score test for an exposure across strata proved to be the Mantel–Haenszel statistic with n_k in place of n_k - 1; and separation, with a proof that the logistic maximum likelihood estimate exists exactly when no direction separates the outcomes. Worked on diabetes, age and obesity among 5,107 NHANES adults, with experiments on how often small samples are separated and on the sizes of the three tests. Worked problems on two Poisson counts, a four-point logistic regression and the score test by hand.

Among 5,107 adults examined in the National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018 (National Center for Health Statistics, 2020), 777 report a diagnosis of diabetes. Grouped by age (18 to 39, 40 to 59, 60 and over) and by obesity (a body mass index of 30 or more), they form six binomial rows: 12 of 1,040, 22 of 640, 69 of 860, 140 of 697, 264 of 1,108 and 270 of 762 (each age group, not obese and then obese). A logistic regression of diabetes on age group and obesity, fitted by iteratively reweighted least squares, has four coefficients for six rows. Does it fit? Is the obesity coefficient different from zero, and how sure can we be of its size? What should be done when the fitting algorithm never stops, because a coefficient runs off to infinity? In short: the model fits only roughly, because the effect of obesity changes with age; the obesity coefficient is far from zero by every test; and when a coefficient runs off, the estimate does not exist, the Wald test fails, and the likelihood ratio and a penalized likelihood still give usable answers.

The deviance answers the first two questions. It measures how far a fitted model falls short of the saturated model, which gives every observation its own mean, on the scale of twice a log-likelihood. Differences of deviances between nested models are likelihood ratio statistics, so the deviance does for generalized linear models what the residual sum of squares does for linear ones.

First we define the deviance and derive it for the three families of the unit. Then we prove that deviance differences are likelihood ratio statistics, and show where the deviance of a single model is and is not a test of fit. Next come residuals, and the Wald, score and likelihood ratio tests with the score test identified as the Mantel–Haenszel test. Finally we prove when the maximum likelihood estimate of a logistic regression exists, and see what fitting software does, and what to do, when it does not. The article assumes the notation and results of Exponential Families, Link Functions and the Structure of a Generalized Linear Model and Iteratively Reweighted Least Squares, and the three tests of Likelihood Ratio, Wald and Score Tests.

The deviance

Observations $y_1, \ldots, y_n$ are independent, each with a density in the exponential dispersion form of the unit,

$$f(y_i; \theta_i, \phi) = \exp\Big\{\frac{w_i[y_i\theta_i - b(\theta_i)]}{\phi} + c(y_i, \phi/w_i)\Big\},$$

with mean $\mu_i = b'(\theta_i)$, variance $\phi V(\mu_i)/w_i$ and known prior weights $w_i$. For a binomial row, $y_i$ is the proportion of successes among $m_i$ trials and $w_i = m_i$. Write $\ell(\boldsymbol\mu; \mathbf{y})$ for the log-likelihood as a function of the means, since $\theta_i$ is determined by $\mu_i$.

A generalized linear model with $p$ coefficients restricts the means through $g(\mu_i) = \mathbf{x}_i^{\mathsf T}\boldsymbol\beta$, and its fitted means are $\hat{\boldsymbol\mu}$. The largest possible model puts no restriction at all: each $\mu_i$ is free.

Lemma 1 (the saturated model). The log-likelihood $\ell(\boldsymbol\mu; \mathbf{y})$ is maximized over all means by $\boldsymbol\mu = \mathbf{y}$, whenever each $y_i$ is a possible mean.

Proof. The terms are separate, so maximize each. As a function of $\theta$, $y\theta - b(\theta)$ has derivative $y - b'(\theta)$ and second derivative $-b''(\theta) < 0$ ($b''$ is the variance function up to $\phi$, positive; Exponential Families). It is strictly concave, with its maximum where $b'(\theta) = y$, that is where the mean equals $y$. $\square$

For a proportion of 0 or 1 or a count of 0, $y$ is a limit of possible means rather than a mean, and the same conclusion holds in the limit, with $0\ln 0 = 0$. The deviance of the fitted model is

$$D(\mathbf{y}; \hat{\boldsymbol\mu}) = 2\phi\,\big[\ell(\mathbf{y}; \mathbf{y}) - \ell(\hat{\boldsymbol\mu}; \mathbf{y})\big],$$

and $D/\phi$ is the scaled deviance. The factor $\phi$ cancels the $\phi$ in the denominator of the log-likelihood, and the terms $c(y_i, \phi/w_i)$ cancel in the difference, so

$$D = \sum_i w_i\,d(y_i; \hat\mu_i), \qquad d(y; \mu) = 2\big[y(\tilde\theta - \theta) - b(\tilde\theta) + b(\theta)\big],$$

where $\tilde\theta$ is the natural parameter with mean $y$ and $\theta$ the one with mean $\mu$. The unit deviance $d(y; \mu)$ does not involve $\phi$. By Lemma 1 it is nonnegative, and it is zero only when $\mu = y$.

The three families of the unit. Each unit deviance follows by substituting $\theta$ and $b$:

  • Normal: $\theta = \mu$, $b(\theta) = \theta^2/2$, so $d = 2[y(y - \mu) - y^2/2 + \mu^2/2] = (y - \mu)^2$. The deviance is the residual sum of squares, and the scaled deviance is that sum over $\sigma^2$.
  • Poisson: $\theta = \ln\mu$, $b(\theta) = e^\theta$, so $d = 2[y\ln(y/\mu) - (y - \mu)]$.
  • Binomial (proportion $y$, weight $m$): $\theta = \ln[\mu/(1 - \mu)]$, $b(\theta) = \ln(1 + e^\theta)$, and $b(\theta) = -\ln(1 - \mu)$. Then $y(\tilde\theta - \theta) - b(\tilde\theta) + b(\theta) = y\ln\frac{y(1 - \mu)}{(1 - y)\mu} + \ln\frac{1 - y}{1 - \mu}$, so

$$d = 2\Big[y\ln\frac{y}{\mu} + (1 - y)\ln\frac{1 - y}{1 - \mu}\Big],$$

and $m\,d$ is twice the familiar $\sum O\ln(O/E)$ over the two cells (successes and failures) of the row.

For the Poisson family with an intercept the term $-(y - \mu)$ sums to zero at the fitted values, because the fitted total equals the observed total (Link Functions proves this for any canonical link with an intercept). Then $D = 2\sum y_i\ln(y_i/\hat\mu_i)$, the likelihood ratio statistic $G^2$ of a table of counts.

Differences of deviances are likelihood ratio statistics

Here is the idea in one sentence: the saturated log-likelihood is the same for every model fitted to the same data, so it cancels from the difference of two deviances, leaving twice the difference of their maximized log-likelihoods.

Theorem 1 (analysis of deviance). Let model $M_0$, with $p_0$ coefficients, be nested in model $M_1$, with $p_1 > p_0$ (every mean vector allowed by $M_0$ is allowed by $M_1$), both fitted to the same data with the same known $\phi$. Then

$$\frac{D_0 - D_1}{\phi} = 2\big[\ell(\hat{\boldsymbol\mu}_1; \mathbf{y}) - \ell(\hat{\boldsymbol\mu}_0; \mathbf{y})\big] = W_{\mathrm{LR}},$$

the likelihood ratio statistic for the restrictions that turn $M_1$ into $M_0$. It is nonnegative, and under $M_0$ and the regularity conditions of Wilks's theorem it converges in distribution to the chi-squared distribution with $p_1 - p_0$ degrees of freedom.

Proof. $D_0 - D_1 = 2\phi[\ell(\mathbf{y}; \mathbf{y}) - \ell(\hat{\boldsymbol\mu}_0)] - 2\phi[\ell(\mathbf{y}; \mathbf{y}) - \ell(\hat{\boldsymbol\mu}_1)] = 2\phi[\ell(\hat{\boldsymbol\mu}_1) - \ell(\hat{\boldsymbol\mu}_0)]$. Since $\hat{\boldsymbol\mu}_0$ is allowed by $M_1$, the maximum over $M_1$ is at least $\ell(\hat{\boldsymbol\mu}_0)$. The limit is Wilks's theorem for several parameters, which Likelihood Ratio, Wald and Score Tests states and cites (Wilks, 1938); it is not proved in the library. $\square$

This decomposition goes back to the paper that introduced generalized linear models: Nelder and Wedderburn (1972) state that "a generalization of the analysis of variance is given for these models using log-likelihoods" (their abstract). The case $\phi$ unknown is taken up below. For the binomial and Poisson families $\phi = 1$ and the deviance difference is the likelihood ratio statistic itself.

The analysis of deviance for diabetes. Four nested logistic models for the six rows:

Model Coefficients Deviance df Pearson $X^2$
Intercept only 1 645.71 5 580.69
Age group 3 87.90 3 88.44
Age group + obesity 4 8.26 2 8.16
Age group × obesity 6 0 0 0

Adding age group lowers the deviance by 557.81 on 2 degrees of freedom; adding obesity, given age group, lowers it by 79.64 on 1. Both are far beyond the chi-squared quantiles. The last model has a separate proportion for each row and is saturated, so its deviance is zero, and the drop from the third model to it, 8.26 on 2 degrees of freedom, tests whether the effect of obesity on the log odds is the same in the three age groups. The $p$-value is 0.016 (0.017 from Pearson's statistic). That is the question of homogeneity of the odds ratio across strata, and The Mantel–Haenszel Test reaches the same conclusion with the Breslow–Day statistic, 8.16 on 2 degrees of freedom ($p = 0.017$) on these tables: the odds ratio for obesity falls with age.

Grouped and individual records. The same models can be fitted to the 5,107 individual 0/1 responses. Their deviances are larger, 4,355.35, 3,797.54, 3,717.90 and 3,709.65 for the four models, but every difference is the same as in the grouped fit: $4{,}355.35 - 3{,}797.54 = 557.81$, and $3{,}797.54 - 3{,}717.90 = 79.64$. The reason is Theorem 1. A binomial row and its $m_i$ individual Bernoulli responses have log-likelihoods that differ by $\ln\binom{m_i}{m_iy_i}$, a constant free of the parameters. That constant cancels in a difference of maximized log-likelihoods. (Summed over the six rows it is 1,836.86, the gap between the null log-likelihoods, $-340.81$ grouped and $-2{,}177.68$ individual.) What changes is the saturated model: for the individual records it fits each person's 0 or 1 exactly, a model with 5,107 parameters, and so the deviances themselves differ.

Is the deviance of one model a test of fit?

The deviance of a model is its likelihood ratio statistic against the saturated model, $p_1 = n$. Theorem 1 would refer it to $\chi^2_{n - p}$. Whether that is justified depends on the limit taken.

Grouped data with large rows. For the six diabetes rows the number of rows stays fixed while each $m_i$ is large. The saturated model has a fixed number of parameters, the regularity conditions concern a fixed-dimensional model, and the reference $\chi^2_{2}$ for the deviance 8.26 is the large-sample one. The $p$-value 0.016 above is of this kind. The same holds for the Poisson deviance of a table of counts whose expected counts all grow. Breslow and Day (1987, §4.3(c), pp. 137–138) refer the deviance and Pearson's statistic of a fitted table of rates to the chi-squared distribution with degrees of freedom equal to the number of cells minus the number of parameters, and they caution that both may overstate the departure from the model when many cells contain small counts.

Individual binary data. For 0/1 responses the saturated model gains one parameter with each new observation, and Wilks's theorem says nothing. The next theorem shows that the deviance then cannot measure lack of fit at all.

Theorem 2 (the deviance of 0/1 data). For a logistic regression on 0/1 responses fitted by maximum likelihood, the deviance depends on the data only through the fitted probabilities:

$$D = -2\sum_{i=1}^n\big[\hat\mu_i\ln\hat\mu_i + (1 - \hat\mu_i)\ln(1 - \hat\mu_i)\big].$$

Proof. For a 0/1 response the saturated log-likelihood is 0 (with $0\ln 0 = 0$), so $D = -2\sum_i[y_i\ln\hat\mu_i + (1 - y_i)\ln(1 - \hat\mu_i)]$. Write $y_i\ln\hat\mu_i + (1 - y_i)\ln(1 - \hat\mu_i) = y_i\hat\eta_i + \ln(1 - \hat\mu_i)$, where $\hat\eta_i = \ln[\hat\mu_i/(1 - \hat\mu_i)] = \mathbf{x}_i^{\mathsf T}\hat{\boldsymbol\beta}$. The likelihood equations of the canonical link are $X^{\mathsf T}(\mathbf{y} - \hat{\boldsymbol\mu}) = \mathbf{0}$ (Link Functions), so $\sum_i y_i\hat\eta_i = \hat{\boldsymbol\beta}^{\mathsf T}X^{\mathsf T}\mathbf{y} = \hat{\boldsymbol\beta}^{\mathsf T}X^{\mathsf T}\hat{\boldsymbol\mu} = \sum_i\hat\mu_i\hat\eta_i$. Substituting, $D = -2\sum_i[\hat\mu_i\hat\eta_i + \ln(1 - \hat\mu_i)] = -2\sum_i[\hat\mu_i\ln\hat\mu_i + (1 - \hat\mu_i)\ln(1 - \hat\mu_i)]$. $\square$

Any two data sets that lead to the same fitted probabilities have the same deviance, however differently the outcomes are arranged among the people. The deviance only measures how close to 0 or 1 the fitted probabilities are, a property of the fit, not of its agreement with the data.

Example (a deviance smaller than its degrees of freedom is not a good fit). A logistic regression of diabetes on age and BMI, both in their measured units, fitted to the 5,107 adults, has coefficients 0.0593 per year and 0.0653 per kg/m² and a deviance of 3,700.55 on 5,104 degrees of freedom. Read against $\chi^2_{5104}$, whose mean is 5,104, it looks like an excellent fit. It is not evidence of one. The script simulated 1,000 sets of responses from the fitted model itself and refitted it each time; the simulated deviances had median 3,700.8, interquartile range (3,642.9, 3,754.2) and standard deviation 79.7, nowhere near the mean 5,104 and standard deviation $\sqrt{2 \times 5{,}104} = 101.0$ of the chi-squared reference. Data generated by the model do not give chi-squared deviances, so the chi-squared reference cannot judge the model. The script also confirms Theorem 2 on two fits: the deviance computed from the data equals the formula in the fitted values.

For individual binary data, fit is examined by grouping (as in the six rows, or by bands of fitted probability) or by adding terms and testing them with deviance differences, which Theorem 1 does cover.

Pearson's statistic. The other summary of fit is

$$X^2 = \sum_i \frac{w_i(y_i - \hat\mu_i)^2}{V(\hat\mu_i)},$$

and the two usually agree, as in the table above (8.26 and 8.16). The reason is that each unit deviance is approximately the corresponding Pearson term. Here is the derivation. Fix $\mu$ and expand $d(y; \mu)$ in $y$ around $\mu$. At $y = \mu$, $d = 0$. Its derivative in $y$ is $2(\tilde\theta - \theta)$, since the terms from differentiating $\tilde\theta$ cancel ($\partial\tilde\theta/\partial y$ multiplies $y - b'(\tilde\theta) = 0$); this is 0 at $y = \mu$. The second derivative is $2\,\partial\tilde\theta/\partial y = 2/b''(\tilde\theta) = 2/V(y)$, because $y = b'(\tilde\theta)$ gives $dy = b''(\tilde\theta)\,d\tilde\theta$. So

$$d(y; \mu) = \frac{(y - \mu)^2}{V(\mu)} + O(|y - \mu|^3),$$

and $D$ and $X^2$ differ by terms of third order in the residuals, which are small when every row is large. The Chi-Square Goodness-of-Fit Test proves the corresponding statement for multinomial counts: the likelihood ratio statistic and Pearson's statistic differ by an amount that tends to zero in probability. Breslow and Day (1987, §4.3(c)) note that the two "tend to give similar values in most applications."

Residuals

Each row contributes a term to $D$ and to $X^2$, and the signed square roots of those terms are the two common residuals:

$$r_{P,i} = \frac{\sqrt{w_i}\,(y_i - \hat\mu_i)}{\sqrt{V(\hat\mu_i)}}, \qquad r_{D,i} = \operatorname{sign}(y_i - \hat\mu_i)\sqrt{w_i\,d(y_i; \hat\mu_i)},$$

the Pearson and deviance residuals, with $\sum r_{P,i}^2 = X^2$ and $\sum r_{D,i}^2 = D$. A fitted value is pulled toward its own observation, more so for rows with more influence on the fit. The leverage $h_{ii}$, the $i$th diagonal entry of the hat matrix $H = W^{1/2}X(X^{\mathsf T}WX)^{-1}X^{\mathsf T}W^{1/2}$ of the final weighted fit, measures that pull. The standardized Pearson residual $r_{P,i}/\sqrt{1 - h_{ii}}$ divides by an estimate of its standard deviation; the rationale is the one Leverage, Influence and Residual Diagnostics proves for least squares, applied to the weighted least squares problem of the last IRLS iteration (Pregibon, 1981, develops these diagnostics for logistic regression).

For the age + obesity model:

Age, obesity Observed Fitted $r_P$ $r_D$ $h_{ii}$
18–39, not obese 12 15.0 −0.78 −0.81 0.48
18–39, obese 22 19.0 0.70 0.68 0.58
40–59, not obese 69 82.4 −1.56 −1.60 0.59
40–59, obese 140 126.6 1.32 1.30 0.71
60+, not obese 264 247.6 1.18 1.18 0.83
60+, obese 270 286.4 −1.23 −1.23 0.81

The leverages sum to 4, the number of coefficients. The pattern of signs is the interaction: in the two younger groups the obese have more diabetes than the common odds ratio predicts and the non-obese less, and in the oldest group the reverse. With six rows and four coefficients the leverages are large, and the standardized residuals are much larger than the raw ones: ±1.08 in the youngest group, ±2.44 in the middle one and ±2.85 in the oldest. In this fit each pair has equal size and opposite sign.

Wald, score and likelihood ratio tests in a generalized linear model

Likelihood Ratio, Wald and Score Tests defines the three statistics for a parameter vector and states their common $\chi^2_r$ limit for $r$ restrictions. In a generalized linear model they take concrete forms, with the score and information of Iteratively Reweighted Least Squares: $\mathbf{U}(\boldsymbol\beta) = X^{\mathsf T}WG(\mathbf{y} - \boldsymbol\mu)/\phi$ and $\mathcal{I}(\boldsymbol\beta) = X^{\mathsf T}WX/\phi$.

  • Likelihood ratio: the deviance difference of Theorem 1.
  • Wald, for one coefficient: $z = \hat\beta_j/\operatorname{se}(\hat\beta_j)$, with $\operatorname{se}^2$ the $j$th diagonal entry of $\hat\phi(X^{\mathsf T}\hat WX)^{-1}$, and $W_{\mathrm{W}} = z^2$. For several coefficients, the quadratic form in their estimates with the inverse of their block of the covariance matrix.
  • Score: fit only the smaller model, and evaluate $W_{\mathrm{S}} = \mathbf{U}(\tilde{\boldsymbol\beta})^{\mathsf T}\mathcal{I}(\tilde{\boldsymbol\beta})^{-1}\mathbf{U}(\tilde{\boldsymbol\beta})$ at its estimate $\tilde{\boldsymbol\beta}$ (padded with zeros for the coefficients it omits), using the score and information of the larger model.

For obesity given age group:

Statistic Value
$\hat\beta$ for obesity, standard error 0.7386, 0.0833
Odds ratio $e^{\hat\beta}$ 2.093
Wald $z^2$ 78.66
Likelihood ratio (deviance difference) 79.64
Score 80.44
95% Wald interval for the odds ratio (1.778, 2.464)
95% profile likelihood interval (1.779, 2.465)

The profile likelihood interval is the set of values $\beta_0$ of the obesity coefficient that the likelihood ratio test does not reject: fix the coefficient at $\beta_0$ (as an offset $\beta_0 x_i$), refit the rest, and keep $\beta_0$ while the deviance exceeds the minimum by less than 3.8415, the 0.95 quantile of $\chi^2_1$. With 777 cases the log-likelihood is close to a parabola and the three tests and the two intervals nearly agree. They part company in small samples and near a boundary, as the next sections show.

The score test is the Mantel–Haenszel test. The logistic model with age group as a factor and obesity as an indicator is a model for a series of $2 \times 2$ tables with a common odds ratio $e^{\beta}$. Day and Byar (1979) showed, in their words, that "score statistics derived from the likelihood function in the latter approach are identical to the Mantel-Haenszel test statistics" (their abstract). Their identity is demonstrated "by a permutational argument" (their abstract); we read this, though the abstract does not say it in these words, as concerning the likelihood conditional on the margins of each table, whose score test has the exact hypergeometric variance that The Mantel–Haenszel Test derives. The unconditional score test of the theorem below differs from it only in that variance. In the notation of The Mantel–Haenszel Test (stratum $k$ has $n_{1k}$ exposed and $n_{0k}$ unexposed, $m_{1k}$ with the outcome and $m_{0k}$ without, $a_k$ exposed with the outcome, $n_k$ in all), the precise statement for this model is as follows.

Theorem 3 (the score test for an exposure across strata). In the logistic model $\operatorname{logit}\mu_i = \alpha_{k(i)} + \beta x_i$, with a separate intercept for each stratum and a 0/1 exposure $x_i$, and with both outcomes present in every stratum ($0 < m_{1k} < n_k$), the score statistic for $\beta = 0$ is

$$W_{\mathrm{S}} = \frac{\big[\sum_k(a_k - E_k)\big]^2}{\sum_k n_{1k}n_{0k}m_{1k}m_{0k}/n_k^3}, \qquad E_k = \frac{n_{1k}m_{1k}}{n_k}.$$

It equals the square of the Mantel–Haenszel statistic computed with $n_k$ in place of $n_k - 1$ in the variance.

Here is the plan: compute the score at the fit without exposure, where every fitted probability in a stratum is that stratum's proportion; then remove from the information about $\beta$ the part shared with the intercepts.

Proof. Under $\beta = 0$ the model has one parameter per stratum, and by the fitted-totals property every fitted probability in stratum $k$ is $\tilde p_k = m_{1k}/n_k$. With the canonical link and $\phi = 1$ the score is $X^{\mathsf T}(\mathbf{y} - \tilde{\boldsymbol\mu})$. Its components for the intercepts vanish at the fit. Its component for $\beta$ is $\sum_i x_i(y_i - \tilde\mu_i) = \sum_k(a_k - n_{1k}\tilde p_k) = \sum_k(a_k - E_k)$. The information is $X^{\mathsf T}WX$ with $W_i = \tilde p_k(1 - \tilde p_k) = v_k$ in stratum $k$. Its entries are $\mathcal{I}_{\beta\beta} = \sum_k v_kn_{1k}$, $\mathcal{I}_{\beta\alpha_k} = v_kn_{1k}$, and a diagonal block $\mathcal{I}_{\alpha\alpha} = \operatorname{diag}(v_kn_k)$. Since only the $\beta$ component of the score is nonzero, $W_{\mathrm{S}} = U_\beta^2\,[\mathcal{I}^{-1}]_{\beta\beta}$, and by the formula for the inverse of a partitioned matrix $[\mathcal{I}^{-1}]_{\beta\beta} = (\mathcal{I}_{\beta\beta} - \mathcal{I}_{\beta\alpha}\mathcal{I}_{\alpha\alpha}^{-1}\mathcal{I}_{\alpha\beta})^{-1}$. The quantity in parentheses is

$$\sum_k v_kn_{1k} - \sum_k\frac{(v_kn_{1k})^2}{v_kn_k} = \sum_k v_k\frac{n_{1k}(n_k - n_{1k})}{n_k} = \sum_k\frac{m_{1k}m_{0k}}{n_k^2}\cdot\frac{n_{1k}n_{0k}}{n_k},$$

which is the denominator stated. The Mantel–Haenszel variance of a table is $n_{1k}n_{0k}m_{1k}m_{0k}/[n_k^2(n_k - 1)]$, the exact hypergeometric variance; the score test's is the same with $n_k$ for $n_k - 1$. $\square$

The condition $0 < m_{1k} < n_k$ makes every $v_k$ positive and every intercept finite. A stratum with no cases or only cases has, under $\beta = 0$, an infinite intercept (the separation of the section below) and contributes $a_k - E_k = 0$ to the numerator and 0 to the denominator, so it is dropped, as it is from the Mantel–Haenszel statistic.

For the three age groups the numerator is $\sum_k a_k - \sum_k E_k = 432 - 324.11 = 107.89$, and $W_{\mathrm{S}} = 80.44$. The Mantel–Haenszel statistic of the earlier article is $z_{\mathrm{MH}} = 8.97$, whose square is 80.39. The ratio of the two variances is 1.00057, a difference of $1/(n_k - 1)$ in each stratum of a thousand or more adults. With strata of two (matched pairs) the factor is 2, and the exact conditional variance of the Mantel–Haenszel test is the one to use. The third worked problem carries out the computation by hand.

Several restrictions. The test of the interaction above, 2 degrees of freedom, has the same three forms. The likelihood ratio form is the deviance 8.26 of the age + obesity model, and Pearson's statistic for that model, 8.16, is its score form. (The Breslow–Day statistic of The Mantel–Haenszel Test is also 8.16 on these tables; the two are different statistics that agree here to two decimals.) The reason is short. The saturated model of $n$ rows can be written with an invertible $n \times n$ design matrix $X_s$ whose columns span those of the smaller model. With the canonical link and $\phi = 1$, the score at the smaller fit is $X_s^{\mathsf T}\mathbf{r}$, where $\mathbf{r}$ has entries $w_i(y_i - \tilde\mu_i)$, and the information is $X_s^{\mathsf T}WX_s$ with $W = \operatorname{diag}(w_iV(\tilde\mu_i))$. Since $X_s$ is invertible, $(X_s^{\mathsf T}WX_s)^{-1} = X_s^{-1}W^{-1}(X_s^{\mathsf T})^{-1}$, and

$$W_{\mathrm{S}} = \mathbf{r}^{\mathsf T}X_s(X_s^{\mathsf T}WX_s)^{-1}X_s^{\mathsf T}\mathbf{r} = \mathbf{r}^{\mathsf T}W^{-1}\mathbf{r} = \sum_i\frac{w_i(y_i - \tilde\mu_i)^2}{V(\tilde\mu_i)},$$

Pearson's statistic. The tests article shows the same for two proportions and the goodness-of-fit article for any number of categories.

When $\phi$ is unknown. For the normal family, $\phi = \sigma^2$ must be estimated, and the deviance difference over $\hat\phi$ is no longer chi-squared. With $\hat\phi$ the residual mean square of the larger model, $(D_0 - D_1)/[(p_1 - p_0)\hat\phi]$ is exactly the $F$ statistic of The General Linear Hypothesis, which proves its $F$ distribution and that it is the likelihood ratio test. For other families with a dispersion to estimate, the same ratio is referred to the $F$ distribution as an approximation; Overdispersion and Quasi-Likelihood derives the estimate of $\phi$ and uses the test.

Which test? In the diabetes data it makes no difference. In general the likelihood ratio and score tests do not depend on how the coefficient is written, and the Wald test does, as Likelihood Ratio, Wald and Score Tests proves for one parameter (the score test with the expected information). That article also shows a Wald statistic that shrinks as the evidence against the null value grows, the behavior that Hauck and Donner (1977) studied in logit models, as that article cites them. The extreme case of that behavior is separation.

Separation: when the estimate does not exist

Draw 100 adults at random from the 5,107 (the fourth sample of the experiment below, seed 1). Their six rows are 0 of 19, 0 of 13, 3 of 20, 0 of 12, 3 of 14 and 8 of 22. Nobody in the youngest age group has diabetes. Fit the age + obesity model by IRLS, starting from the empirical logits $\ln[(s_i + \tfrac12)/(m_i - s_i + \tfrac12)]$ of the rows ($s_i$ cases of $m_i$), and watch the iterations:

Iterations Intercept 40–59 coefficient SE of intercept Deviance
5 −8.57 6.27 12.7 3.9333
10 −13.57 11.27 154.1 3.9209
25 −28.57 26.27 278,385.7 3.9208

The values at a given iteration depend on the starting point; the drift and the limit of the deviance do not. The deviance settles while the intercept, the log odds of the youngest group, falls by one at each iteration and the 40–59 coefficient rises to compensate. The standard errors explode. A program with a convergence test on the deviance stops and reports these numbers, sometimes with a warning. The Wald statistic for the 40–59 coefficient after 25 iterations is $26.27/278{,}385.7 = 0.0001$: no evidence of an age effect, in a sample where the youngest group has no cases and the older groups have 14.

The model's log-likelihood has no maximum. It approaches its supremum as the probability of diabetes in the youngest group goes to 0, a limit that no finite intercept reaches. What determines whether a maximum exists?

Theorem 4 (existence of the logistic estimate). Let $y_1, \ldots, y_n \in \{0, 1\}$, let $X$ ($n \times p$) have full column rank, and write $s_i = 2y_i - 1$ ($+1$ for a success, $-1$ for a failure). The maximum likelihood estimate of the logistic regression exists if and only if there is no nonzero $\mathbf{b}$ with

$$s_i\,\mathbf{x}_i^{\mathsf T}\mathbf{b} \ge 0 \quad \text{for every } i .$$

When it exists it is unique.

A vector $\mathbf{b}$ with this property separates the outcomes: the linear score $\mathbf{x}_i^{\mathsf T}\mathbf{b}$ is at least 0 for every success and at most 0 for every failure. Albert and Anderson (1984) classify data into three mutually exclusive patterns: complete separation (strict inequality for every $i$), quasi-complete separation (a separating $\mathbf{b}$ exists but only with equality for some $i$), and overlap (no separating $\mathbf{b}$). They prove existence theorems for multinomial logistic models by considering these patterns, and Santner and Duffy (1986) state that the estimate exists only under overlap, give modified statements and proofs for the two separated cases, and present a linear program that decides which pattern holds (both from their abstracts; Silvapulle, 1981, gives necessary and sufficient conditions for binomial response models in general). The proof below covers the binary case; Multinomial and Softmax Regression proves the version for $K$ classes.

Here is the plan. If a separating direction exists, moving along it never lowers the log-likelihood and strictly raises it, so no point is a maximum. If none exists, the log-likelihood goes to $-\infty$ along every ray, uniformly by compactness of the unit sphere, so a maximum is attained in a bounded set.

Proof. Write $\sigma(t) = 1/(1 + e^{-t})$. A success has probability $\sigma(\eta_i)$ and a failure $1 - \sigma(\eta_i) = \sigma(-\eta_i)$, so in both cases the probability is $\sigma(s_i\eta_i)$, and

$$\ell(\boldsymbol\beta) = \sum_{i=1}^n\ln\sigma(s_i\mathbf{x}_i^{\mathsf T}\boldsymbol\beta).$$

Every term is negative, and $\ln\sigma$ is strictly increasing.

If a separating $\mathbf{b} \ne \mathbf{0}$ exists, there is no maximum. Since $X$ has full column rank, $X\mathbf{b} \ne \mathbf{0}$, so some $\mathbf{x}_j^{\mathsf T}\mathbf{b} \ne 0$, and by the separating property $s_j\mathbf{x}_j^{\mathsf T}\mathbf{b} > 0$. Take any $\boldsymbol\beta$. In $\ell(\boldsymbol\beta + \mathbf{b})$ each term $\ln\sigma(s_i\mathbf{x}_i^{\mathsf T}\boldsymbol\beta + s_i\mathbf{x}_i^{\mathsf T}\mathbf{b})$ is at least the corresponding term of $\ell(\boldsymbol\beta)$, since $s_i\mathbf{x}_i^{\mathsf T}\mathbf{b} \ge 0$, and the $j$th is strictly larger. So $\ell(\boldsymbol\beta + \mathbf{b}) > \ell(\boldsymbol\beta)$, and $\boldsymbol\beta$ is not a maximum.

If none exists, a maximum exists. Then for every $\mathbf{u}$ on the unit sphere $S = \{\mathbf{u}: \|\mathbf{u}\| = 1\}$ some $i$ has $s_i\mathbf{x}_i^{\mathsf T}\mathbf{u} < 0$, so $g(\mathbf{u}) = \min_i s_i\mathbf{x}_i^{\mathsf T}\mathbf{u}$ is negative on $S$. As a minimum of finitely many continuous functions, $g$ is continuous, and $S$ is compact, so $g$ attains its largest value on $S$, some $-c < 0$. Every $\boldsymbol\beta \ne \mathbf{0}$ is $r\mathbf{u}$ with $r = \|\boldsymbol\beta\| > 0$ and $\mathbf{u} \in S$, and some $i$ has $s_i\mathbf{x}_i^{\mathsf T}\boldsymbol\beta = r\,s_i\mathbf{x}_i^{\mathsf T}\mathbf{u} \le -cr$. Dropping the other (negative) terms, $\ell(\boldsymbol\beta) \le \ln\sigma(-cr)$, which tends to $-\infty$ as $r \to \infty$. Choose $R$ with $\ln\sigma(-cR) < \ell(\mathbf{0}) = n\ln\tfrac12$. Then every $\boldsymbol\beta$ with $\|\boldsymbol\beta\| > R$ has $\ell(\boldsymbol\beta) < \ell(\mathbf{0})$. The continuous function $\ell$ attains a maximum on the compact ball $\|\boldsymbol\beta\| \le R$, at some $\hat{\boldsymbol\beta}$ with $\ell(\hat{\boldsymbol\beta}) \ge \ell(\mathbf{0})$, and that is a maximum over all $\boldsymbol\beta$.

Uniqueness. The Hessian of $\ell$ is $-X^{\mathsf T}WX$ with $W_i = \mu_i(1 - \mu_i) > 0$, negative definite when $X$ has full column rank, so $\ell$ is strictly concave and has at most one maximum (IRLS proves the uniqueness of a maximum under strict concavity). $\square$

In the sample of 100 the separating direction is easy to see. Take $\mathbf{b}$ that lowers the intercept by 1 and raises both older-age coefficients by 1: the linear score changes by $-1$ for the youngest group and by 0 for everyone else. Every person in the youngest group is a failure ($s_i = -1$), so $s_i\mathbf{x}_i^{\mathsf T}\mathbf{b} = 1 > 0$ there, and 0 elsewhere. That is quasi-complete separation, and it is exactly the direction in which the iterations drifted, one unit per step.

The same problem for counts. For Poisson regression the corresponding failure is a level of a factor with no events, which forces a fitted total of zero; Poisson Regression and Rate Models gives the argument.

What still works. The supremum of the log-likelihood is a limit that can be computed: fit the youngest rows exactly (probability 0) and fit the other four rows by the remaining coefficients. The deviance of that limit is 3.9208, the value the iterations approached. So the likelihood ratio statistic for age, comparing the obesity-only model with the limit, is computable: 16.10 on 2 degrees of freedom. Its chi-squared calibration is not covered by Wilks's theorem, whose conditions assume a maximum inside the parameter space, but unlike the Wald statistic it reflects the evidence. The profile likelihood for the 40–59 coefficient is also computable. Its 95% limit on the low side is 0.16 (an odds ratio of 1.18 against the youngest group), and there is no limit on the high side: the profile deviance keeps falling toward its limit as the coefficient grows, so the interval is $(0.16, \infty)$. That is an honest summary of 100 adults with no young cases.

Penalized likelihood. Firth (1993) showed that a suitable modification of the score function removes the first-order term of the bias of maximum likelihood estimates, and that in exponential families with canonical parameter, logistic regression among them, the modification amounts to penalizing the likelihood by the Jeffreys invariant prior, that is, adding $\tfrac12\ln\det\mathcal{I}(\boldsymbol\beta)$ to the log-likelihood (his abstract). In the words of Heinze and Schemper (2002), Firth's procedure "is shown to provide an ideal solution to separation": it produces finite estimates, and they found penalized likelihood ratio tests and profile penalized likelihood intervals often preferable to the corresponding Wald forms (their abstract). For the sample of 100, Firth's estimates are $-4.16$ for the intercept, 2.03 and 3.34 for the two older age groups and 0.06 for obesity (an odds ratio of 1.06). The 40–59 coefficient of 2.03 is close to the value in the full population of 5,107, 1.98. That closeness is one sample, not a property of the method; the script checks only that the penalized score is zero at the reported estimate.

How often is a small sample separated?

The experiment. For each sample size from 50 to 800, the script drew 1,000 random samples of adults from the 5,107, for each of 30 seeds, and decided by a linear program whether the age + obesity model's estimate exists in each sample (Theorem 4). The share of separated samples, median and interquartile range over the 30 seeds:

Sample size Separated IQR No case aged 18–39 Exact probability of no young case
50 0.761 (0.748, 0.770) 0.716 0.715
100 0.522 (0.514, 0.529) 0.513 0.509
200 0.254 (0.249, 0.269) 0.254 0.256
400 0.058 (0.053, 0.066) 0.058 0.062
800 0.003 (0.002, 0.004) 0.003 0.003

Diabetes is rare among the young, 34 of 1,680, and almost every separated sample is one with no young case; the last column is the hypergeometric probability that a random sample contains none of the 34. At 50 adults a few more samples are separated in other ways, by other patterns of empty rows that the linear program detects. Separation is not an exotic event: in this population a sample of 200 adults is separated one time in four, and the estimate with three age groups does not exist.

Sizes of the three tests. The script then asked whether the three tests of obesity have the right size when obesity truly has no effect within age groups. It drew samples of 200 and 500 adults and generated each person's diabetes from the population proportion of their age group alone, so that the null hypothesis holds. Samples without an estimate, under either model, were set aside (a median share of 0.260 at 200 and 0.036 at 500), and 1,000 usable samples were collected for each of 30 seeds. Rejection rates at the 5% level, median and interquartile range over the seeds:

Test $n = 200$ $n = 500$
Likelihood ratio 0.055 (0.049, 0.060) 0.052 (0.045, 0.056)
Wald 0.049 (0.044, 0.054) 0.049 (0.042, 0.054)
Score 0.053 (0.047, 0.058) 0.051 (0.044, 0.055)

All three are close to 0.05 in these conditions, conditionally on the estimate existing. The experiment does not show that they agree in general: here the obesity indicator is common (41% of adults) and the null value is zero, where the log-likelihood is close to a parabola. The case they disagree about most, a coefficient running toward infinity, is exactly the case the experiment set aside.

What is proved here, what is cited, and what the experiments show

Proved on this page: the saturated model maximizes the likelihood (Lemma 1); the unit deviances of the normal, Poisson and binomial families; deviance differences are likelihood ratio statistics, and are the same for grouped and individual records (Theorem 1, apart from the limit distribution); the deviance of 0/1 data depends only on the fitted values (Theorem 2); the second-order agreement of unit deviance and Pearson term; the score test for an exposure across strata and its relation to the Mantel–Haenszel statistic (Theorem 3); the existence and uniqueness of the logistic estimate exactly under overlap (Theorem 4).

Taken from other pages: the score, information and fitted totals of canonical-link models and the IRLS algorithm (Link Functions, IRLS); the three tests for one parameter and their invariance properties (Likelihood Ratio, Wald and Score Tests); the multinomial likelihood ratio and Pearson statistics (The Chi-Square Goodness-of-Fit Test); the $F$ test of the normal linear model (The General Linear Hypothesis).

Cited, not proved: Wilks's theorem for several parameters (Wilks, 1938), on which every chi-squared $p$-value of a deviance difference rests; the classification of separation patterns and the linear program (Albert and Anderson, 1984; Santner and Duffy, 1986; Silvapulle, 1981); the identity of score and Mantel–Haenszel statistics in its general form (Day and Byar, 1979); Firth's penalty and its use under separation (Firth, 1993; Heinze and Schemper, 2002); the regression diagnostics (Pregibon, 1981).

Observed in the experiments: the shares of separated samples and the sizes of the three tests, for this population, these sample sizes and 30 seeds; the distribution of the deviance of data simulated from one fitted 0/1 model. Every number on the page is computed by tools/experiments/statistics/deviance-and-inference-for-glms.py.

Worked problems

These problems are posed for this article, on small exact cases and on the unit's NHANES tables; the unit has no course assignments. Every number is recomputed by the article's script.

Problem 1 (two Poisson counts). Two clinics record 30 and 50 events in equal periods. Fit the model with a common rate. Find its deviance and Pearson statistic, with $p$-values on the right degrees of freedom, and say why they differ slightly.

Solution. With equal exposure and one parameter, the fitted mean of each count is the average, $\hat\mu = 40$ (the fitted total equals the observed total). The saturated model fits each count exactly. The deviance is $2[30\ln(30/40) + 50\ln(50/40)] = 5.053$; the terms $-(y - \hat\mu)$ sum to zero. Pearson's statistic is $(30 - 40)^2/40 + (50 - 40)^2/40 = 5.000$. The saturated model has 2 parameters and the fitted one 1, so each is referred to $\chi^2_1$ (two counts that both grow, so Theorem 1's large-sample reference applies). The $p$-values are 0.0246 and 0.0253. They differ by the third-order terms of the expansion of the unit deviance. Each count contributes 2.5 to Pearson's statistic. To the deviance the count below the mean contributes $2[30\ln 0.75 + 10] = 2.739$ and the count above it $2[50\ln 1.25 - 10] = 2.314$: the unit deviance of a Poisson count rises faster below the mean than above it, and the two departures from 2.5 do not cancel.

Problem 2 (separation with four points). A logistic regression with an intercept and one input $x$ is fitted to four points, $x = -2, -1, 1, 2$. (a) With outcomes 0, 0, 1, 1, show that no estimate exists, and say what the iterations do. (b) With outcomes 0, 1, 0, 1, show that the estimate exists, and find it.

Solution. (a) Take $\mathbf{b} = (0, 1)$, so that $\mathbf{x}_i^{\mathsf T}\mathbf{b} = x_i$. The successes have $x = 1, 2 > 0$ and the failures $x = -2, -1 < 0$, so $s_ix_i > 0$ for all four points: complete separation. By Theorem 4 no estimate exists. The log-likelihood along $\boldsymbol\beta = (0, t)$ is $2\ln\sigma(t) + 2\ln\sigma(2t)$, which increases to 0 as $t \to \infty$; the fitted curve becomes a step at $x = 0$, and the iterations increase the slope without end.

(b) Now the point $x = -1$ is a success and $x = 1$ a failure. Suppose $\mathbf{b} = (b_0, b_1)$ separates. The four conditions $s_i(b_0 + b_1x_i) \ge 0$ are: $-(b_0 - 2b_1) \ge 0$, $b_0 - b_1 \ge 0$, $-(b_0 + b_1) \ge 0$, $b_0 + 2b_1 \ge 0$. The second and third give $b_1 \le b_0 \le -b_1$, so $b_1 \le 0$; the first and fourth give $-2b_1 \le b_0 \le 2b_1$, so $b_1 \ge 0$. Hence $b_1 = 0$, and then $b_0 = 0$. Only $\mathbf{b} = \mathbf{0}$ satisfies them, so the data overlap and the estimate exists. The data are unchanged by the map $x \to -x$ with the outcomes exchanged ($y \to 1 - y$). Under that map each term $\ln\sigma(s_i(\beta_0 + \beta_1x_i))$ becomes $\ln\sigma(-s_i(\beta_0 - \beta_1x_i))$, so $\ell(\beta_0, \beta_1) = \ell(-\beta_0, \beta_1)$, and since the maximum is unique, $\hat\beta_0 = -\hat\beta_0 = 0$. The slope solves the remaining likelihood equation $\sum_i x_i(y_i - \sigma(\beta_1x_i)) = 0$. With $\sigma(-t) = 1 - \sigma(t)$ its four terms sum to $4 - 4\sigma(2\beta_1) - 2\sigma(\beta_1)$, so the equation is $\sigma(2\beta_1) + \tfrac12\sigma(\beta_1) = 1$. Its root is $\hat\beta_1 = 0.4196$, with maximized log-likelihood $-2.5678$. The single overlapping pair is enough to keep the slope finite, and it keeps it small.

Problem 3 (the score test by hand). Compute the score statistic of Theorem 3 for obesity in the three age groups from the six rows.

Solution. In each age group the exposed are the obese. Their cases are $a_k = 22, 140, 270$, so $\sum a_k = 432$. The expected values $E_k = n_{1k}m_{1k}/n_k$ are $640 \times 34/1{,}680 = 12.95$, $697 \times 209/1{,}557 = 93.56$ and $762 \times 534/1{,}870 = 217.60$, with sum 324.11. The variances $n_{1k}n_{0k}m_{1k}m_{0k}/n_k^3$ are 7.86, 44.74 and 92.11, with sum 144.71. So $W_{\mathrm{S}} = (432 - 324.11)^2/144.71 = 80.44$, the value in the table of tests. With the Mantel–Haenszel variances (each larger by $n_k/(n_k - 1)$) the statistic is 80.39, the square of the $z_{\mathrm{MH}} = 8.97$ of The Mantel–Haenszel Test.

Summary

Quantity Formula
Deviance $D = 2\phi[\ell(\mathbf{y}; \mathbf{y}) - \ell(\hat{\boldsymbol\mu}; \mathbf{y})] = \sum_i w_i d(y_i; \hat\mu_i)$
Unit deviances normal $(y - \mu)^2$; Poisson $2[y\ln(y/\mu) - (y - \mu)]$; binomial $2[y\ln\frac{y}{\mu} + (1 - y)\ln\frac{1 - y}{1 - \mu}]$
Nested models $(D_0 - D_1)/\phi = W_{\mathrm{LR}}$, approximately $\chi^2_{p_1 - p_0}$ (Wilks)
Goodness of fit $D \approx \chi^2_{n - p}$ for a fixed number of large groups; meaningless for 0/1 data
Pearson $X^2 = \sum w_i(y_i - \hat\mu_i)^2/V(\hat\mu_i)$; $d(y; \mu) = (y - \mu)^2/V(\mu) + O(\lvert y - \mu\rvert^3)$
Residuals $r_P$, $r_D$ with $\sum r_P^2 = X^2$, $\sum r_D^2 = D$; standardized $r_P/\sqrt{1 - h_{ii}}$
Score test across strata $[\sum(a_k - E_k)]^2/\sum n_{1k}n_{0k}m_{1k}m_{0k}/n_k^3$
Existence (logistic) exists, uniquely, iff no $\mathbf{b} \ne \mathbf{0}$ with $s_i\mathbf{x}_i^{\mathsf T}\mathbf{b} \ge 0$ for all $i$

What we have learned

The deviance compares a fitted model with the saturated one, and its differences between nested models are likelihood ratio statistics, the same whether the data are grouped or not. As a test of fit on its own it works for a fixed number of large groups and fails completely for individual binary records, where it is a function of the fitted values. In the diabetes data the deviance, the score and the Wald statistic agree that obesity matters and that its odds ratio declines with age, and the score test for an exposure across strata is the Mantel–Haenszel test. When the outcomes can be separated by a linear score, which in small samples of a rare outcome is common, the estimate does not exist, the Wald test collapses, and the likelihood ratio and penalized likelihood still give usable answers.

The binomial family has $\phi = 1$, and so does the Poisson. Poisson Regression and Rate Models applies the deviance to rates and finds deviances far larger than their degrees of freedom: the variation of real counts around a well-fitting mean exceeds what $\phi = 1$ allows. Overdispersion and Quasi-Likelihood estimates $\phi$ and asks what remains valid.

References