Link Functions and the Structure of a Generalized Linear Model

How a generalized linear model joins an exponential family response to a linear predictor through a link function, and what the choice of link decides. The three components are defined; transforming the response is shown to answer a different question; the canonical link is proved to make the log-likelihood strictly concave, the weighted totals of each input sufficient, and the fitted totals equal to the observed ones; the logit, probit and complementary log-log links are derived from latent thresholds and from Poisson counts; and on NHANES diabetes data the five binomial links are compared, the absence of interaction is shown to depend on the scale, and the identity link is shown to put its maximum on the boundary where the likelihood equations fail. Worked problems on the canonical links of the gamma and inverse Gaussian families, the complementary log-log link with exposure, the two-group model under every link, and symmetric links.

Among the 5,107 adults of this unit's running population, drawn from the National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018, the share with diagnosed diabetes rises from 1.15% among adults aged 18 to 39 who are not obese to 35.43% among those aged 60 and over who are. A regression model should let both age and obesity raise that share while keeping it between 0 and 1. A straight line in age cannot do that for every age, and Logistic Regression showed one remedy: model the log-odds, not the probability, as a linear function. The log-odds is one choice among several. What exactly does such a choice connect, which choice is the natural one for a given family of distributions, and what do the different choices say about the data?

A link function is the function $g$ that connects the mean $\mu_i$ of a response to a linear combination of inputs: $g(\mu_i) = \mathbf{x}_i^{\mathsf T}\boldsymbol\beta$. Together with a distribution for the response and the linear combination itself, it makes up a generalized linear model, the framework that Nelder and Wedderburn (1972) introduced under that name for observations, in the words of their abstract, "distributed according to some exponential family and systematic effects that can be made linear by a suitable transformation".

First we define the three components of the model and ask why the mean is transformed rather than the response. Then we prove what the canonical link, the one that matches the natural parameter of Exponential Families, does for the likelihood. Next we derive the standard links for a probability from latent variables and from counts, and read their coefficients. Finally we fit all five binomial links to the diabetes data, see that whether two factors interact depends on the link, and find the identity link failing at the edge of its range. The article assumes the exponential dispersion form and its properties from Exponential Families (Theorem 7 there: $E[Y] = b'(\theta)$, $\operatorname{Var}(Y) = \phi V(\mu)/w$) and the matrix notation of Linear Models.

The three components

A generalized linear model for responses $Y_1, \ldots, Y_n$ has three parts.

  1. The random component. The $Y_i$ are independent, and each has a distribution of the exponential dispersion form $$f(y_i; \theta_i, \phi) = \exp\Big\{\frac{w_i[\,y_i\theta_i - b(\theta_i)\,]}{\phi} + c(y_i, \phi/w_i)\Big\},$$ with its own natural parameter $\theta_i$, a cumulant function $b$ and a dispersion $\phi$ shared by all observations, and a known prior weight $w_i$. Its mean is $\mu_i = b'(\theta_i)$ and its variance $\phi V(\mu_i)/w_i$.
  2. The systematic component. A linear predictor $\eta_i = \mathbf{x}_i^{\mathsf T}\boldsymbol\beta$, where $\mathbf{x}_i^{\mathsf T}$ is row $i$ of an $n \times p$ matrix $X$ of full column rank $p$ (the intercept column counted in $p$), and $\boldsymbol\beta$ holds $p$ unknown coefficients. A known offset $o_i$ may be added, $\eta_i = o_i + \mathbf{x}_i^{\mathsf T}\boldsymbol\beta$; Poisson Regression and Rate Models uses it for exposure.
  3. The link. A strictly monotone, differentiable function $g$ with $g(\mu_i) = \eta_i$, so that $\mu_i = g^{-1}(\eta_i)$.

The normal linear model is the case of a normal response with the identity link $g(\mu) = \mu$: then $\mu_i = \mathbf{x}_i^{\mathsf T}\boldsymbol\beta$ and the dispersion is the error variance $\sigma^2$. Logistic regression is the case of a binomial response with the logit link. Efron and Hastie (2016, Section 8.2) set out the model in the same three steps: a one-parameter exponential family for each observation, a structure matrix $X$, and a linear specification of a parameter.

In the running example the response is the diabetes indicator, a binomial proportion. For the hand-sized version of the data, the adults are grouped by age band (18 to 39, 40 to 59, 60 and over) and obesity (body mass index of 30 or more), which gives six binomial observations: $y_i$ is the share with diabetes and the prior weight $w_i = m_i$ is the size of the group.

Group Diabetes / adults Share $y_i$
18 to 39, not obese 12 / 1,040 0.0115
18 to 39, obese 22 / 640 0.0344
40 to 59, not obese 69 / 860 0.0802
40 to 59, obese 140 / 697 0.2009
60 and over, not obese 264 / 1,108 0.2383
60 and over, obese 270 / 762 0.3543

The design matrix has four columns: an intercept, indicators for the two older age bands, and an indicator for obesity. So the model says that age band and obesity act additively on the scale of $g$.

Why transform the mean and not the response?

A long-standing alternative to a generalized linear model is to transform the response, $\ln y$ or $\operatorname{logit} y$, and fit a linear model to the result by least squares. The two are not the same model.

Example (the logit of a single response). For one adult the response is 0 or 1, and $\ln\frac{y}{1 - y}$ is $-\infty$ or $+\infty$. No least squares fit can use it. Grouping helps only where every group has at least one case and one non-case, and the youngest group's 12 cases out of 1,040 show how close to that edge real data can be.

Even where the transformed response is finite, it answers a different question. If $g$ is strictly convex, Jensen's inequality (Probability Inequalities) gives $E[g(Y)] > g(E[Y])$ whenever $Y$ is not constant, and the reverse for strictly concave $g$. So a linear model for $E[\ln Y]$ is a model for the mean of the logarithm, which is not the logarithm of the mean, and the gap depends on the variance of $Y$. A generalized linear model with the log link describes $\ln E[Y]$ directly.

The transformation also has to do two jobs at once: make the mean linear in the inputs and make the variance constant, so that least squares is appropriate. One function rarely does both. A generalized linear model separates the jobs. The link describes how the mean depends on the inputs, the variance function $V(\mu)$ of the family describes how the spread depends on the mean, and the fitting algorithm of Iteratively Reweighted Least Squares weights each observation accordingly.

Any strictly monotone $g$ that maps the possible means onto the range of the linear predictor can serve as a link. One choice is singled out by the family. By Theorem 3 of Exponential Families, the mean $\mu = b'(\theta)$ is a strictly increasing function of the natural parameter, with inverse $\theta(\mu) = (b')^{-1}(\mu)$. The canonical link is that inverse, $g = (b')^{-1}$. With it, the natural parameter of each observation is its linear predictor: $\theta_i = \eta_i$.

Family Canonical link $g(\mu)$ Range of $\eta$
Normal $\mu$ (identity) all real numbers
Poisson $\ln\mu$ (log) all real numbers
Binomial proportion $\ln\frac{\mu}{1 - \mu}$ (logit) all real numbers
Gamma $-1/\mu$ (negative reciprocal) $\eta < 0$
Inverse Gaussian $-1/(2\mu^2)$ $\eta < 0$

The table follows from the $\theta(\mu)$ column of the dispersion table in Exponential Families. Here is what the choice buys, in one sentence: with $\theta_i = \eta_i$ the log-likelihood is linear in $\boldsymbol\beta$ except for a sum of convex functions, so the data enter through one linear statistic and the curvature does not depend on the data.

Theorem 1 (the canonical link). Suppose the link is canonical, $\eta_i = o_i + \mathbf{x}_i^{\mathsf T}\boldsymbol\beta$, and the dispersion $\phi$ is known. Let $\mathbf{w}\circ\mathbf{y}$ be the vector with entries $w_iy_i$.

  1. The log-likelihood is $$\ell(\boldsymbol\beta) = \frac{1}{\phi}\Big[\boldsymbol\beta^{\mathsf T}X^{\mathsf T}(\mathbf{w}\circ\mathbf{y}) - \sum_{i=1}^n w_i\,b(o_i + \mathbf{x}_i^{\mathsf T}\boldsymbol\beta)\Big] + \text{const},$$ so $X^{\mathsf T}(\mathbf{w}\circ\mathbf{y}) = \sum_i w_iy_i\mathbf{x}_i$ is a sufficient statistic for $\boldsymbol\beta$, of dimension $p$ whatever $n$ is.
  2. The score is $\mathbf{U}(\boldsymbol\beta) = \frac1\phi X^{\mathsf T}(\mathbf{w}\circ(\mathbf{y} - \boldsymbol\mu))$, and the likelihood equations say that for every column $j$ of $X$, $\sum_i w_ix_{ij}\hat\mu_i = \sum_i w_ix_{ij}y_i$. With an intercept, the weighted fitted total equals the observed total; with an indicator column, the same holds within the group it marks.
  3. The matrix of second derivatives is $-\frac1\phi X^{\mathsf T}\operatorname{diag}(w_iV(\mu_i))X$. It does not depend on $\mathbf{y}$, and it is negative definite, so $\ell$ is strictly concave and has at most one maximizer; any solution of the likelihood equations is the maximum likelihood estimate.

Proof. (1) Substitute $\theta_i = \eta_i$ in the log of the product of the densities. The terms $w_iy_i\eta_i/\phi$ add up to $[\sum_i w_iy_io_i + \boldsymbol\beta^{\mathsf T}\sum_i w_iy_i\mathbf{x}_i]/\phi$; the first part and the $c(y_i, \phi/w_i)$ do not involve $\boldsymbol\beta$. The likelihood is therefore a function of $\boldsymbol\beta$ and $X^{\mathsf T}(\mathbf{w}\circ\mathbf{y})$ times a function of the data alone, which is the factorization criterion of Sufficient Statistics.

(2) By the chain rule, $\partial b(\eta_i)/\partial\beta_j = b'(\eta_i)x_{ij} = \mu_ix_{ij}$, and the linear term contributes $\sum_i w_iy_ix_{ij}$. So $\partial\ell/\partial\beta_j = \frac1\phi\sum_i w_ix_{ij}(y_i - \mu_i)$. For an intercept $x_{ij} = 1$; for an indicator, $x_{ij}$ is 1 in the group and 0 elsewhere.

(3) Differentiating again, $\partial^2\ell/\partial\beta_j\partial\beta_k = -\frac1\phi\sum_i w_ib''(\eta_i)x_{ij}x_{ik}$, and $b''(\eta_i) = V(\mu_i)$ by the definition of the variance function. For any nonzero vector $\mathbf{a}$, $\mathbf{a}^{\mathsf T}X^{\mathsf T}\operatorname{diag}(w_iV(\mu_i))X\mathbf{a} = \sum_i w_iV(\mu_i)(\mathbf{x}_i^{\mathsf T}\mathbf{a})^2 > 0$, because the weights and variances are positive and $X\mathbf{a} \ne \mathbf{0}$ when $X$ has full column rank. A function whose matrix of second derivatives is negative definite everywhere on a convex set is strictly concave there; two maximizers would give their midpoint a larger value; and a stationary point of a concave function is a maximum, since the function lies below its tangent plane. $\square$

The second derivatives in part 3 involve no $y_i$, so the observed and expected information matrices coincide. That is the fact behind the equality of Newton's method and Fisher scoring for canonical links, proved in Iteratively Reweighted Least Squares. Part 1 is Efron and Hastie's observation (2016, eqs. 8.24 and 8.25) that a generalized linear model with the canonical link is itself a $p$-parameter exponential family, with sufficient statistic $X^{\mathsf T}\mathbf{y}$ in their unweighted notation.

Example (fitted totals on the six groups). With the logit link, the additive model reproduces four totals exactly: 777 diabetic adults in all, 209 in the 40 to 59 band, 534 aged 60 and over, and 432 among the obese. These are the four entries of $X^{\mathsf T}(\mathbf{w}\circ\mathbf{y})$, and part 2 says that the fit must match them. With the probit link, which is not canonical for the binomial, the same model gives fitted totals of 776.51, 207.79, 534.79 and 433.36. Neither fit is wrong. The probit fit satisfies its own likelihood equations, which weight the residuals differently.

Example (the misreading "the canonical link is the right link"). The canonical link is the convenient one: it gives concavity, sufficiency and the simplest equations. Whether it describes the data best is a separate question. On the six groups the probit link has the larger log-likelihood ($-20.40$ against $-22.09$), and on all 5,107 adults with age in years as well ($-1{,}864.1$ against $-1{,}873.6$). The two pairs are on different scales. Log-likelihoods of the six grouped observations include the binomial coefficients $\ln\binom{m_i}{m_iy_i}$, which add 1,836.86 to the Bernoulli log-likelihood of the same adults; the log-likelihoods of the 5,107 individual records do not. Differences between models fitted to the same data are unaffected. Efron and Hastie (2016, p. 120) write that logistic regression displaced probit analysis "not only because of estimating efficiencies and computational advantages (which are actually rather modest)", and go on to the clearer analogy with ordinary least squares. How to compare the fits of models with different links is taken up in Deviance and Inference for Generalized Linear Models; the log-likelihoods here are descriptions of these data, not tests.

For a binary or binomial response the mean is a probability, between 0 and 1, and the linear predictor ranges over all real numbers. Any continuous, strictly increasing distribution function $F$ maps the real line onto $(0, 1)$, so $g = F^{-1}$ is a link. Each standard link comes from such an $F$, and the choice of $F$ has a concrete reading.

Theorem 2 (links from latent thresholds). Let $Z_i$ have a continuous, strictly increasing distribution function $F$, and let the binary response be $Y_i = 1$ when $Z_i \le \eta_i$ and $Y_i = 0$ otherwise. Then $\mu_i = P(Y_i = 1) = F(\eta_i)$, so the link is $g = F^{-1}$. In particular:

  • the logit link $\ln\frac{\mu}{1 - \mu}$ comes from the logistic distribution, $F(z) = 1/(1 + e^{-z})$;
  • the probit link $\Phi^{-1}(\mu)$ comes from the standard normal distribution function $\Phi$;
  • the complementary log-log link $\ln(-\ln(1 - \mu))$ comes from the distribution $F(z) = 1 - \exp(-e^{z})$, the smallest-extreme-value (Gumbel minimum) distribution.

Proof. $P(Y_i = 1) = P(Z_i \le \eta_i) = F(\eta_i)$, and $F$ is invertible because it is continuous and strictly increasing. For the logistic distribution, solving $\mu = 1/(1 + e^{-\eta})$ gives $\eta = \ln\frac{\mu}{1 - \mu}$, as Logistic Regression showed. For the third, $\mu = 1 - \exp(-e^{\eta})$ gives $e^{\eta} = -\ln(1 - \mu)$, so $\eta = \ln(-\ln(1 - \mu))$. $\square$

The threshold reading is the origin of probit analysis in bioassay. Efron and Hastie (2016, eqs. 8.50 and 8.51) describe it: each test animal has its own lethal dose, the doses are normally distributed in the population, and the probability of death at dose $x$ is $\Phi(\beta_0 + \beta_1x)$; replacing $\Phi$ by the logistic distribution function turns the model into logistic regression. Here the latent variable can be read as each adult's own threshold of age and obesity at which diabetes appears.

Three differences between the links follow from the three distributions.

  • Scale. The standard logistic distribution has standard deviation $\pi/\sqrt3 = 1.814$, the standard normal 1. If the two latent distributions had the same shape apart from scale, logit coefficients would be 1.814 times probit coefficients. On the six groups the ratios of the three slopes are 2.13, 1.98 and 1.74, and on the full data with age in years they are 1.78 for age and 1.79 for obesity. The ratios are not constant, so the two models do not differ only by a scale factor; this page does not measure where the difference arises.
  • Symmetry. The logistic and normal distributions are symmetric about 0, so their links satisfy $g(1 - \mu) = -g(\mu)$: a model for "diabetes" and a model for "no diabetes" have coefficients of opposite sign and nothing else changes. The complementary log-log link is not symmetric (Worked problem 4).
  • Rare outcomes. For small $\mu$, $\ln\frac{\mu}{1 - \mu} = \ln\mu - \ln(1 - \mu)$ and $\ln(-\ln(1 - \mu)) = \ln(\mu + \mu^2/2 + \cdots)$ are both $\ln\mu$ plus a term that vanishes as $\mu \to 0$. So when the outcome is rare everywhere, the logit, complementary log-log and log links give nearly the same coefficients. The fitted probabilities in the youngest non-obese group are close under all three: 0.0144 (logit), 0.0152 (complementary log-log) and 0.0160 (log).

The complementary log-log link has a second derivation, from counts.

Theorem 3 (the complementary log-log link from Poisson counts). Let $N_i$ be a Poisson count with mean $t_i e^{\mathbf{x}_i^{\mathsf T}\boldsymbol\beta}$, where $t_i > 0$ is a known exposure (a length of time, an area), and let only $Y_i = 1$ if $N_i \ge 1$, $Y_i = 0$ otherwise, be observed. Then $\ln(-\ln(1 - \mu_i)) = \ln t_i + \mathbf{x}_i^{\mathsf T}\boldsymbol\beta$: the complementary log-log link with offset $\ln t_i$, and $e^{\beta_j}$ is the ratio of the underlying event rates for a unit change in $x_{ij}$.

Proof. $1 - \mu_i = P(N_i = 0) = \exp(-t_ie^{\mathbf{x}_i^{\mathsf T}\boldsymbol\beta})$. Taking logarithms twice, $\ln(-\ln(1 - \mu_i)) = \ln t_i + \mathbf{x}_i^{\mathsf T}\boldsymbol\beta$. The rate is $e^{\mathbf{x}_i^{\mathsf T}\boldsymbol\beta}$, and changing $x_{ij}$ by 1 multiplies it by $e^{\beta_j}$. $\square$

So when a 0/1 outcome records whether at least one event of a Poisson process occurred during a known time, the complementary log-log link recovers the rate ratios of the process, and the log of the time is an offset, as in Poisson Regression and Rate Models.

Under any link, $\beta_j$ is the change in $g(\mu)$ when $x_{ij}$ increases by 1 and the other inputs are held fixed, because $g(\mu)$ is linear in the inputs. What that change means depends on $g$.

Link $\beta_j$ is the change in $e^{\beta_j}$ or $\beta_j$ reads as
Logit the log-odds $e^{\beta_j}$: odds ratio
Log $\ln\mu$ $e^{\beta_j}$: risk ratio (relative risk)
Identity $\mu$ $\beta_j$: risk difference
Complementary log-log $\ln(-\ln(1 - \mu))$ $e^{\beta_j}$: ratio of underlying rates (Theorem 3)
Probit $\Phi^{-1}(\mu)$ $\beta_j$: shift of the latent normal threshold, in standard deviations

The measures in the first three rows are those of Relative Risk and Odds Ratio; a generalized linear model estimates them adjusted for the other inputs. On the six groups, the obesity coefficient of the additive model gives an odds ratio of $e^{0.7386} = 2.093$ (logit), a risk ratio of 1.709 (log), a risk difference of 0.0582 (identity), a rate ratio of 1.885 (complementary log-log), and a threshold shift of 0.425 standard deviations (probit). Each is a single number for all three age bands, because each model has no interaction term.

Link Intercept 40 to 59 60 and over Obese Log-likelihood
Logit −4.2250 1.9808 2.9793 0.7386 −22.09
Probit −2.2565 0.9314 1.5063 0.4246 −20.40
Complementary log-log −4.1786 1.9189 2.7941 0.6337 −23.48
Log −4.1378 1.8613 2.6230 0.5360 −25.08
Identity 0.0088 0.0937 0.2514 0.0582 −34.96

The identity link fits worst by far: it forces the obesity effect to add the same 0.0582 to the probability at every age, and it fits the youngest non-obese group at 0.0088.

No interaction on one scale is interaction on another

Is the effect of obesity the same at every age? The question has no answer until a scale is named. Within each age band, the counts give the obesity effect exactly on three scales:

Age band Odds ratio Risk ratio Risk difference
18 to 39 3.05 2.98 0.023
40 to 59 2.88 2.50 0.121
60 and over 1.75 1.49 0.116

The risk difference grows with age between the first two bands, while both ratios fall. A model with no interaction on the logit scale holds the odds ratio fixed at 2.093 and lets everything else move: its fitted risk differences are 0.015, 0.086 and 0.152, and its fitted risk ratios 2.06, 1.89 and 1.68. These are forced by the model, not estimated separately for each band. If the odds ratio were exactly constant, the risk difference would have to vary with the baseline risk, because the logistic curve is steepest at $\mu = 1/2$: the derivative of $\mu$ with respect to $\eta$ is $\mu(1 - \mu)$, the source of the divide-by-4 rule in Logistic Regression.

Example (the misreading "the model shows that obesity and age do not interact"). An additive model does not show it; it assumes it, on its own scale. On these data the ratios fall with age, so the additive logit model is itself open to question. Whether the departure is larger than chance would produce is a test of an interaction term, which Deviance and Inference for Generalized Linear Models provides. Whichever link is chosen, "no interaction" is a statement about that link's scale, and two analysts using different links can both be right about their own scales.

Share of adults with diabetes in six age bands (points, with Wald 95% intervals, the share plus or minus 1.96 standard errors), for adults who are not obese (left) and who are (right), with the fitted curves of four binomial models in age (years) and obesity. Notice that the logit, complementary log-log and log curves stay within 0.03 of one another at every age, so that these data barely distinguish them, while in the left panel the identity line runs above the observed shares in the four age bands from 18 to 59 and reaches exactly 0 at age 18, the boundary discussed below.
Share of adults with diabetes in six age bands (points, with Wald 95% intervals, the share plus or minus 1.96 standard errors), for adults who are not obese (left) and who are (right), with the fitted curves of four binomial models in age (years) and obesity. Notice that the logit, complementary log-log and log curves stay within 0.03 of one another at every age, so that these data barely distinguish them, while in the left panel the identity line runs above the observed shares in the four age bands from 18 to 59 and reaches exactly 0 at age 18, the boundary discussed below.

The logit, probit and complementary log-log links map $(0, 1)$ onto the whole real line, so every $\boldsymbol\beta$ gives valid probabilities. The log link maps $(0, 1)$ onto the negative numbers, and the identity link onto $(0, 1)$ itself; with either, some coefficients give "probabilities" above 1 or below 0. On the six groups that never happens at the estimates. On all 5,107 adults, with age in years as a single input (NHANES records ages of 80 and over as 80) and the obesity indicator, the fits are these:

Link Intercept Age (per year) Obese Log-likelihood
Logit −5.2876 0.05695 0.8051 −1,873.6
Probit −3.0044 0.03194 0.4498 −1,864.1
Complementary log-log −4.9391 0.04971 0.6919 −1,880.9
Log −4.5866 0.04257 0.5749 −1,890.2
Identity (on the boundary) −0.0775 0.00431 0.0442 −1,904.7

The log link fits without trouble, with a largest fitted probability of 0.546. It gives a risk ratio of $e^{10 \times 0.04257} = 1.53$ per decade of age, against an odds ratio of 1.77 per decade from the logit link.

The identity link does not fit without trouble. A straight line in age for the probability of diabetes, steep enough to reach the older adults, passes below 0 before age 18. The maximum of the likelihood over the coefficients that keep every probability in $[0, 1]$ therefore lies on the edge of that set.

Theorem 4 (a maximum on the boundary). For the binomial model with the identity link, $\mu_i = \beta_0 + \beta_1\,\text{age}_i + \beta_2\,\text{obese}_i$, on the 5,107 adults, let $\hat{\boldsymbol\beta} = (-0.0775, 0.00431, 0.0442)$ be the maximizer of the log-likelihood among coefficients with $\beta_0 + 18\beta_1 = 0$. Then $\hat{\boldsymbol\beta}$ maximizes the log-likelihood over all coefficients that give every adult a probability in $[0, 1]$ (and a positive one to every adult with diabetes). At $\hat{\boldsymbol\beta}$ the score is not zero, and the likelihood equations have no solution with valid probabilities.

The idea: the log-likelihood is concave, and at $\hat{\boldsymbol\beta}$ its gradient points straight out of the allowed set, so no allowed direction leads uphill.

Proof. The log-likelihood $\ell(\boldsymbol\beta) = \sum_i [y_i\ln\mu_i + (1 - y_i)\ln(1 - \mu_i)]$ is a sum of concave functions of the linear functions $\mu_i = \mathbf{x}_i^{\mathsf T}\boldsymbol\beta$, so it is concave on the convex set $\mathcal{B}$ of allowed coefficients. At $\hat{\boldsymbol\beta}$ the 93 non-obese adults aged 18 have $\mu_i = 0$, and none of them has diabetes, so their terms are $\ln(1 - \mu_i)$, which is differentiable; every adult with diabetes has $\mu_i > 0$ (the smallest, for a non-obese adult aged 22, is 0.0172), and the largest fitted probability is 0.311, for obese adults at the top code of 80. So $\ell$ is differentiable at $\hat{\boldsymbol\beta}$. The script computes its gradient there and finds $\nabla\ell(\hat{\boldsymbol\beta}) = -\lambda\,\mathbf{c}$ with $\mathbf{c} = (1, 18, 0)$ and $\lambda = 1{,}178.4 > 0$; the last component is zero because $\hat{\boldsymbol\beta}$ maximizes $\ell$ along the face. For any $\boldsymbol\beta$ in $\mathcal{B}$, concavity puts $\ell$ below its tangent plane:

$$\ell(\boldsymbol\beta) \le \ell(\hat{\boldsymbol\beta}) + \nabla\ell(\hat{\boldsymbol\beta})^{\mathsf T}(\boldsymbol\beta - \hat{\boldsymbol\beta}) = \ell(\hat{\boldsymbol\beta}) - \lambda\,\mathbf{c}^{\mathsf T}\boldsymbol\beta \le \ell(\hat{\boldsymbol\beta}),$$

because $\mathbf{c}^{\mathsf T}\hat{\boldsymbol\beta} = \beta_0 + 18\beta_1 = 0$ and $\mathbf{c}^{\mathsf T}\boldsymbol\beta$ is the probability of a non-obese adult aged 18, which is at least 0 in $\mathcal{B}$. So $\hat{\boldsymbol\beta}$ is a maximizer, with score $-\lambda\mathbf{c} \ne \mathbf{0}$. Finally, $\ell$ is strictly concave on $\mathcal{B}$: its matrix of second derivatives is $-\sum_i[y_i/\mu_i^2 + (1 - y_i)/(1 - \mu_i)^2]\mathbf{x}_i\mathbf{x}_i^{\mathsf T}$, and the weights are positive on the many rows with $y_i = 0$, which span all three coordinates. A valid solution of the likelihood equations would be a second maximizer (a stationary point of a concave function is a maximum), and strict concavity allows only one, which has a nonzero score. $\square$

The maximum has log-likelihood $-1{,}904.7$, below the logit fit's $-1{,}873.6$ with the same number of coefficients. More important than the fit is the failure of the standard machinery. The large-sample theory of Asymptotics of the Maximum Likelihood Estimator describes a root of the likelihood equations at an interior point. Here there is no root, the estimate sits on the boundary, and Wald intervals computed from it are not justified. A fitting routine that ignores the range simply fails: Iteratively Reweighted Least Squares shows the algorithm stalling at the same edge. Log-binomial models for risk ratios can meet the same difficulty when fitted probabilities approach 1, since the log link allows probabilities above 1; it did not arise in the log-link fits above.

What is proved, what is cited, what the experiment shows

Proved on this page: the consequences of the canonical link (Theorem 1: sufficiency of $X^{\mathsf T}(\mathbf{w}\circ\mathbf{y})$, the likelihood equations as matched totals, a nonrandom negative definite matrix of second derivatives and at most one maximum); the derivation of the logit, probit and complementary log-log links from latent thresholds (Theorem 2) and of the complementary log-log link from Poisson counts with exposure (Theorem 3); the boundary maximum of the identity-link model on these data (Theorem 4, with the gradient computed by the script); the readings of the coefficients; the small-$\mu$ agreement of the logit, log and complementary log-log links.

Used from other pages: the dispersion form and its mean and variance, and the monotone relation between the natural parameter and the mean (Exponential Families); Jensen's inequality; the factorization criterion for sufficiency.

Cited: the framework and its name (Nelder and Wedderburn, 1972, from the abstract); the remark on logistic regression's advantages (Efron and Hastie, 2016, Section 8.2, p. 120) and the bioassay derivation of probit analysis (their Section 8.5, eqs. 8.50 and 8.51).

Computed on the unit's data: the fits of the five links to the six groups and four links to the 5,107 adults, by glmdata.fit (Newton's method; Fisher scoring gives the same estimates); the scale-specific effects within each age band; the boundary maximum of the identity link. These describe these adults as a finite population, without the survey weights, and are not tests: whether the probit link or an interaction is supported beyond chance is a question for the deviance article. Every number is computed and checked against this page by tools/experiments/statistics/link-functions.py, and the figure is drawn by tools/figures/link-functions.py.

Worked problems

The problems were written for this article, on the unit's NHANES groups and on the families of Exponential Families. Each number is recomputed by the article's script.

Problem 1 (canonical links of the gamma and inverse Gaussian families). Find the canonical links of the gamma and inverse Gaussian families, and say what constraint they put on the linear predictor.

Solution. For the gamma family, $b(\theta) = -\ln(-\theta)$ on $\theta < 0$, so $b'(\theta) = -1/\theta = \mu$, and the inverse is $\theta = -1/\mu$. The canonical link is $g(\mu) = -1/\mu$, and since $\mu > 0$ the linear predictor must be negative. Writing the link as $1/\mu$ and absorbing the sign into $\boldsymbol\beta$ gives the reciprocal link with $\eta > 0$. Either way, coefficients that make $\eta$ cross 0 at some input give a negative or infinite mean, so the canonical link of the gamma family has the range problem of the previous section, which is one reason to prefer a log link for gamma responses. For the inverse Gaussian family, $b(\theta) = -\sqrt{-2\theta}$ gives $b'(\theta) = (-2\theta)^{-1/2} = \mu$, so $\theta = -1/(2\mu^2)$: the canonical link is $-1/(2\mu^2)$, again with $\eta < 0$. By Theorem 1, with either canonical link and a full-rank $X$ the log-likelihood is strictly concave in $\boldsymbol\beta$ on the coefficients it allows.

Problem 2 (the complementary log-log link with exposure). An adult is followed for $t$ years, and $Y = 1$ records at least one event of a Poisson process with rate $r$ per year. Show that doubling the rate does not double $P(Y = 1)$, and find $P(Y = 1)$ for $rt = 0.1054$ (a probability of 0.1) before and after doubling. What does the complementary log-log coefficient of a factor that doubles the rate equal?

Solution. By Theorem 3, $P(Y = 1) = 1 - e^{-rt}$. With $rt = -\ln 0.9 = 0.1054$ the probability is $1 - 0.9 = 0.1$. Doubling the rate gives $1 - e^{-2rt} = 1 - 0.9^2 = 0.19$, not 0.2, because the second event in a period adds nothing to $Y$. On the complementary log-log scale the change is exact: $\ln(-\ln(1 - 0.19)) - \ln(-\ln(1 - 0.1)) = \ln(2rt) - \ln(rt) = \ln 2$. So the coefficient of a factor that doubles the rate is $\ln 2 = 0.6931$, whatever the baseline probability, and $e^{\beta} = 2$ is the rate ratio.

Problem 3 (two groups under every link). Fit the model with an intercept and the obesity indicator only, so that there are two groups: 345 of 3,008 adults who are not obese and 432 of 2,099 who are obese have diabetes. Show that under any link the fitted probabilities are the two observed shares, and find the obesity coefficient under each of the five links.

Solution. With two groups and two coefficients, the linear predictor takes one free value in each group: $\eta = \beta_0$ for the non-obese and $\beta_0 + \beta_1$ for the obese, and every pair of values is reached. So the model allows any pair of probabilities $(\mu_0, \mu_1)$ in the range of $g^{-1}$, and the log-likelihood splits into two binomial log-likelihoods, one for each group. Each is maximized by the group's own share (Maximum Likelihood Estimation): $\hat\mu_0 = 345/3{,}008 = 0.1147$ and $\hat\mu_1 = 432/2{,}099 = 0.2058$. The coefficient is $\hat\beta_1 = g(\hat\mu_1) - g(\hat\mu_0)$:

  • logit: $\ln\frac{0.2058/0.7942}{0.1147/0.8853}$, the log of the odds ratio 2.000, which is 0.6933;
  • log: the log of the risk ratio 1.794, which is 0.5847;
  • identity: the risk difference 0.0911;
  • probit: $\Phi^{-1}(0.2058) - \Phi^{-1}(0.1147) = 0.3809$;
  • complementary log-log: $\ln(-\ln 0.7942) - \ln(-\ln 0.8853) = 0.6374$.

The fitted probabilities are the same under every link, and only the scale on which the difference is reported changes. Links differ in what they assume only when the model has fewer free coefficients than there are distinct combinations of inputs, as in the additive model of the six groups.

Problem 4 (symmetric and asymmetric links). Show that a link $g = F^{-1}$ from Theorem 2 satisfies $g(1 - \mu) = -g(\mu)$ for all $\mu$ if and only if $F(-z) = 1 - F(z)$ for all $z$. Evaluate the complementary log-log link at $\mu = 0.1$ and $\mu = 0.9$.

Solution. Suppose $F(-z) = 1 - F(z)$ for all $z$. Given $\mu$, put $z = g(\mu)$, so $F(z) = \mu$; then $F(-z) = 1 - \mu$, that is, $g(1 - \mu) = -z = -g(\mu)$. Conversely, if $g(1 - \mu) = -g(\mu)$ for all $\mu$, take any $z$ and put $\mu = F(z)$: then $g(1 - F(z)) = -z$, and applying $F$ gives $1 - F(z) = F(-z)$. The logistic and normal distributions are symmetric about 0, so the logit and probit links have the property. For the complementary log-log link, $g(0.1) = \ln(-\ln 0.9) = -2.2504$ and $g(0.9) = \ln(-\ln 0.1) = 0.8340$: not negatives of each other. Coding "no diabetes" as 1 instead of "diabetes" therefore changes a complementary log-log model, not just the signs of its coefficients; the model with the codes reversed corresponds to the log-log link $-\ln(-\ln\mu)$.

Summary

  • A generalized linear model has independent responses from an exponential dispersion family, a linear predictor $\eta_i = o_i + \mathbf{x}_i^{\mathsf T}\boldsymbol\beta$, and a strictly monotone link with $g(\mu_i) = \eta_i$.
  • Transforming the response models $E[g(Y)]$, not $g(E[Y])$, and fails for 0/1 data; the link models the mean and the variance function handles the spread.
  • The canonical link $g = (b')^{-1}$ makes $\theta_i = \eta_i$: $X^{\mathsf T}(\mathbf{w}\circ\mathbf{y})$ is sufficient, the fitted totals of every column equal the observed ones, and the log-likelihood is strictly concave with curvature free of the data (Theorem 1).
  • The logit, probit and complementary log-log links are inverse distribution functions of logistic, normal and smallest-extreme-value thresholds (Theorem 2); the complementary log-log link also describes "at least one Poisson event" with log exposure as offset (Theorem 3).
  • Coefficients are log odds ratios (logit), log risk ratios (log), risk differences (identity), log rate ratios (complementary log-log), or threshold shifts (probit).
  • "No interaction" holds on one scale at most; on the six groups the odds ratio, risk ratio and risk difference for obesity all vary with age.
  • Links that do not map onto the whole range of $\mu$ can put the maximum on the boundary, where the likelihood equations have no valid solution (Theorem 4).

What we have learned

A generalized linear model separates three decisions: the distribution of the response, the inputs that enter linearly, and the scale on which they do. The canonical link ties the scale to the family's natural parameter and makes the likelihood strictly concave with data-free curvature. Other links buy interpretable coefficients, such as risk ratios and risk differences, at the price of concavity and sometimes of a valid interior maximum. Every model in this article was fitted by the same algorithm, whatever its link. Iteratively Reweighted Least Squares derives that algorithm, shows why each of its steps is a weighted least squares fit, and explains why it converges faster for the canonical link.

References

  • Efron, B., and Hastie, T. (2016). Computer Age Statistical Inference: Algorithms, Evidence, and Data Science. Cambridge University Press. Section 8.2, "Generalized Linear Models", pp. 116–120, and Section 8.5, "Notes and Details", p. 128. record · 9 pages
  • Nelder, J. A., and Wedderburn, R. W. M. (1972). Generalized linear models. Journal of the Royal Statistical Society, Series A, 135(3), 370–384. https://doi.org/10.2307/2344614 record · 5 pages