A generalized linear model keeps the linear predictor of Linear Models, $\eta_i = \mathbf{x}_i^{\mathsf T}\boldsymbol\beta$, and changes two things. The response may have any distribution of an exponential family (binomial, Poisson, gamma and others) instead of the normal, and the linear predictor describes a transformed mean, $g(\mu_i) = \eta_i$, instead of the mean itself. Logistic regression, Poisson regression for rates and the softmax layer of a classifier are all cases. The models were defined by Nelder and Wedderburn (1972), whose abstract opens: "The technique of iterative weighted linear regression can be used to obtain maximum likelihood estimates of the parameters with observations distributed according to some exponential family and systematic effects that can be made linear by a suitable transformation." The library's Logistic Regression uses one case as a working method; the course pages Relative Risk and Odds Ratio and Direct and Indirect Standardization compute by hand quantities that these models estimate with adjustment. This unit is the theory beneath them.
The unit is organized around one structure. Each distribution of the exponential dispersion form, $\exp\{w[y\theta - b(\theta)]/\phi + c(y, \phi/w)\}$, has mean $b'(\theta)$ and variance $\phi\,b''(\theta)/w$, and almost everything else follows from those two derivatives. The mean and variance give the score and the information of the regression. The information gives the weights of the fitting algorithm and the standard errors. The log-likelihood ratio between the fitted model and the data gives the deviance and the tests. Where only the mean and the variance are trusted, the same equations define quasi-likelihood.
What it assumes. The unit on Likelihood and Estimation, especially Maximum Likelihood Estimation, Fisher Information and the Cramér–Rao Bound and Likelihood Ratio, Wald and Score Tests; the matrix notation and weighted least squares of Linear Models (The Geometry of Least Squares and The Gauss–Markov Theorem); and moment generating functions from Probability Theory. Maximum Entropy Distributions explains where exponential families come from, and is useful but not required.
What is proved and what is cited. Proved across the unit: the convexity of the log-partition function and its derivatives as the cumulants of the sufficient statistic, maximum likelihood in an exponential family as moment matching, the determination of a family by its variance function, and the divergence between two members; the consequences of the canonical link (sufficiency, matched totals, strict concavity) and the derivation of the binomial links from latent thresholds and from Poisson counts; the score and both information matrices of any generalized linear model, Fisher scoring as weighted least squares, the equality of Newton's method and scoring for canonical links, the local rate of convergence, and quadratic convergence of Newton's method; the deviance and its relation to likelihood ratio statistics, and the condition under which a maximum likelihood estimate exists; the Poisson rate model with offsets and its link to standardized mortality ratios; the quasi-likelihood equations; and the multinomial model's score, curvature and identifiability. Cited rather than proved: the large-sample distribution of the estimates for several parameters and for observations that do not share a distribution, which rests on the one-parameter theory of Asymptotics of the Maximum Likelihood Estimator and is stated there for several parameters; a few results of the general theory of exponential families, named where they are used.
Notation. Responses $y_1, \ldots, y_n$ with means $\mu_i$; the exponential dispersion form with natural parameter $\theta_i$, cumulant function $b$, dispersion $\phi$ and known prior weights $w_i$; variance function $V(\mu)$, so that $\operatorname{Var}(Y_i) = \phi V(\mu_i)/w_i$. A binomial response is the proportion $y_i$ of successes in $m_i$ trials, with $w_i = m_i$. Linear predictor $\eta_i = o_i + \mathbf{x}_i^{\mathsf T}\boldsymbol\beta$ with an offset $o_i$ where needed, $X$ of size $n \times p$ with the intercept counted in $p$, link $g(\mu_i) = \eta_i$, canonical link $g = (b')^{-1}$. Fitting uses working weights $W_i = w_i/[V(\mu_i)g'(\mu_i)^2]$ and working responses $z_i = \eta_i - o_i + (y_i - \mu_i)g'(\mu_i)$; the expected information is $\mathcal{I}(\boldsymbol\beta) = X^{\mathsf T}WX/\phi$ and the observed information $J(\boldsymbol\beta)$. The deviance is $D = 2\phi[\ell(\mathbf{y}; \mathbf{y}) - \ell(\hat{\boldsymbol\mu}; \mathbf{y})]$ and the scaled deviance $D/\phi$; a Kullback–Leibler divergence is always written with two arguments and a double bar, $D(p \,\|\, q)$. In the general exponential family of the first article the natural parameter is $\boldsymbol\theta$, the sufficient statistic $\mathbf{T}$ and the log-partition function $A$; the letter $\eta$ is kept for the linear predictor.
Data. The running example is the 5,107 adults of the National Health and Nutrition Examination Survey 2017–2018 who are 18 or older, not pregnant, and have a blood pressure reading, a body mass index, an answer to the diabetes question and an answer on smoking. They are treated as a finite population without the survey's weights. Diabetes means a doctor's diagnosis; an answer of borderline diabetes counts as no diabetes. Age is in years, with 80 standing for 80 and over, as NHANES records it. Of them, 777 have diagnosed diabetes, and its share rises with age and with obesity (body mass index of 30 or more). Grouped by three age bands and obesity, the data become six binomial observations, small enough to fit by hand and large enough to separate the links. The Poisson article uses U.S. deaths by age from CDC WONDER and state mortality ratios; the overdispersion article continues the weekly death counts of the likelihood unit; the multinomial article uses self-rated health and the output layer of a small character model. Every number is computed by a script in tools/experiments/statistics/, and the shared module glmdata.py there holds the data and a reference fitter.
Articles
The model and its fitting
- Exponential Families: the common form of the binomial, Poisson, normal, gamma, inverse Gaussian and negative binomial distributions; the log-partition function proved convex and shown to generate the cumulants of the sufficient statistic; the mean parametrization; maximum likelihood as moment matching, with the case where no estimate exists; the variance function, which determines a natural exponential family; the divergence between two members as a Bregman divergence; and the exponential dispersion form used by the rest of the unit.
- Link Functions and the Structure of a Generalized Linear Model: the three components of the model; why the mean is transformed and not the response; the canonical link and the sufficiency, matched totals and strict concavity it gives; the logit, probit and complementary log-log links from latent thresholds and from Poisson counts; what a coefficient means under each link; interaction as a property of the scale; and a maximum on the boundary for the identity link.
- Iteratively Reweighted Least Squares: the score and the expected and observed information of any generalized linear model; Fisher scoring as a weighted least squares fit of a working response, with inverse-variance weights; Newton's method equal to scoring for canonical links; the local rate of convergence; two standard errors; and step halving, with cases where unmodified steps fail.
Inference and models for counts
- Deviance and Inference for Generalized Linear Models: the deviance from the saturated model and its unit deviances; deviance differences as likelihood ratio statistics; why the deviance of 0/1 data does not measure fit; Pearson's statistic and residuals; Wald, score and likelihood ratio tests in the model, with the Mantel–Haenszel statistic as a score test; and separation, with the condition under which the logistic estimate exists.
- Poisson Regression and Rate Models: counts of events in known person-time, with the log of the exposure as an offset; the standardized mortality ratio as a maximum likelihood estimate; the rate ratio adjusted for age, its relation to the Mantel–Haenszel estimate, and when a fitted level is infinite; worked on U.S. deaths by age.
- Overdispersion and Quasi-Likelihood: counts more variable than the Poisson model allows; quasi-likelihood, which uses only a mean and a variance function; the estimated dispersion and its effect on standard errors; and negative binomial regression; worked on weekly death counts.
Several categories
- Multinomial and Softmax Regression: the categorical distribution as an exponential family whose mean map is the softmax; the softmax from noisy scores; identifiability and what gradient descent and a penalty do with the free direction; the score, the Hessian and a curvature bound; when the estimate exists; and maximum likelihood as conditional maximum entropy; worked on self-rated health and on the output layer of a small language model.
Where the unit leads
Three planned units of the collection build on this one. Survival Analysis needs it for the Cox model and uses the rate models of the Poisson article. Prediction Error and Model Choice asks how many inputs a fitted model should have, a question the deviance answers only for nested models. Bayesian Inference puts prior distributions on the coefficients. The softmax article connects the unit to the output layers described in Activation Functions and Classification Loss Functions.
References
Each article lists its own references. The original paper and two textbooks that treat the unit's subject as a whole: record
- Efron, B., and Hastie, T. (2016). Computer Age Statistical Inference: Algorithms, Evidence, and Data Science. Cambridge University Press. Chapter 5, "Parametric Models and Exponential Families", and Chapter 8, "Generalized Linear Models and Regression Trees". record · 9 pages
- McCullagh, P., and Nelder, J. A. (1989). Generalized Linear Models (2nd ed.). Monographs on Statistics and Applied Probability. Chapman and Hall. record · 2 pages
- Nelder, J. A., and Wedderburn, R. W. M. (1972). Generalized linear models. Journal of the Royal Statistical Society, Series A, 135(3), 370–384. https://doi.org/10.2307/2344614 record · 5 pages