Poisson Regression and Rate Models

Regression for counts of events in a known amount of exposure, with the logarithm of the exposure as an offset. The Poisson likelihood derived from constant hazards in person-time; the standardized mortality ratio proved to be the maximum likelihood estimate of a model with expected deaths as offset; the two-group model of rates by age solved by alternating equations whose first cycle is a ratio of standardized mortality ratios, its estimate proved to be a weighted average of the age-specific rate ratios, its profile likelihood shown to be a conditional binomial likelihood and the Mantel–Haenszel rate ratio shown to be the same equation with fixed weights; and when a fitted level is infinite. Worked on U.S. deaths by age from CDC WONDER (2007 against 2013, trends from 1999 to 2019 and the deaths of 2020) and on four states against the 2021 national rates. Worked problems on a small occupational cohort, an exact interval for a rate ratio, and the model fitted by hand to two age groups.

In 2013 the United States recorded 2,596,861 deaths whose age was stated, against 2,423,511 in 2007. The crude death rate rose from 804.5 to 821.5 per 100,000, and yet the death rate fell in every one of the eleven age groups that the National Center for Health Statistics uses. Direct Standardization resolves the paradox by weighting the age-specific rates with the 2000 standard population, and finds that age-adjusted mortality fell by a factor of 0.944. That is a sound summary, but it is a recipe: it has no parameter that it estimates, no standard error that follows from a model, and no test of whether one number can describe eleven age-specific ratios. Is there a model of the counts in which such a summary is an estimate, and what does fitting it add?

Poisson regression is that model. The count $y_i$ of events in cell $i$ (an age group in a year, a state, a week) is modeled as a Poisson variable whose mean is the exposure $t_i$ (person-years at risk, or expected deaths) times a rate that depends on explanatory variables through a log-linear form:

$$\ln \mu_i = \ln t_i + \mathbf{x}_i^{\mathsf T}\boldsymbol\beta .$$

The term $\ln t_i$ has no coefficient. It is called an offset, and it turns a model for counts into a model for rates, $\mu_i/t_i = \exp(\mathbf{x}_i^{\mathsf T}\boldsymbol\beta)$. Each $e^{\beta_j}$ is a rate ratio.

First we set out the model in the notation of the unit and prove what its fitted values must satisfy. Then we derive the Poisson likelihood from person-time with a constant hazard, which is why the model suits death rates even though the deaths of a population are not literally Poisson. Next we redo the standardization of indirect standardization and direct standardization as models: the standardized mortality ratio is a maximum likelihood estimate, and the model of rates by age and year gives a rate ratio that is a weighted average of the age-specific ratios, with weights the model chooses. Then we ask, with the deviance, whether one ratio describes the data, fit trends over twenty-one years and compare 2020 with them. The article assumes the structure of a generalized linear model from Link Functions and the Structure of a Generalized Linear Model, its fitting from Iteratively Reweighted Least Squares, and the tests and the deviance of Deviance and Inference for Generalized Linear Models.

The model

The data are counts $y_1, \ldots, y_n$ in $n$ cells, with exposures $t_i > 0$ and explanatory variables in the rows $\mathbf{x}_i^{\mathsf T}$ of an $n \times p$ matrix $X$ of full column rank. The counts are independent, and $y_i$ is Poisson with mean $\mu_i$. In the dispersion form of the unit (Exponential Families) the Poisson distribution has natural parameter $\theta = \ln\mu$, cumulant function $b(\theta) = e^\theta$, dispersion $\phi = 1$ and variance function $V(\mu) = \mu$. The log link is the canonical link, and the linear predictor carries the offset $o_i = \ln t_i$:

$$\eta_i = o_i + \mathbf{x}_i^{\mathsf T}\boldsymbol\beta, \qquad \mu_i = t_i\, e^{\mathbf{x}_i^{\mathsf T}\boldsymbol\beta}.$$

The log-likelihood is

$$\ell(\boldsymbol\beta) = \sum_{i=1}^{n} \big[y_i\ln\mu_i - \mu_i - \ln(y_i!)\big] = \sum_{i=1}^{n}\big[y_i(o_i + \mathbf{x}_i^{\mathsf T}\boldsymbol\beta) - t_i e^{\mathbf{x}_i^{\mathsf T}\boldsymbol\beta}\big] + \text{const}.$$

Its gradient is $\mathbf{U}(\boldsymbol\beta) = X^{\mathsf T}(\mathbf{y} - \boldsymbol\mu)$, the score of any canonical-link model with $\phi = 1$ (Link Functions derives it), and the information matrix is $X^{\mathsf T}\operatorname{diag}(\mu_i)X$. The offset changes neither formula; it only enters $\mu_i$.

Reading the coefficients. If two cells differ only in the $j$th explanatory variable, by one unit, their rates differ by the factor $e^{\beta_j}$, whatever their exposures. For an indicator variable, $e^{\beta_j}$ is the rate ratio between the two levels. This is the multiplicative model for rates of Breslow and Day (1987, §4.1), the model their chapter on fitting models to grouped cohort data is built on, and they introduce the offset in exactly this role: the log person-years "offset the model equation" from the origin (§4.3, p. 137, equation 4.11).

Proposition 1 (fitted totals). Let column $j$ of $X$ be the indicator of a set $S$ of cells (a level of a factor, or all cells when the column is the intercept). At the maximum likelihood estimate the fitted total in $S$ equals the observed total: $\sum_{i \in S}\hat\mu_i = \sum_{i \in S} y_i$.

Proof. The $j$th component of the likelihood equation $X^{\mathsf T}(\mathbf{y} - \hat{\boldsymbol\mu}) = \mathbf{0}$ is $\sum_i x_{ij}(y_i - \hat\mu_i) = 0$, and $x_{ij}$ is 1 for $i \in S$ and 0 otherwise. $\square$

So a model with a factor for age reproduces the deaths at every age, and a factor for year reproduces the deaths of every year. Breslow and Day (1987, §3.5) note the same property of internally fitted expected values, that their sum equals the total observed.

The smallest model. With only an intercept, $\ln\mu = \ln t + \beta_0$ for a single cell with $D$ deaths in $N$ person-years, the likelihood equation gives $e^{\hat\beta_0} = D/N$, the crude rate, and the information is $\hat\mu = D$, so the standard error of $\hat\beta_0 = \ln(D/N)$ is $1/\sqrt{D}$. For 2013, with $D = 2{,}596{,}861$ and $N = 316{,}128{,}839$ (the mid-year population, as NCHS uses it), the rate is 821.5 per 100,000 and the standard error of its logarithm is 0.000621. Crude Rates and Their Precision gives the same rate and a relative standard error $1/\sqrt{D}$ from the Poisson model for one count, and the standard error of the log rate is that relative standard error; here both come from the smallest Poisson regression.

Why a Poisson model for deaths in a population

The number of deaths in a population is not a Poisson count in any literal sense. Each person can die once, the population changes during the year, and the mid-year population is an estimate of the person-years lived. Why should a Poisson likelihood be right? Breslow and Day (1987, §4.2) answer with person-time, and the argument is short enough to give in full.

Here is the idea in one sentence: if every person's risk of dying is constant within a cell of age and time, the likelihood of the observed survival and deaths has exactly the form of a Poisson likelihood for the number of deaths, with the person-time in the cell as the exposure.

Theorem 1 (person-time and the Poisson likelihood). Let the time axis be divided into cells $c = 1, \ldots, C$ (intervals of age, of calendar time, or both), and suppose that a person in cell $c$ dies at the constant rate $\lambda_c$: the probability of death in a short interval $dt$ is $\lambda_c\,dt$, given survival to its start. Person $m$ is followed from entry until death or the end of follow-up, and the times of entry and of the end of follow-up are fixed or determined independently of the times of death, with a distribution that does not involve the rates (independent, non-informative censoring). Let $T_c$ be the total time that all persons spend in cell $c$ and $D_c$ the number of deaths that occur in it. Then the likelihood of the observed follow-up is

$$L(\lambda_1, \ldots, \lambda_C) = \prod_{c=1}^{C}\lambda_c^{D_c}\, e^{-\lambda_c T_c},$$

which, as a function of the rates, is proportional to the likelihood of independent Poisson counts $D_c$ with means $\lambda_c T_c$.

Proof. Take one person, followed for time $\tau$, who spends time $\tau_c$ in cell $c$ (so $\sum_c\tau_c = \tau$). The probability of surviving a stretch of length $s$ at constant rate $\lambda$ is $e^{-\lambda s}$: the survival function $G$ satisfies $G(s + ds) = G(s)(1 - \lambda\,ds)$, so $G' = -\lambda G$ with $G(0) = 1$. Survival over consecutive stretches multiplies, since each factor is a probability conditional on surviving the earlier ones. So the probability of surviving the whole follow-up is $\exp(-\sum_c \lambda_c\tau_c)$. If the person dies at the end of follow-up, in cell $c^*$, the contribution is the density of death at that moment, survival times the rate: $\lambda_{c^*}\exp(-\sum_c\lambda_c\tau_c)$. Write $\delta_{c}$ for 1 if the person dies in cell $c$ and 0 otherwise; then both cases are $\prod_c \lambda_c^{\delta_c}e^{-\lambda_c\tau_c}$. Survival is counted from entry, so a person who enters late contributes only the time after entry. By the censoring condition, the times of entry and of the end of follow-up have a probability (or are fixed) that does not involve the rates, and that factor drops out of the likelihood as a constant. Persons are independent, so the likelihood is the product over persons, and collecting the exponents of each $\lambda_c$ gives $\prod_c\lambda_c^{D_c}e^{-\lambda_cT_c}$. A Poisson count $D_c$ with mean $\lambda_cT_c$ has probability $e^{-\lambda_cT_c}(\lambda_cT_c)^{D_c}/D_c!$, which differs from the factor $\lambda_c^{D_c}e^{-\lambda_cT_c}$ only by $T_c^{D_c}/D_c!$, a quantity that does not involve the rates. $\square$

Three consequences follow. Maximum likelihood estimates, likelihood ratio statistics and the observed information computed from a Poisson regression on $(D_c, T_c)$ are those of the constant-hazard model, because the two likelihoods are proportional. The person-time $T_c$ enters as the exposure, so $\ln T_c$ is the offset. And a model for the rates, $\ln\lambda_c = \mathbf{x}_c^{\mathsf T}\boldsymbol\beta$, is a Poisson regression on the cells.

What the theorem does not give is the Poisson distribution of $D_c$: $T_c$ is itself random, because the dead stop contributing time. The large-sample theory of the estimates comes from the likelihood (Asymptotics of the Maximum Likelihood Estimator and its several-parameter statement), not from the counts being Poisson. Two approximations remain in the national data used below. The mid-year population stands in for the person-years lived in the year, as in the rates that NCHS publishes. And the rate within an age group of ten years is not constant, so the model describes the average rate of the group. The same piecewise-constant hazard reappears as a model for survival times in the Survival Analysis unit.

Standardization against an external standard

Indirect Standardization and the Standardized Mortality Ratio compares a population's observed deaths $O = \sum_j d_j$ with the expected deaths $E = \sum_j n_jR_j$ that standard rates $R_j$ would produce in its age groups $j$, and calls $O/E$ the standardized mortality ratio (SMR). As a model, the population's rates are a multiple $\theta$ of the standard rates at every age: $d_j$ is Poisson with mean $\theta e_j$, where $e_j = n_jR_j$. In regression form,

$$\ln\mu_j = \ln e_j + \beta_0, \qquad \theta = e^{\beta_0},$$

a Poisson regression with only an intercept and the logarithm of the expected deaths as the offset. Breslow and Day (1987, §4.6) introduce the model in this form, with the "log expected standard deaths" declared as the offset in place of the log person-years.

Theorem 2 (the SMR as a maximum likelihood estimate). In this model the maximum likelihood estimate of $\theta$ is $O/E$, the standard error of $\ln(O/E)$ from the observed information is $1/\sqrt{O}$, and the three statistics for $H_0\colon \theta = 1$ are

$$W_{\mathrm{S}} = \frac{(O - E)^2}{E}, \qquad W_{\mathrm{LR}} = 2\Big[O\ln\frac{O}{E} - (O - E)\Big], \qquad W_{\mathrm{W}} = O\,\Big(\ln\frac{O}{E}\Big)^2 \ \text{(Wald, on the log scale)}.$$

Proof. The log-likelihood in $\beta_0$ is $\sum_j[d_j(\ln e_j + \beta_0) - e_je^{\beta_0}] + \text{const} = O\beta_0 - Ee^{\beta_0} + \text{const}$. Its derivative $O - Ee^{\beta_0}$ vanishes at $e^{\hat\beta_0} = O/E$, and its second derivative $-Ee^{\beta_0}$ is negative, so this is the maximum. The observed information at the estimate is $Ee^{\hat\beta_0} = O$, which gives the standard error. For the score statistic, at $\beta_0 = 0$ the score is $O - E$ and the information is $E$, so $W_{\mathrm{S}} = (O - E)^2/E$. For the likelihood ratio, $2[\ell(\hat\beta_0) - \ell(0)] = 2[O\ln(O/E) - O + E]$. The Wald statistic on the log scale is $(\hat\beta_0 - 0)^2$ times the observed information, $O(\ln(O/E))^2$. $\square$

The likelihood depends on the age-specific data only through the totals $O$ and $E$. That is why indirect standardization needs only a population's total deaths and its age structure, and the model makes the reason explicit. Breslow and Day (1987, §4.6) state the general version: when the explanatory variables depend only on the exposure group, the likelihood of this model is a function of each group's observed and expected totals, and with a separate parameter for each group the SMRs $O_k/E_k$ are the maximum likelihood estimates.

The four states of the indirect standardization article, against the U.S. age-specific rates of 2021 (deaths from Murphy et al., 2024, Tables 2 and 17; state populations by age from U.S. Census Bureau, 2022), fitted as one Poisson regression with a separate intercept for each state and $\ln E$ as the offset:

State Observed Expected SMR $W_{\mathrm{S}}$ $W_{\mathrm{LR}}$ $W_{\mathrm{W}}$
Maine 17,269 17,261.9 1.0004 0.003 0.003 0.003
Alaska 6,208 5,854.3 1.0604 21.37 20.95 21.36
Florida 261,369 279,888.5 0.9338 1,225.38 1,253.34 1,224.90
Utah 22,569 24,973.7 0.9037 231.54 239.36 231.35

The fitted SMRs are the ones of the earlier article, and the standard error $1/\sqrt{O}$ of their logarithms is the one Breslow and Day (1987, equation 2.9) give and that article used; they recall it in §3.4: the Poisson variance of the log SMR "is estimated approximately by the reciprocal of the number of deaths". With thousands of deaths the three statistics nearly agree, as Likelihood Ratio, Wald and Score Tests predicts when the log-likelihood is close to a parabola; with few deaths they part, as the first worked problem shows.

The model also tests whether one multiple of the national rates serves two states. Fitting a common $\theta$ to Maine and Alaska gives 1.0156, and the likelihood ratio statistic against separate values is 15.35 on 1 degree of freedom ($p = 0.0001$). The hypothesis tested is that the two states' rates are the same multiple of the national rates at every age. That is not the hypothesis that the two states have equal age-specific rates, and the earlier article explains why comparisons of SMRs with each other need that caution. Between a separate rate for each state and one rate for all lies partial pooling, in which the states' rates are drawn from a common distribution; Hierarchical Models and Shrinkage fits that model to state cancer death rates from short windows of weekly counts.

Two populations and no external standard

Back to 2007 and 2013. There is no outside standard to compare with: the question is how the rates of one year compare with those of the other. Index the eleven age groups by $j$ and the two years by $k = 1$ (2007) and $k = 2$ (2013), with deaths $d_{jk}$ and populations $n_{jk}$:

Age Rate 2007 Rate 2013 Ratio
under 1 702.5 594.7 0.847
1–4 29.4 25.5 0.868
5–14 15.2 13.0 0.855
15–24 78.8 64.8 0.823
25–34 107.2 106.1 0.990
35–44 186.0 172.0 0.925
45–54 420.3 406.1 0.966
55–64 866.7 860.0 0.992
65–74 1,976.0 1,802.1 0.912
75–84 4,987.1 4,648.1 0.932
85 and over 14,160.9 13,660.4 0.965

Rates are per 100,000, from CDC WONDER. The multiplicative model gives each age group its own rate and the later year a common factor $\psi$:

$$\ln\mu_{jk} = \ln n_{jk} + \alpha_j + \beta\,[k = 2], \qquad \psi = e^\beta .$$

There are 12 parameters for 22 cells. Write $\theta_j = e^{\alpha_j}$ for the 2007 rate at age $j$, $D_j = d_{j1} + d_{j2}$ for the deaths at age $j$ in both years, and $O_2 = \sum_j d_{j2}$ for the deaths of 2013.

Theorem 3 (the likelihood equations). The maximum likelihood estimates satisfy

$$\hat\theta_j = \frac{D_j}{n_{j1} + \hat\psi\, n_{j2}} \quad (j = 1, \ldots, 11), \qquad \hat\psi = \frac{O_2}{\sum_j n_{j2}\hat\theta_j} .$$

Proof. By Proposition 1 applied to the indicator of age group $j$, $\hat\mu_{j1} + \hat\mu_{j2} = D_j$, that is $n_{j1}\hat\theta_j + n_{j2}\hat\theta_j\hat\psi = D_j$. Applied to the indicator of 2013, $\sum_j n_{j2}\hat\theta_j\hat\psi = O_2$. Solving each for the parameter on its left gives the two equations. $\square$

The second equation says that $\hat\psi$ is an SMR: the deaths of 2013 divided by the deaths expected at the rates $\hat\theta_j$. Breslow and Day (1987, §4.1(d), p. 128) solve the equations by alternating between them, with a factor $\psi_k$ for each year: start with every $\psi_k = 1$; compute $\theta_j = D_j/\sum_k n_{jk}\psi_k$; then $\psi_k = O_k/\sum_j n_{jk}\theta_j$; repeat. At the first cycle $\theta_j = D_j/(n_{j1} + n_{j2})$, the rate of the two years pooled, so the first $\psi_k$ are the SMRs of each year with the pooled rates as the standard, and their ratio is the first estimate of the rate ratio. At convergence, $\hat\psi$ "may be interpreted as an SMR using the estimated rates $\hat\theta_j$ as standard" (their p. 128). Mantel and Stark (1968) had computed indirectly adjusted rates in the same way, with the standard rates derived internally by "a convergent iterative procedure" (their summary).

For 2007 and 2013 the first cycle gives SMRs of 1.0293 for 2007 and 0.9742 for 2013, a ratio of 0.946461. The second cycle gives 0.946345, and further cycles change nothing at six decimals; scoring (IRLS) converges to the same value. The estimate is $\hat\psi = 0.9463$ with standard error 0.00089 for $\hat\beta$, and the 95% Wald interval for the rate ratio is (0.9447, 0.9480).

What kind of average is $\hat\psi$? Each age group has its own ratio $r_j = (d_{j2}/n_{j2})/(d_{j1}/n_{j1})$. Here is the idea of the next theorem: substituting the first equation into the second shows that $\hat\psi$ is an average of these ratios, with weights that are larger where there are more deaths.

Theorem 4 (the rate ratio as an average of the age-specific ratios). The estimate $\hat\psi$ is the root of

$$\sum_j \frac{d_{j2}n_{j1} - \psi\, d_{j1}n_{j2}}{n_{j1} + \psi\, n_{j2}} = 0,$$

and if every $d_{j1} > 0$ it is the weighted average $\hat\psi = \sum_j w_jr_j\big/\sum_j w_j$ of the age-specific ratios, with the positive weights $w_j = d_{j1}n_{j2}/(n_{j1} + \hat\psi n_{j2})$. When both years have deaths and every age group has deaths ($D_j > 0$), the root exists and is unique, and it is the maximum likelihood estimate. In particular $\hat\psi$ lies between the smallest and the largest $r_j$.

Proof. The second equation of Theorem 3 is $\sum_j(d_{j2} - \psi n_{j2}\theta_j) = 0$. Substitute $\theta_j = D_j/(n_{j1} + \psi n_{j2})$:

$$d_{j2} - \frac{\psi n_{j2}(d_{j1} + d_{j2})}{n_{j1} + \psi n_{j2}} = \frac{d_{j2}n_{j1} + \psi d_{j2}n_{j2} - \psi n_{j2}d_{j1} - \psi n_{j2}d_{j2}}{n_{j1} + \psi n_{j2}} = \frac{d_{j2}n_{j1} - \psi d_{j1}n_{j2}}{n_{j1} + \psi n_{j2}},$$

which is the equation stated. Since $r_j\,d_{j1}n_{j2} = d_{j2}n_{j1}$, each numerator is $d_{j1}n_{j2}(r_j - \psi)$, and the equation reads $\sum_j w_j(\psi)(r_j - \psi) = 0$ with $w_j(\psi) = d_{j1}n_{j2}/(n_{j1} + \psi n_{j2})$. Solving for the $\psi$ outside the weights gives the weighted average. For uniqueness, write the left side as $F(\psi) = \sum_j g_j(\psi)$ with $g_j(\psi) = (d_{j2}n_{j1} - \psi d_{j1}n_{j2})/(n_{j1} + \psi n_{j2})$. Each $g_j$ is a ratio of a decreasing linear function to an increasing positive one, and its derivative is $-n_{j1}n_{j2}D_j/(n_{j1} + \psi n_{j2})^2 < 0$, so $F$ is strictly decreasing on $\psi > 0$ and has at most one root. (An age group with $D_j = 0$ would contribute $g_j \equiv 0$ and drop out of the equation, but its $\hat\theta_j$ would be 0, the boundary case of the section on zero rates; the national data have no such group.) It is continuous, with $F(\psi) \to \sum_j d_{j2} = O_2 > 0$ as $\psi \to 0^+$ and $F(\psi) \to -\sum_j d_{j1} = -O_1 < 0$ as $\psi \to \infty$, so a root exists. The Poisson log-likelihood is concave in $(\boldsymbol\alpha, \beta)$, so the point where the likelihood equations hold is its maximum. A weighted average with positive weights lies between the smallest and largest of the values averaged. $\square$

At $\hat\psi$ the weights of the eleven age groups, as shares of their total, are:

Age under 1 1–4 5–14 15–24 25–34 35–44 45–54 55–64 65–74 75–84 85+
Weight 0.011 0.002 0.002 0.013 0.017 0.030 0.072 0.121 0.170 0.257 0.303

The ages 65 and over carry 0.731 of the weight, because that is where the deaths are. Each weight $w_j$ is close to $d_{j1}n_{j2}/(n_{j1} + n_{j2})$ when $\hat\psi$ is near 1, the 2007 deaths of the age group times the share of 2013 in its population of the two years. The rate ratios at those ages, 0.912, 0.932 and 0.965, dominate the average.

The Mantel–Haenszel rate ratio. Put $\psi = 1$ inside the denominators of Theorem 4 and solve the resulting linear equation for the $\psi$ in the numerators:

$$\hat\psi_{\mathrm{MH}} = \frac{\sum_j d_{j2}n_{j1}/N_j}{\sum_j d_{j1}n_{j2}/N_j}, \qquad N_j = n_{j1} + n_{j2}.$$

This is the Mantel–Haenszel estimate for rates, the cohort-study form that Breslow and Day (1987, §3.6(d)) give, and they report that Clayton (1982) showed that it arises at the first stage of one of the iterations that find the maximum likelihood estimate. Theorem 4 shows which iteration: the weighted-average equation with its weights fixed at $\psi = 1$. When $\hat\psi$ is close to 1 the fixed weights are close to the final ones, and here the two estimates agree to four decimals, $\hat\psi_{\mathrm{MH}} = 0.9463$. The third worked problem carries both through by hand for two age groups.

Theorem 5 (the profile likelihood is a conditional likelihood). Given the total deaths $D_j$ of each age group, the deaths of 2013 are independent binomial counts, $d_{j2} \sim \text{Binomial}(D_j, \pi_j)$ with $\pi_j = \psi n_{j2}/(n_{j1} + \psi n_{j2})$, whose distribution depends on $\psi$ alone. The profile log-likelihood of $\psi$, obtained by maximizing over the $\theta_j$, equals the log-likelihood of these binomial counts up to a constant.

Proof. Two independent Poisson counts $X_1, X_2$ with means $\mu_1, \mu_2$ have, for $0 \le x \le D$,

$$P(X_2 = x \mid X_1 + X_2 = D) = \frac{e^{-\mu_1}\mu_1^{D - x}/(D - x)!\cdot e^{-\mu_2}\mu_2^{x}/x!}{e^{-(\mu_1 + \mu_2)}(\mu_1 + \mu_2)^{D}/D!} = \binom{D}{x}\Big(\frac{\mu_2}{\mu_1 + \mu_2}\Big)^{x}\Big(\frac{\mu_1}{\mu_1 + \mu_2}\Big)^{D - x},$$

using that the sum is Poisson with mean $\mu_1 + \mu_2$ (The Poisson Distribution). With $\mu_1 = n_{j1}\theta_j$ and $\mu_2 = \psi n_{j2}\theta_j$ the probability of the second count is $\pi_j$, and $\theta_j$ cancels. For the profile, the log-likelihood of age group $j$ is $d_{j1}\ln(n_{j1}\theta_j) + d_{j2}\ln(\psi n_{j2}\theta_j) - \theta_j(n_{j1} + \psi n_{j2})$ plus a constant. For fixed $\psi$ it is maximized at $\theta_j = D_j/(n_{j1} + \psi n_{j2})$, where the last term equals $-D_j$. Substituting,

$$d_{j1}\ln\frac{n_{j1}D_j}{n_{j1} + \psi n_{j2}} + d_{j2}\ln\frac{\psi n_{j2}D_j}{n_{j1} + \psi n_{j2}} - D_j = d_{j2}\ln\pi_j + d_{j1}\ln(1 - \pi_j) + D_j\ln D_j - D_j,$$

which is the binomial log-likelihood plus terms free of $\psi$. Summing over $j$ proves the claim. $\square$

So the conditional analysis that Breslow and Day (1987, §3.6) build on the binomial distribution of the exposed deaths and the unconditional Poisson regression give the same estimate and the same likelihood ratio statistics for $\psi$. Breslow and Day state the result, "Under the Poisson model, conditional and unconditional maximum likelihood estimators are identical" (§3.6(c), p. 109, citing Haberman, 1974), and note that the difficulties of the odds ratio in many small tables do not arise here. The contrast with the odds ratio of a series of $2 \times 2$ tables is instructive: there the unconditional estimate is not consistent when the strata are many and small, as The Mantel–Haenszel Test proves for matched pairs. For rates the nuisance parameters $\theta_j$ are eliminated exactly by maximization, so the difficulty does not arise.

The model beside the standardized ratios. The same pair of years, summarized six ways:

Summary Value
Crude rates, ratio 1.0210
Ratio of the pooled-standard SMRs (first cycle) 0.9465
Maximum likelihood rate ratio $\hat\psi$ 0.9463
Mantel–Haenszel rate ratio 0.9463
SMR of 2013 with the 2007 rates as standard 0.9466
Ratio of directly standardized rates, 2000 standard 0.9441

All but the crude ratio are averages of the same eleven age-specific ratios with different weights. The SMR of 2013 with 2007's rates as the standard is $\sum_j d_{j2}\big/\sum_j n_{j2}(d_{j1}/n_{j1})$, which is also the ratio of directly standardized rates when 2013's own population is used as the weights: the identity is $\sum_j n_{j2}(d_{j2}/n_{j2}) = \sum_j d_{j2}$. Breslow and Day (1987, §4.1(c), pp. 125–126) show that under the multiplicative model the ratio of directly standardized rates estimates the common ratio, while the ratio of two SMRs need not unless further conditions hold. The ratio with the 2000 standard, 0.9441, gives more weight to the young ages, whose ratios (0.823 to 0.868 below age 25) are the lowest. The model's weights follow the deaths and lean toward the old.

Is one ratio enough?

The model assumes that the eleven age-specific ratios share one value $\psi$. The table shows that they do not look alike: from 0.823 at 15 to 24 to 0.992 at 55 to 64. With five million deaths, chance cannot account for differences of that size. The test is the deviance of the model (Deviance and Inference for Generalized Linear Models defines it and proves the properties used here). Adding an age-by-year interaction gives each age group its own ratio and fits all 22 cells exactly. So the likelihood ratio statistic for a common ratio is the deviance of the model, 1,466.1 on 10 degrees of freedom. The Pearson statistic is 1,465.0. Either is far beyond any chi-squared quantile with 10 degrees of freedom (the deviance per degree of freedom is 146.6).

Example (what the fitted ratio does not claim). "The model estimates the rate ratio as 0.9463 with a standard error of 0.00089, so mortality fell by 5.4% at every age, give or take 0.2%." No. The deviance has rejected a common ratio. The standard error measures only Poisson variation around the model, and describes $\hat\psi$ as an estimate of a parameter that does not exist in the data. What remains true is Theorem 4: $\hat\psi$ is a weighted average of the eleven ratios, with weights set by the deaths. Read that way it is a summary of the same kind as the ratio of directly standardized rates, with a different and data-chosen standard, and it should be reported beside the age-specific ratios. Breslow and Day (1987, §4.3(c), pp. 137–138) make the same point about goodness of fit in general: a deviance larger than its degrees of freedom means either "systematic effects that have not been accounted for by the model" or variation greater than Poisson theory specifies.

A trend over twenty-one years

A model can carry more structure than a table. For 1999 to 2019 there are 21 years and 231 cells. Let $\tau = \text{year} - 2009$, so that $e^{\beta}$ for $\tau$ is the ratio of the rates of consecutive years. Five models:

Model Parameters Deviance df $\hat\phi = X^2/\text{df}$
Age only 11 311,893 220 1,445.0
Age + linear trend 12 93,163 219 433.1
Age + a factor for each year 31 71,873 200 363.0
Age × linear trend 22 36,378 209 174.0
Age × (linear + quadratic) trend 33 11,774 198 59.4

With a common trend the rates fall by the factor 0.98947 a year, 1.05 per cent a year, with Poisson standard error 0.000023 for $\hat\beta$. Every model in the table is rejected by its deviance. Giving each year its own factor, the most general multiplicative model, removes less deviance than giving each age group its own linear trend, because the trends differ by age. With separate trends the rates change, in per cent per year:

Age under 1 1–4 5–14 15–24 25–34 35–44 45–54 55–64 65–74 75–84 85+
% per year −1.51 −1.88 −1.92 −0.97 1.18 −0.43 −0.44 −0.50 −1.68 −1.43 −0.87

Only the 25 to 34 age group has rising rates over the period. Adding a quadratic term in each age group lowers the deviance by another two thirds, so the trends also bent. The remaining deviance is still 59 times its degrees of freedom.

What should be done with a deviance that large? In the older age groups the counts are in the hundreds of thousands, so the Poisson standard deviation of a cell is well under 1% of its mean, and a departure of one per cent from a smooth trend is several standard deviations. Something moves the rates of some ages in some years by more than Poisson chance would, and the models do not say what. The best of the five models has the mean structure about as nearly right as a smooth trend can, but the Poisson variance understates the variation around it. One remedy keeps the estimates and inflates the standard errors by $\sqrt{\hat\phi}$, which is meant for a model whose mean is right. For the model with quadratic trends by age, $\hat\phi = 59.4$, and every standard error grows by the factor $\sqrt{59.4} = 7.7$. Applied to the common-trend model, the same recipe would inflate by 20.8, but that $\hat\phi$ of 433.1 mostly measures the misfit of one trend for all ages, which the larger models remove; it is not overdispersion around a correct mean. Overdispersion and Quasi-Likelihood proves when the remedy is valid and compares it with the alternatives.

The deaths of 2020. In 2020 the United States recorded 3,383,613 deaths of known age, against 2,854,691 in 2019. How many would the trends of 1999 to 2019 have predicted? Apply each fitted model to the 2020 population by age:

Trend model, fitted to 1999–2019 Fitted deaths, 2019 Expected deaths, 2020 Observed / expected Observed − expected
Age × linear trend 2,786,826 2,800,862 1.208 582,751
Age × (linear + quadratic) trend 2,877,234 2,924,797 1.157 458,816

The observed-to-expected ratio is an SMR with a projected standard, and its size depends on the projection. The linear trends already fall short of the deaths of 2019, because in these data the decline in rates slowed: the quadratic coefficient is positive in 9 of the 11 age groups (negative only under 1 and at 45 to 54), so in most age groups the log rates curve upward over the period. The quadratic trends follow the slowdown and expect more deaths in 2020. The two answers differ by more than 120,000 deaths. Published estimates of excess deaths use their own models and data, and neither row here reproduces them. What the table shows is the structure of every such estimate: an observed count, an expected count from a model fitted to earlier years, and a dependence on that model which the Poisson standard error does not express.

When a fitted rate is zero

One failure is specific to counts. Suppose some level of a factor has no events at all: an age group with no deaths in a small cohort, say. Proposition 1 requires the fitted total for that level to equal its observed total, 0, but every fitted mean $t_ie^{\mathbf{x}_i^{\mathsf T}\boldsymbol\beta}$ is positive. So no finite $\boldsymbol\beta$ solves the likelihood equation. The log-likelihood is bounded above (each term $y\ln\mu - \mu$ is at most $y\ln y - y$), and it increases toward its supremum, which is approached only in the limit in which that level's coefficient tends to $-\infty$. (If the empty level is the reference level, it is the intercept that tends to $-\infty$ and the other levels' coefficients to $+\infty$.) A maximum over all of $\mathbb{R}^p$ would be a stationary point of the differentiable log-likelihood, and there is none. Fitting software reports a large negative coefficient with an enormous standard error. This is the Poisson version of the separation problem of binary regression, which Deviance and Inference for Generalized Linear Models treats in full: the maximum is approached but not attained, the Wald statistic for the coefficient is useless, and the likelihood ratio statistic, computed at the limit, remains usable. The national data have no such cells; small cohorts often do.

What is proved here, what is cited, and what is computed

Proved on this page: the fitted totals of every level of a factor (Proposition 1); the Poisson form of the constant-hazard likelihood in person-time (Theorem 1); the SMR as the maximum likelihood estimate, with its standard error and the three test statistics (Theorem 2); the likelihood equations of the two-group multiplicative model (Theorem 3); the estimate as a weighted average of the age-specific ratios, its existence and uniqueness, and the Mantel–Haenszel estimate as the same equation with weights fixed at $\psi = 1$ (Theorem 4); the conditional binomial distribution and the identity of the profile and conditional likelihoods (Theorem 5); the identity of the SMR with an internal standard and the directly standardized ratio with the study population as weights; the non-existence of a finite estimate when a level has no events.

Taken from other pages: the score and information of canonical-link models (Link Functions, IRLS); the deviance and the chi-squared calibration of nested deviance differences (Deviance and Inference, which cites Wilks's theorem for several parameters); the sum of independent Poisson counts (The Poisson Distribution).

Cited: the multiplicative model, the offset, the alternating algorithm and its interpretation, internal and external standards in the model (Breslow and Day, 1987, §§3.5, 3.6, 4.1–4.3, 4.6); the result that the Mantel–Haenszel estimate arises at the first stage of an iteration (Clayton, 1982, as Breslow and Day report it); internally derived standard rates by iteration (Mantel and Stark, 1968).

Computed: every rate, estimate, statistic and table on the page, by tools/experiments/statistics/poisson-regression.py, from the CDC WONDER deaths and populations by age for 1999 to 2020 and the 2021 state figures of the indirect standardization article. There is no simulation in this article.

Worked problems

These problems are posed for this article: the first on the small cohort of the indirect standardization article, the second on an example of Breslow and Day (1987), the third on the national data. Every number is recomputed by the article's script.

Problem 1 (a small cohort). A cohort of workers records 12 deaths from a cause where the standard rates predict 7.5. Using Theorem 2, find the maximum likelihood estimate of the SMR and the score, likelihood ratio and Wald statistics for $\theta = 1$, with their chi-squared $p$-values. Find the 95% likelihood ratio interval and compare it with the exact interval (0.827, 2.795).

Solution. The estimate is $12/7.5 = 1.60$. The score statistic is $(12 - 7.5)^2/7.5 = 2.700$, with $p = 0.1003$ on 1 degree of freedom. The likelihood ratio statistic is $2[12\ln(1.6) - 4.5] = 2.280$, with $p = 0.1310$. On the log scale $\ln 1.6 = 0.4700$, and the Wald statistic is $12 \times 0.4700^2 = 2.651$, with $p = 0.1035$. None rejects at the 5% level, and the exact two-sided $p$-value of the earlier article, 0.158, agrees in direction. The likelihood ratio interval is the set of $\theta$ with $2[12\ln(12/(7.5\theta)) - (12 - 7.5\theta)] \le 3.8415$. Solving the two boundary equations numerically gives (0.857, 2.683). It is shorter than the exact interval at both ends. The exact interval is conservative by construction (its coverage is at least 95% for every $\theta$), while the likelihood ratio interval aims at 95% only approximately.

Problem 2 (an exact interval for a rate ratio). Breslow and Day (1987, Example 3.10) observe 5 lung cancer deaths in 7,300 person-years among unexposed persons and 14 in 5,500 among the exposed. Estimate the rate ratio, and find an exact 95% interval from the conditional binomial distribution of Theorem 5.

Solution. The estimate is $(14/5{,}500)/(5/7{,}300) = 3.716$, or 3.72. Given the 19 deaths in all, the exposed deaths are binomial with $n = 19$ and probability $\pi = 5{,}500\psi/(7{,}300 + 5{,}500\psi)$, by Theorem 5 with one stratum. The estimate of $\pi$ is $14/19 = 0.7368$, and the exact (Clopper–Pearson) 95% limits for a binomial proportion with 14 successes in 19 trials are 0.4880 and 0.9085. Solving $\pi = 5{,}500\psi/(7{,}300 + 5{,}500\psi)$ for $\psi$ gives $\psi = 7{,}300\pi/(5{,}500(1 - \pi))$, an increasing function of $\pi$, so the limits transform to (1.26, 13.2). These are the exact limits Breslow and Day report. The Wald interval on the log scale, with standard error $\sqrt{1/5 + 1/14} = 0.5210$ from the inverse of the information of the two-cell Poisson model, is (1.34, 10.32), narrower at both ends with so few deaths.

Problem 3 (the model by hand for two age groups). Use only the age groups 75 to 84 and 85 and over: deaths 652,682 and 713,647 in populations 13,087,439 and 5,039,545 in 2007, and 625,013 and 825,198 in populations 13,446,519 and 6,040,789 in 2013. Run the alternating equations for two cycles, and compare the result with the Mantel–Haenszel estimate and with the two age-specific ratios.

Solution. The pooled rates are $D_j/(n_{j1} + n_{j2})$: 4,815.3 per 100,000 at 75 to 84 and 13,888.1 at 85 and over. Applied to each year's populations they give 1,330,098 expected deaths in 2007 and 1,486,442 in 2013. The observed deaths are 1,366,329 and 1,450,211, so the first-cycle SMRs are $1{,}366{,}329/1{,}330{,}098$ and $1{,}450{,}211/1{,}486{,}442$, and their ratio is 0.94975. The second cycle, with $\theta_j = D_j/(n_{j1}\psi_1 + n_{j2}\psi_2)$ from the first-cycle factors, gives 0.94968, and that is the maximum likelihood estimate to five decimals ($\hat\psi = 0.94968$; scoring agrees). The Mantel–Haenszel estimate is

$$\frac{\sum_j d_{j2}n_{j1}/N_j}{\sum_j d_{j1}n_{j2}/N_j} = 0.94967,$$

which agrees to four decimals. The two age-specific ratios are 0.9320 and 0.9647, and by Theorem 4 the estimate lies between them, as their weighted average with weights in the proportion 0.459 to 0.541. The deviance of the two-group model, on 1 degree of freedom, is 206.0: with over a million deaths in each age group, chance cannot explain the difference between 0.9320 and 0.9647, and the estimate is the summary that Theorem 4 describes rather than a common ratio.

Summary

Quantity Formula
Poisson regression $\ln\mu_i = \ln t_i + \mathbf{x}_i^{\mathsf T}\boldsymbol\beta$; rate ratio $e^{\beta_j}$
Likelihood equation $X^{\mathsf T}(\mathbf{y} - \hat{\boldsymbol\mu}) = \mathbf{0}$; fitted totals = observed totals for every level
Person-time constant hazard: $L = \prod_c\lambda_c^{D_c}e^{-\lambda_cT_c}$, Poisson with offset $\ln T_c$
SMR MLE of $\theta$ with offset $\ln e_j$; $\operatorname{SE}(\ln\text{SMR}) = 1/\sqrt{O}$; score $(O - E)^2/E$
Two groups, by age $\hat\theta_j = D_j/(n_{j1} + \hat\psi n_{j2})$, $\hat\psi = O_2/\sum_j n_{j2}\hat\theta_j$
Rate ratio as an average $\hat\psi = \sum_j w_jr_j/\sum_j w_j$, $w_j = d_{j1}n_{j2}/(n_{j1} + \hat\psi n_{j2})$
Mantel–Haenszel the same equation with $\psi = 1$ in the weights
Conditional likelihood $d_{j2} \mid D_j \sim \text{Binomial}(D_j, \psi n_{j2}/(n_{j1} + \psi n_{j2}))$ = profile likelihood

What we have learned

Poisson regression with an offset is the model under rates and standardization. The SMR is its maximum likelihood estimate against an external standard, and the multiplicative model of two populations by age gives a rate ratio that is an SMR with an internally fitted standard and a weighted average of the age-specific ratios, close to the Mantel–Haenszel estimate. The model adds what the recipes lack: standard errors, likelihood ratio tests, and a deviance that says whether one ratio can describe the data. For U.S. deaths it says no, and every larger model is rejected too, because counts in the millions vary more around any smooth model than the Poisson distribution allows.

That variation is the subject of Overdispersion and Quasi-Likelihood, which returns to the weekly deaths of Iowa, proves what can be inferred when only the mean and the variance function are trusted, and fits the negative binomial regression.

References

  • Breslow, N. E., and Day, N. E. (1987). Statistical Methods in Cancer Research, Volume II: The Design and Analysis of Cohort Studies. IARC Scientific Publications No. 82. International Agency for Research on Cancer. Chapter 2 (§2.3, equation 2.9), Chapter 3 (§§3.4–3.6, Example 3.10) and Chapter 4 (§§4.1–4.3, 4.6). record · 7 pages
  • Centers for Disease Control and Prevention, National Center for Health Statistics. Underlying Cause of Death 1999–2020, CDC WONDER online database, released 2021, from the Multiple Cause of Death Files compiled through the Vital Statistics Cooperative Program. https://wonder.cdc.gov/ucd-icd10.html (queried November 2026). record · 5 pages
  • Clayton, D. G. (1982). The analysis of prospective studies of disease aetiology. Communications in Statistics: Theory and Methods, 11, 2129–2155. Cited as Breslow and Day (1987) cite it. record
  • Haberman, S. J. (1974). The Analysis of Frequency Data. University of Chicago Press. Cited as Breslow and Day (1987) cite it. record
  • Mantel, N., and Stark, C. R. (1968). Computation of indirect-adjusted rates in the presence of confounding. Biometrics, 24(4), 997–1005. https://doi.org/10.2307/2528886 record
  • Murphy, S. L., Kochanek, K. D., Xu, J., and Arias, E. (2024). Deaths: Final data for 2021. National Vital Statistics Reports, 73(8). National Center for Health Statistics. Table 2 and Table 17. record · 7 pages
  • U.S. Census Bureau, Population Division (2022). Annual State Resident Population Estimates for 6 Race Groups (5 Race Alone Groups and Two or More Races) by Age, Sex, and Hispanic Origin: April 1, 2020 to July 1, 2021 (Vintage 2021), file sc-est2021-alldata6. https://www2.census.gov/programs-surveys/popest/datasets/2020-2021/state/asrh/ record · 6 pages