Overdispersion and Quasi-Likelihood

What can be inferred from counts that vary more than the Poisson model allows, when only the mean and the form of the variance are trusted. Quasi-likelihood defined from a mean and a variance function; the quasi-score proved unbiased with an information identity, and proved optimal among linear estimating functions, a Gauss–Markov theorem for estimating equations; the dispersion estimate from Pearson's statistic, quasi-Poisson standard errors and F tests, and the sandwich variance; the negative binomial with fixed shape shown to be a generalized linear model with variance mu + mu^2/k, its score equations and the orthogonality of shape and coefficients. Worked on Iowa's weekly deaths of 2014 to 2019 with a seasonal regression, and on all 53 reporting areas, where the dispersion grows with the size of the area as the negative binomial predicts; a simulation of interval coverage under independent and serially correlated counts. Worked problems on quasi-Poisson standard errors, the expected Pearson statistic, and California.

Iowa recorded 570.52 deaths a week from all causes over the 313 weeks of 2014 to 2019. Overdispersion in Count Data found their variance to be 3.850 times their mean and traced most of the excess to a rate that changes: deaths rise every winter and drift upward over the years. Its first remedy was to model the variation in the rate. Do that here, with a Poisson regression on a trend and a seasonal cycle, and the counts still vary 1.566 times as much around the fitted means as a Poisson count would. The coefficients of the model are what we want: how fast deaths rose, how large the winter peak is. Can we estimate them, with honest standard errors, without knowing the distribution of the counts?

The answer is quasi-likelihood. Wedderburn (1974) observed that "to define a quasi-likelihood function we need only specify a relation between the mean and variance of the observations" (his abstract). The estimating equations of a generalized linear model use only the mean and the variance function, so they remain sensible when nothing else about the distribution is known. The variance is allowed a free multiplier, the dispersion $\phi$, estimated from the data.

First we fit the seasonal Poisson regression to Iowa and measure the dispersion that remains. Then we define the quasi-score and prove three properties: it has mean zero, it satisfies the information identity, and among linear estimating functions it is the best. Next we estimate $\phi$, correct the standard errors and tests, and derive the sandwich variance, which needs not even the variance function. Then we fit the negative binomial regression, a full likelihood with a quadratic variance, and use all 53 reporting areas to decide between the linear and quadratic forms. Finally a simulation shows what each method's intervals do when the counts are overdispersed and when they are also correlated in time. The article assumes Deviance and Inference for Generalized Linear Models and Poisson Regression and Rate Models, and the negative binomial distribution and its fit by maximum likelihood from Overdispersion in Count Data and Maximum Likelihood Estimation.

The weekly deaths, modeled

The National Center for Health Statistics publishes weekly counts of deaths by state of occurrence, by the week in which they occurred (National Center for Health Statistics, 2021). For Iowa's 313 weeks, number the weeks $t = 0, \ldots, 312$ and model the expected deaths $\mu_t$ by

$$\ln\mu_t = \beta_0 + \beta_1\frac{t - 156}{P} + \sum_{h=1}^{3}\Big[\gamma_h\cos\frac{2\pi ht}{P} + \delta_h\sin\frac{2\pi ht}{P}\Big], \qquad P = \frac{365.25}{7}.$$

The period $P = 365.25/7 \approx 52.18$ weeks is a year (the computations use $365.25/7$ itself, not the rounded value), so $e^{\beta_1}$ is the ratio of the expected deaths a year apart, and the three pairs of harmonics allow a winter peak of any shape that repeats each year. This is the trend and cycle that Resampling Dependent and Clustered Data removed by least squares; here they act on the log of the mean. The model has $p = 8$ coefficients and no offset, since the population of Iowa changed little in six years and its change is absorbed by the trend.

Weekly deaths in Iowa, 2014 to 2019 (gray), with the fitted mean of the Poisson regression on a trend and three harmonics (orange) and a band of two quasi-Poisson standard deviations, the square root of 1.566 times the mean, on each side. Notice that the fitted mean follows the winter peaks and the upward drift, and that the remaining scatter mostly fits within the widened band, while several winter peaks (early 2015, early 2017) rise above it.
Weekly deaths in Iowa, 2014 to 2019 (gray), with the fitted mean of the Poisson regression on a trend and three harmonics (orange) and a band of two quasi-Poisson standard deviations, the square root of 1.566 times the mean, on each side. Notice that the fitted mean follows the winter peaks and the upward drift, and that the remaining scatter mostly fits within the widened band, while several winter peaks (early 2015, early 2017) rise above it.

The fitted means run from 507.5 in the summer troughs to 646.0 at the winter peaks. The trend coefficient is $\hat\beta_1 = 0.0115$: deaths rose by 1.16 per cent a year after allowing for the season. The amplitude of the first harmonic, $\sqrt{\hat\gamma_1^2 + \hat\delta_1^2} = 0.0856$, says that this harmonic alone moves the expected deaths about 9% above and below their average ($e^{0.0856} = 1.089$).

Does the Poisson variance now fit? The deviance is 475.4 and Pearson's statistic 477.6, on $313 - 8 = 305$ degrees of freedom. Their ratios to the degrees of freedom are 1.559 and 1.566. With means near 570, each weekly count is close to normal under the Poisson model, so Pearson's statistic is close to chi-squared on 305 degrees of freedom if the model is right (the deviance article shows why it is a fair reference for counts this large). The chance of 477.6 or more is about one in a billion. For contrast, the same model fitted to Iowa's weekly cancer deaths has Pearson's statistic 285.6 on 305 degrees of freedom, a ratio of 0.936 and an upper-tail $p$-value of 0.78. Cancer deaths are Poisson around their mean; all-cause deaths are not, even around a mean that follows the season.

So modeling the rate removed most of the excess (from 3.850 to 1.566) but not all. The excess that remains is what the year-to-year differences in the winter peaks, short epidemics and other changes in the weekly rate leave behind. The rest of the article is about inference in its presence.

Quasi-likelihood

Here is the idea in one sentence: the score of a generalized linear model is built from the residuals $y_i - \mu_i$, weighted by the reciprocal of a variance and by the derivative of the mean, and its useful properties follow from the first two moments alone.

The assumptions. Observations $y_1, \ldots, y_n$ are independent, with

$$E[y_i] = \mu_i(\boldsymbol\beta), \qquad \operatorname{Var}(y_i) = \phi\,V(\mu_i)/w_i,$$

where $g(\mu_i) = \mathbf{x}_i^{\mathsf T}\boldsymbol\beta$ for a known link $g$, $V$ is a known variance function, the prior weights $w_i$ are known, and $\phi > 0$ is unknown. Nothing else about the distribution is assumed. Write $D$ for the $n \times p$ matrix of derivatives $\partial\mu_i/\partial\beta_j$ (its $i$th row is $\mathbf{x}_i^{\mathsf T}/g'(\mu_i)$) and $\Sigma = \operatorname{diag}(\phi V(\mu_i)/w_i)$ for the covariance matrix of $\mathbf{y}$.

The quasi-score is

$$\mathbf{U}(\boldsymbol\beta) = D^{\mathsf T}\Sigma^{-1}(\mathbf{y} - \boldsymbol\mu) = \sum_{i=1}^n \frac{w_i(y_i - \mu_i)}{\phi V(\mu_i)}\cdot\frac{\mathbf{x}_i}{g'(\mu_i)}.$$

It is the score $X^{\mathsf T}WG(\mathbf{y} - \boldsymbol\mu)/\phi$ of a generalized linear model in the unit's notation, with $W_i = w_i/[V(\mu_i)g'(\mu_i)^2]$ and $G = \operatorname{diag}(g'(\mu_i))$, as the substitution shows. The quasi-likelihood is a function whose gradient it is:

$$Q(\boldsymbol\mu; \mathbf{y}) = \sum_{i=1}^n w_i\int_{y_i}^{\mu_i}\frac{y_i - t}{\phi V(t)}\,dt,$$

since $\partial Q/\partial\mu_i = w_i(y_i - \mu_i)/(\phi V(\mu_i))$ and the chain rule supplies $\partial\mu_i/\partial\boldsymbol\beta$. For $V(\mu) = \mu$ the integral is $y\ln\mu - \mu$ up to terms free of $\mu$, which is the Poisson log-likelihood divided by $\phi$. Wedderburn (1974) states the general fact: for a one-parameter exponential family the log-likelihood is the same as the quasi-likelihood. The maximum quasi-likelihood estimate $\hat{\boldsymbol\beta}$ solves $\mathbf{U}(\hat{\boldsymbol\beta}) = \mathbf{0}$. The factor $1/\phi$ multiplies the whole equation, so $\hat{\boldsymbol\beta}$ does not depend on $\phi$, and it is computed by the same iteratively reweighted least squares.

Theorem 1 (the quasi-score behaves like a score). Under the assumptions above, at the true $\boldsymbol\beta$,

$$E[\mathbf{U}] = \mathbf{0}, \qquad \operatorname{Cov}(\mathbf{U}) = D^{\mathsf T}\Sigma^{-1}D = -E\Big[\frac{\partial\mathbf{U}}{\partial\boldsymbol\beta^{\mathsf T}}\Big].$$

Proof. $D$ and $\Sigma$ are fixed numbers at the true $\boldsymbol\beta$, so $E[\mathbf{U}] = D^{\mathsf T}\Sigma^{-1}E[\mathbf{y} - \boldsymbol\mu] = \mathbf{0}$, and $\operatorname{Cov}(\mathbf{U}) = D^{\mathsf T}\Sigma^{-1}\operatorname{Cov}(\mathbf{y})\Sigma^{-1}D = D^{\mathsf T}\Sigma^{-1}\Sigma\Sigma^{-1}D = D^{\mathsf T}\Sigma^{-1}D$, using the independence of the observations (so $\operatorname{Cov}(\mathbf{y}) = \Sigma$). For the derivative, write $\mathbf{U} = \sum_i\mathbf{a}_i(\boldsymbol\beta)(y_i - \mu_i)$ with $\mathbf{a}_i = w_i\mathbf{x}_i/(\phi V(\mu_i)g'(\mu_i))$. Then

$$\frac{\partial\mathbf{U}}{\partial\boldsymbol\beta^{\mathsf T}} = \sum_i\frac{\partial\mathbf{a}_i}{\partial\boldsymbol\beta^{\mathsf T}}(y_i - \mu_i) - \sum_i\mathbf{a}_i\frac{\partial\mu_i}{\partial\boldsymbol\beta^{\mathsf T}}.$$

The first sum has expectation zero. In the second, $\mathbf{a}_i = (\partial\mu_i/\partial\boldsymbol\beta)\,w_i/(\phi V(\mu_i))$, so the sum is $D^{\mathsf T}\Sigma^{-1}D$. $\square$

These are the two identities that a true score satisfies (Fisher Information and the Cramér–Rao Bound proves them for a likelihood). They are what the large-sample argument of Asymptotics of the Maximum Likelihood Estimator uses. Expanding $\mathbf{0} = \mathbf{U}(\hat{\boldsymbol\beta})$ around the true value, $\hat{\boldsymbol\beta} - \boldsymbol\beta \approx (D^{\mathsf T}\Sigma^{-1}D)^{-1}\mathbf{U}(\boldsymbol\beta)$, a linear function of a sum of independent terms with mean zero, so

$$\operatorname{Cov}(\hat{\boldsymbol\beta}) \approx (D^{\mathsf T}\Sigma^{-1}D)^{-1} = \phi\,(X^{\mathsf T}WX)^{-1}.$$

McCullagh (1983) gives the theory: consistency and asymptotic normality of the estimates "under second moment assumptions" (his abstract). We cite it rather than prove it; the expansion above is the first-order argument, not a proof.

Which estimating function? The quasi-score weights each residual by $\mathbf{x}_i/(g'(\mu_i)V(\mu_i))$. Any other choice of weights $A$ also gives an unbiased equation $A^{\mathsf T}(\mathbf{y} - \boldsymbol\mu) = \mathbf{0}$: ordinary least squares on the counts takes $A = D$, for instance. The next theorem says the quasi-score's weights give the smallest large-sample covariance. McCullagh (1983) describes the result as "a property of asymptotic optimality similar in spirit to, but more general than, the corresponding optimal property of Gauss-Markov estimators" (his abstract); the Gauss–Markov theorem is the case of a linear mean.

Theorem 2 (optimal weights). Let $A$ be any $n \times p$ matrix not depending on $\mathbf{y}$ with $A^{\mathsf T}D$ invertible, and let

$$K_A = (A^{\mathsf T}D)^{-1}A^{\mathsf T}\Sigma A\,(D^{\mathsf T}A)^{-1}$$

be the covariance of the first-order approximation $(A^{\mathsf T}D)^{-1}A^{\mathsf T}(\mathbf{y} - \boldsymbol\mu)$ to the error of the root of $A^{\mathsf T}(\mathbf{y} - \boldsymbol\mu) = \mathbf{0}$. Then $K_A - (D^{\mathsf T}\Sigma^{-1}D)^{-1}$ is positive semidefinite, with equality when $A = \Sigma^{-1}D$, the quasi-score.

Proof. Write $I = D^{\mathsf T}\Sigma^{-1}D$ (invertible when $D$ has full column rank) and $M = A^{\mathsf T}D$, and let $C = M^{-1}A^{\mathsf T} - I^{-1}D^{\mathsf T}\Sigma^{-1}$, a $p \times n$ matrix. Since $\Sigma$ is positive definite, $C\Sigma C^{\mathsf T}$ is positive semidefinite. Expand it:

$$C\Sigma C^{\mathsf T} = M^{-1}A^{\mathsf T}\Sigma AM^{-\mathsf T} - M^{-1}A^{\mathsf T}DI^{-1} - I^{-1}D^{\mathsf T}AM^{-\mathsf T} + I^{-1}D^{\mathsf T}\Sigma^{-1}DI^{-1}.$$

The first term is $K_A$. In the second, $M^{-1}A^{\mathsf T}D = M^{-1}M$ is the identity, so the term is $I^{-1}$; the third is its transpose, also $I^{-1}$; the fourth is $I^{-1}II^{-1} = I^{-1}$. So $C\Sigma C^{\mathsf T} = K_A - I^{-1}$. With $A = \Sigma^{-1}D$, $M = I$ and $C = 0$. $\square$

The first-order covariance of a root of the equation is the right object to compare under the conditions McCullagh (1983) states. The theorem compares the estimating functions that are linear in the observations. It says nothing against a full likelihood for a known distribution, which may use the data in nonlinear ways.

Example (the quasi-Poisson estimate is the Poisson estimate). With $V(\mu) = \mu$ and the log link, the quasi-score is $X^{\mathsf T}(\mathbf{y} - \boldsymbol\mu)/\phi$, a multiple of the Poisson score. So the quasi-Poisson coefficients for Iowa are exactly the Poisson ones: the trend is still 0.0115. Only the standard errors and tests change, by the factor $\phi$ in the covariance. This is the second remedy of Overdispersion in Count Data, "scale the Poisson variance up by an estimated factor", made precise: the coefficients are justified by Theorem 1 and Theorem 2 under any distribution with variance proportional to the mean.

Estimating the dispersion

Here is the idea: each squared Pearson residual $w_i(y_i - \mu_i)^2/V(\mu_i)$ has expectation $\phi$, so their average estimates it, once the degrees of freedom spent on the fitted mean are allowed for.

Theorem 3 (the Pearson estimate of $\phi$). If the means $\mu_i$ are known, $E\big[\sum_i w_i(y_i - \mu_i)^2/V(\mu_i)\big] = n\phi$. For the normal linear model with $V = 1$, fitted means and full-rank $X$, $E[X^2] = (n - p)\phi$ exactly.

Proof. $E[w_i(y_i - \mu_i)^2/V(\mu_i)] = w_i\operatorname{Var}(y_i)/V(\mu_i) = \phi$; sum over $i$. For the linear model, $X^2$ is the residual sum of squares, whose expectation is $(n - p)\sigma^2$ by the trace argument of The Gauss–Markov Theorem, which needs no normality. $\square$

For other models the fitted means enter $X^2$, and the divisor $n - p$ is the first-order correction for them: at the last iteration of IRLS, $X^2$ is the residual sum of squares of a weighted linear regression of the working response, to which the linear-model result applies. So

$$\hat\phi = \frac{X^2}{n - p}$$

is the standard estimate. It is approximately unbiased, not exactly. The deviance over $n - p$ is the common alternative; for Iowa they are 1.566 and 1.559. In the simulation below, with independent counts whose true dispersion averages 1.52 over the weeks, the median of $\hat\phi$ over 6,000 simulated series was 1.52.

Quasi-Poisson inference for Iowa. The standard errors are the Poisson ones times $\sqrt{\hat\phi} = 1.251$. For a test of $q$ coefficients the deviance difference is divided by $q\hat\phi$ and referred to the $F$ distribution with $q$ and $n - p$ degrees of freedom, the approximate counterpart of the exact $F$ test of The General Linear Hypothesis. Is a cycle with three harmonics needed, or would one do? The model with only the first harmonic has deviance 490.4, so dropping the second and third raises it by 15.0 on 4 degrees of freedom. Referred to $\chi^2_4$, as the Poisson model would have it, the $p$-value is 0.0047. With the dispersion, $F = 15.0/(4 \times 1.566) = 2.40$ on 4 and 305 degrees of freedom, and $p = 0.0502$. The Poisson model overstates the evidence for the finer seasonal shape by a factor of ten in the $p$-value. The point is that ratio, not the side of 0.05 on which the corrected value falls: a $p$-value this close to 0.05 is borderline evidence, and small changes in the model (rounding the period to 52.18 weeks, for instance, gives 0.0499) move it across the line without changing what it says.

The sandwich variance

The quasi-Poisson standard errors still assume a variance function: proportional to the mean, with one $\phi$ for every week. A cruder method assumes only independence. In the first-order approximation $\hat{\boldsymbol\beta} - \boldsymbol\beta \approx \mathcal{I}^{-1}\mathbf{U}$, with $\mathcal{I} = X^{\mathsf T}WX$ the Poisson information and $\mathbf{U} = X^{\mathsf T}(\mathbf{y} - \boldsymbol\mu)$ the Poisson score, the covariance is

$$\operatorname{Cov}(\hat{\boldsymbol\beta}) \approx \mathcal{I}^{-1}\Big[\sum_i\operatorname{Var}(y_i)\,\mathbf{x}_i\mathbf{x}_i^{\mathsf T}\Big]\mathcal{I}^{-1},$$

whatever the variances are, because the score is a sum of independent terms $\mathbf{x}_i(y_i - \mu_i)$. Replacing each unknown $\operatorname{Var}(y_i)$ by the squared residual $(y_i - \hat\mu_i)^2$ gives the sandwich (or robust) estimator, named for the information matrices on either side of the middle term. Each squared residual is a poor estimate of its own variance, but the middle term is a sum of $n$ of them, and averages well. When the variance function is right, the sandwich and the model-based covariance estimate the same thing. Liang and Zeger (1986) built on this idea for longitudinal data, with estimating equations whose estimates and variances are consistent "under mild assumptions about the time dependence" and an approach "closely related to quasi-likelihood" (their abstract).

For Iowa's trend coefficient the four standard errors, and the 95% intervals they give for the annual change in per cent:

Method SE of $\hat\beta_1$ 95% interval, % per year
Poisson 0.00138 (0.89, 1.43)
Quasi-Poisson 0.00173 (0.82, 1.50)
Sandwich 0.00169 (0.83, 1.49)
Negative binomial 0.00171 (0.83, 1.50)

The three methods that allow for overdispersion agree closely, and all are about 25% wider than the Poisson interval. The last row is the subject of the next section.

Negative binomial regression

The third remedy of Overdispersion in Count Data is a distribution with an extra parameter. The negative binomial with mean $\mu$ and shape $k$, derived there as a Poisson count with a gamma-distributed rate, has variance $\mu + \mu^2/k$. Maximum Likelihood Estimation fitted it to Iowa's weekly deaths with one common mean, found $\hat k = 203.49$, and proved that the estimate exists when the variance exceeds the mean. A regression puts a model on the mean, $\ln\mu_i = \mathbf{x}_i^{\mathsf T}\boldsymbol\beta$, and keeps one shape for all.

Proposition 1 (the negative binomial with known shape is a generalized linear model). For fixed $k$ the negative binomial probabilities have the exponential dispersion form with $\phi = 1$, natural parameter $\theta = \ln[\mu/(k + \mu)]$, cumulant function $b(\theta) = -k\ln(1 - e^\theta)$ and variance function $V(\mu) = \mu + \mu^2/k$.

Proof. The probabilities, in the form of Maximum Likelihood Estimation, are

$$P(Y = y) = \frac{\Gamma(y + k)}{y!\,\Gamma(k)}\Big(\frac{k}{k + \mu}\Big)^{k}\Big(\frac{\mu}{k + \mu}\Big)^{y} = \exp\Big\{y\theta + k\ln(1 - e^\theta) + \ln\frac{\Gamma(y + k)}{y!\,\Gamma(k)}\Big\},$$

since $e^\theta = \mu/(k + \mu)$ and $1 - e^\theta = k/(k + \mu)$. So $b(\theta) = -k\ln(1 - e^\theta)$, with $\theta < 0$. Then $b'(\theta) = ke^\theta/(1 - e^\theta) = k\cdot\frac{\mu}{k + \mu}\cdot\frac{k + \mu}{k} = \mu$, and $b''(\theta) = ke^\theta/(1 - e^\theta)^2 = \mu\cdot\frac{k + \mu}{k} = \mu + \mu^2/k$. $\square$

The log link is not the canonical link of this family (that would be $\theta$ itself), so the score carries the weights of a non-canonical link.

Proposition 2 (the score equations). With $\ln\mu_i = \mathbf{x}_i^{\mathsf T}\boldsymbol\beta$, the score for $\boldsymbol\beta$ is

$$\mathbf{U}(\boldsymbol\beta) = \sum_i\frac{y_i - \mu_i}{1 + \mu_i/k}\,\mathbf{x}_i,$$

the quasi-score of Theorem 1 with $V(\mu) = \mu + \mu^2/k$ and $\phi = 1$. The expected value of $\partial\mathbf{U}/\partial k$ is zero: the coefficients and the shape are orthogonal parameters, and the expected information matrix is block-diagonal between them.

Proof. From the log-probability, $\partial\ln P/\partial\mu = y/\mu - (y + k)/(k + \mu) = k(y - \mu)/[\mu(k + \mu)]$. With $\partial\mu_i/\partial\boldsymbol\beta = \mu_i\mathbf{x}_i$, the score is $\sum_i k(y_i - \mu_i)\mathbf{x}_i/(k + \mu_i)$, which is the stated form. The quasi-score with $g = \ln$ has $g'(\mu) = 1/\mu$ and weight $\mathbf{x}_i/(g'(\mu_i)V(\mu_i)) = \mu_i\mathbf{x}_i/(\mu_i + \mu_i^2/k)$, the same. Differentiating the score in $k$ gives $\sum_i(y_i - \mu_i)\mathbf{x}_i\,\partial_k[k/(k + \mu_i)]$, linear in $y_i - \mu_i$, so its expectation is zero. $\square$

Two consequences follow. For a known $k$ the fit is IRLS with working weights $\mu_i/(1 + \mu_i/k)$. And because of the orthogonality, the large-sample covariance of $\hat{\boldsymbol\beta}$ is the inverse of its own block of the information, $(X^{\mathsf T}WX)^{-1}$ with those weights, whether $k$ is known or estimated. The joint fit alternates: IRLS for $\boldsymbol\beta$ with $k$ fixed, then the profile equation for $k$ with the means fixed (the equation of Maximum Likelihood Estimation, with $\mu_i$ in place of the common mean), until neither changes. Lawless (1987) studies negative binomial regression and "examines efficiency and robustness properties of inference procedures based on them", comparing them with quasi-likelihood methods, and Breslow (1984) adapts the reweighted least squares of log-linear models to accommodate extra-Poisson variation (their abstracts).

Iowa. The negative binomial regression with the same eight terms gives $\hat k = 1{,}090$, far larger than the 203.49 of the model with a constant mean: once the season and the trend are in the mean, little variation is left for the gamma rate to explain. The trend is 0.0116, and its standard error is in the table above. The implied dispersion at the average week, $1 + \bar\mu/\hat k$, is 1.523, close to the quasi-Poisson 1.566. Pearson's statistic computed with the negative binomial variance is 312.9, a ratio of 1.026 to its degrees of freedom, so the quadratic variance function accounts for the dispersion on average. The maximized log-likelihood is $-1{,}501.73$ against $-1{,}518.09$ for the Poisson regression, a likelihood ratio statistic of 32.73. The Poisson model is the boundary $k = \infty$ of the negative binomial family, so Wilks's theorem does not supply a reference distribution, as Likelihood Ratio, Wald and Score Tests explains. Dean and Lawless (1989) develop score tests for extra-Poisson variation that require only the Poisson model to be fitted (their abstract). Here the Pearson test above is decisive anyway. The likelihood ratio test of the second and third harmonics within the negative binomial model is 9.51 on 4 degrees of freedom ($p = 0.0496$), close to the quasi-likelihood $F$ test.

Linear or quadratic variance?

Quasi-Poisson and negative binomial regression both have one extra parameter and usually give similar answers. Ver Hoef and Boveng (2007) explain when they do not: "The variance of a quasi-Poisson model is a linear function of the mean while the variance of a negative binomial model is a quadratic function of the mean," and since the variance sets the weights of the fit, "large and small counts get weighted differently" (their abstract). The quasi-Poisson weights are proportional to $\mu_i$; the negative binomial weights $\mu_i/(1 + \mu_i/k)$ grow more slowly and level off at $k$.

For Iowa the difference is small, because the fitted means vary only between 507.5 and 646.0. The ratio of the largest to the smallest weight is 1.273 for quasi-Poisson and 1.171 for the negative binomial. In California, with 5,027.7 deaths a week on average, the shape is $\hat k = 974$ and $\mu/k$ is about 5, so the negative binomial weights are nearly flat: their ratio is 1.047, against 1.331 for quasi-Poisson. Even so the two trends agree, 1.66 and 1.67 per cent a year, probably because within one state the weeks with large and small means are spread evenly over the years; the script does not measure this.

The two forms make different predictions across states. Under quasi-Poisson, the dispersion $\phi$ is a constant, with no reason to depend on the size of the state. Under the negative binomial, a state whose weekly deaths have mean $\mu$ has dispersion about $\operatorname{Var}(y)/\mu = 1 + \mu/k$, which grows with $\mu$. Overdispersion in Count Data showed that the raw ratio of variance to mean grows with the size of the state, but much of that ratio was the season. Fitting the seasonal model separately to all 53 reporting areas (the 50 states, with New York State reported without New York City, the District of Columbia, New York City and Puerto Rico) removes the season from each and leaves the dispersion around it.

The dispersion left after the seasonal Poisson regression (Pearson's statistic over its 305 degrees of freedom) in each of the 53 reporting areas, against the area's mean weekly deaths, on logarithmic scales. The curve is the negative binomial prediction 1 + mean / 1,104 with one shape for all areas; the dashed line is a single dispersion for all, the median 1.65. Notice that the points follow the rising curve from Alaska to California, and that Puerto Rico lies far above it.
The dispersion left after the seasonal Poisson regression (Pearson's statistic over its 305 degrees of freedom) in each of the 53 reporting areas, against the area's mean weekly deaths, on logarithmic scales. The curve is the negative binomial prediction 1 + mean / 1,104 with one shape for all areas; the dashed line is a single dispersion for all, the median 1.65. Notice that the points follow the rising curve from Alaska to California, and that Puerto Rico lies far above it.

The dispersions run from 1.05 to 6.76, with median 1.65 (the raw ratios of variance to mean have median 4.71 over these 53 areas; the 4.728 of Overdispersion in Count Data is over its own set of 51 areas with complete cancer counts). A common shape fitted to the 53 areas by least squares on $\phi - 1 = \mu/k$ gives $k = 1{,}104$, close to Iowa's own 1,090. On the logarithmic scale, the slope of $\ln(\phi - 1)$ against $\ln\bar\mu$ is 0.94: the negative binomial predicts a slope of 1, a constant dispersion a slope of 0. Iowa's dispersion of 1.566 is near its predicted 1.517. A dispersion above 2 is found in 18 areas. The one clear outlier is Puerto Rico, with a mean of 564.4 deaths a week and a dispersion of 3.37; its largest Pearson residual, 13.0, and the next two largest fall in weeks 38 to 40 of 2017, a short episode that no smooth seasonal model describes.

So across areas the quadratic variance function describes the data and the constant dispersion does not. This is a statement about the 53 areas taken together. It supports a negative binomial model, or a quasi-likelihood with $V(\mu) = \mu + \mu^2/k$, when a single regression is fitted to counts of very different sizes. Within one state it hardly matters.

What independence costs

All three methods assume the weeks are independent. They are not quite: the Pearson residuals of Iowa's seasonal model have a lag-1 autocorrelation of 0.16 (and $-0.03$ at lag 52, so the cycle itself is captured). An epidemic that raises deaths one week raises them the next. Quasi-likelihood repairs the variance of each count, not the covariance between counts, and the sandwich as written above uses only squared residuals of single weeks.

The experiment. To see what each method's intervals do, the script simulated 313-week series with Iowa's fitted negative binomial means and asked how often the 95% interval for the trend covered the true value. It ran 200 series for each of 30 seeds, in two scenarios.

  • Independent overdispersion. Each week a gamma rate with shape 1,090 and the fitted mean, then a Poisson count: independent negative binomial weeks, the model the negative binomial regression assumes.
  • Serially correlated overdispersion. The log rate of each week is the fitted value plus an autoregressive disturbance $u_t = 0.6\,u_{t-1} + e_t$, with $\rho = 0.6$ and a variance chosen so that the overall dispersion matches Iowa's; then a Poisson count. In these series the median lag-1 autocorrelation of the Pearson residuals was 0.18, against Iowa's 0.16 (and $-0.03$ in the independent scenario), and the median dispersion 1.53.

Coverage of nominal 95% intervals for the trend, median and interquartile range over the 30 seeds:

Method Independent Serially correlated
Poisson 0.878 (0.866, 0.895) 0.728 (0.705, 0.744)
Quasi-Poisson 0.940 (0.935, 0.950) 0.830 (0.810, 0.845)
Sandwich 0.935 (0.926, 0.950) 0.813 (0.801, 0.840)
Negative binomial 0.935 (0.926, 0.950) 0.823 (0.805, 0.845)

With independent weeks, the three methods that allow for overdispersion come close to 95%, a little under it, while the Poisson intervals cover 88% of the time. The simulated series have a true dispersion that averages 1.523 over the weeks, so the Poisson standard error is too small by about $\sqrt{1.523} = 1.234$, and the predicted coverage is $P(|Z| < 1.96/1.234) = 0.888$; the Poisson row is a point below that, probably a finite-sample effect (the quasi-Poisson intervals also fall a point short of 0.95). With correlated weeks, every method falls short. Allowing for overdispersion still raises the coverage, from 0.728 to between 0.813 and 0.830, but each of the three methods covers eleven to twelve points less often than it did with independent weeks. These are results for this design and these two generating processes. We would expect stronger dependence to lower the coverage further; the experiment tested only $\rho = 0.6$.

Correlation between weeks calls for methods that model it or resample around it: Resampling Dependent and Clustered Data resamples blocks of weeks, and Stochastic Processes and Time Series models the dependence directly.

What is proved here, what is cited, and what the experiment shows

Proved on this page: the quasi-score's mean and information identity (Theorem 1); the optimality of its weights among linear estimating functions, in the first-order covariance (Theorem 2); the expectation of Pearson's statistic with known means and its exact divisor in the linear model (Theorem 3); the invariance of the quasi-Poisson coefficients; the negative binomial with fixed shape as an exponential dispersion family with $V(\mu) = \mu + \mu^2/k$ (Proposition 1); its score equations and the orthogonality of shape and coefficients (Proposition 2).

Taken from other pages: the score identities (Fisher Information and the Cramér–Rao Bound); IRLS; $E[\text{RSS}] = (n - p)\sigma^2$ (The Gauss–Markov Theorem); the exact $F$ test of the linear model (The General Linear Hypothesis); the negative binomial as a gamma mixture and its fit with one mean (Overdispersion in Count Data, Maximum Likelihood Estimation); the boundary problem for the likelihood ratio test (Likelihood Ratio, Wald and Score Tests).

Cited, not proved: the consistency and asymptotic normality of maximum quasi-likelihood estimates (McCullagh, 1983); the identity of log-likelihood and quasi-likelihood in one-parameter exponential families (Wedderburn, 1974); the robust variance in estimating equations for dependent data (Liang and Zeger, 1986); the comparison of quasi-Poisson and negative binomial weighting (Ver Hoef and Boveng, 2007); negative binomial regression and its properties (Lawless, 1987); the modified reweighted least squares fit for extra-Poisson variation in log-linear models (Breslow, 1984); score tests for overdispersion (Dean and Lawless, 1989). The divisor $n - p$ outside the linear model and the sandwich estimator's consistency are given with first-order arguments, not proofs.

Observed in the experiment: the coverages and dispersions of the table, for 313-week series built on Iowa's fitted means, two generating processes and 30 seeds of 200 series each. Every number on the page is computed by tools/experiments/statistics/overdispersion-and-quasi-likelihood.py, with the NCHS weekly counts loaded by tools/experiments/applied-statistics/distdata.py.

Worked problems

These problems are posed for this article on the unit's weekly deaths and on small exact cases; the unit has no course assignments. Every number is recomputed by the article's script.

Problem 1 (quasi-Poisson standard errors by hand). The Poisson regression of Iowa's weekly deaths gives the trend coefficient a standard error of 0.001383, and Pearson's statistic is 477.6 on 305 degrees of freedom. Find the quasi-Poisson standard error and the 95% interval for the annual change, and say what assumption each step uses.

Solution. The dispersion estimate is $\hat\phi = 477.6/305 = 1.5658$ (Theorem 3, with the first-order divisor $n - p$). The quasi-Poisson covariance is $\phi$ times the Poisson one, so the standard error is $0.001383 \times \sqrt{1.5658} = 0.001730$. The coefficient is unchanged, $\hat\beta_1 = 0.0115$, because the quasi-score is a multiple of the Poisson score (it assumes only that the variance is proportional to the mean). The interval for $\beta_1$ is $0.0115 \pm 1.96 \times 0.001730$, and exponentiating its ends gives (0.82, 1.50) per cent a year, the second row of the table. The step from the estimating equation to a normal interval uses the large-sample theory cited from McCullagh (1983), and the independence of the weeks, which the residual autocorrelation of 0.16 puts in doubt.

Problem 2 (the expected Pearson statistic). (a) A Poisson count has mean 4. Show that the expected Pearson term $(Y - 4)^2/4$ is 1. (b) A negative binomial count has mean 10 and shape 2. If a quasi-Poisson model with $V(\mu) = \mu$ is used, what is the expected Pearson term, and what does it say about a constant $\phi$?

Solution. (a) $E[(Y - \mu)^2] = \operatorname{Var}(Y) = \mu$ for a Poisson count, so $E[(Y - 4)^2/4] = 4/4 = 1$, which is $\phi = 1$ in Theorem 3. (b) The variance is $\mu + \mu^2/k = 10 + 100/2 = 60$, so $E[(Y - 10)^2/10] = 6$. In general the expected term is $1 + \mu/k$, which depends on $\mu$. A quasi-Poisson model applied to counts with negative binomial variance therefore has a dispersion that varies from observation to observation, larger for larger means. Its $\hat\phi$ averages these values, which is harmless when the means vary little (as within Iowa) and misleading when they vary a lot (as across the 53 areas). The script checks both expectations by summing over the probability distributions.

Problem 3 (California). California's weekly deaths average 5,027.7. Predict the dispersion of its seasonal Poisson regression from the common shape $k = 1{,}104$, compare it with the observed 6.76, and say by how much the Poisson standard errors understate the uncertainty.

Solution. The prediction is $1 + 5{,}027.7/1{,}104 = 5.55$, against an observed 6.76. California lies above the curve, though far closer to it than to the constant dispersion of 1.65. The quasi-Poisson standard errors are the Poisson ones times $\sqrt{6.76} = 2.60$. In California the Poisson model makes every interval less than half as wide as it should be, even after the season and the trend are modeled. Its negative binomial fit has $\hat k = 974$, so on its own data California is a little more dispersed than the common shape predicts.

Summary

Quantity Formula
Assumptions $E[y_i] = \mu_i(\boldsymbol\beta)$, $\operatorname{Var}(y_i) = \phi V(\mu_i)/w_i$, independent
Quasi-score $\mathbf{U} = D^{\mathsf T}\Sigma^{-1}(\mathbf{y} - \boldsymbol\mu)$; $E\mathbf{U} = \mathbf{0}$; $\operatorname{Cov}\mathbf{U} = D^{\mathsf T}\Sigma^{-1}D$
Quasi-likelihood $Q = \sum w_i\int_{y_i}^{\mu_i}(y_i - t)/(\phi V(t))\,dt$; Poisson log-likelihood$/\phi$ when $V = \mu$
Optimality $K_A \succeq (D^{\mathsf T}\Sigma^{-1}D)^{-1}$ for every linear estimating function
Dispersion $\hat\phi = X^2/(n - p)$; SE $\times\sqrt{\hat\phi}$; $F = \Delta D/(q\hat\phi)$
Sandwich $\mathcal{I}^{-1}\big[\sum(y_i - \hat\mu_i)^2\mathbf{x}_i\mathbf{x}_i^{\mathsf T}\big]\mathcal{I}^{-1}$
Negative binomial $V(\mu) = \mu + \mu^2/k$; score $\sum(y_i - \mu_i)\mathbf{x}_i/(1 + \mu_i/k)$; $k$ orthogonal to $\boldsymbol\beta$
Across areas dispersion $\approx 1 + \mu/k$, $k = 1{,}104$; slope 0.94 on the log scale

What we have learned

When counts vary more than the Poisson model allows, the regression coefficients can still be estimated from the mean and the variance function alone. The quasi-score is unbiased, satisfies the information identity and has the best weights among linear estimating functions, and the dispersion it leaves free is estimated from Pearson's statistic. For Iowa's weekly deaths, a seasonal model reduces the excess variance from 3.85 times the mean to 1.57 times, and the remaining excess widens every interval by a quarter and turns a strong-looking test of the seasonal shape into a borderline one. Across the 53 areas the excess grows with the size of the area as the negative binomial's quadratic variance predicts. What none of these methods repairs is dependence between weeks, which in a simulation built on Iowa cost each method eleven to twelve points of coverage.

The unit's last article, Multinomial and Softmax Regression, leaves counts and rates for outcomes with several categories, the model at the output layer of a language model.

References