Overdispersion in Count Data

Why real counts vary more than the Poisson model allows, and what to do about it. Weekly deaths from NCHS for 2014 to 2019, where cancer deaths fit the Poisson and deaths from all causes, with their winter peaks, vary nearly four times as much in Iowa; the proof that a varying rate adds its own variance to the mean; the growth of the excess with the size of the state; and the remedies, with the negative binomial derived as a gamma mixture of Poisson counts. A worked problem on California's weekly deaths.

Iowa recorded a mean of 570.52 deaths a week from all causes over the 313 weeks of 2014 to 2019. A Poisson count has a variance equal to its mean, so the Poisson model allows these weekly counts a variance of 570.52. Their variance is 2,196.66, which is 3.85 times as much. Why do real counts vary more than the Poisson model allows, and what should be done about it? They vary more because the rate behind them is not constant: a count whose rate varies has a variance equal to its mean plus the variance of the rate. The remedies are to model the rate, to scale the variance up, or to use a distribution with one more parameter, the negative binomial.

Counts whose variance exceeds their mean are called overdispersed. The Poisson Distribution proves that a Poisson count has variance equal to its mean, defines the Poisson process with its three assumptions, and gives the test used here, Fisher's index of dispersion, which is referred to the chi-square distribution. This article assumes all three, and the law of total variance of Random Variables, Expectation and Variance.

First we compare two series of weekly deaths with the Poisson model: one fits and one does not. Then we prove why a varying rate inflates the variance, and follow the consequence across 51 states and other areas. Finally we list the models for overdispersed counts and derive the negative binomial.

Overdispersion in weekly deaths

Modern vital statistics give both answers: one cause of death behaves as a Poisson count, and deaths from all causes do not. The National Center for Health Statistics publishes the number of deaths in each state in each week, by the week in which they occurred, for 2014 to 2019 (National Center for Health Statistics, 2021). Iowa recorded a mean of 570.52 deaths a week from all causes over the 313 weeks, and 122.67 a week from cancer (malignant neoplasms, ICD-10 codes C00 to C97).

Weekly deaths in Iowa, 2014 to 2019, from all causes (top) and from cancer (bottom). The shaded band is the mean plus or minus two Poisson standard deviations, twice the square root of the mean. Notice that cancer deaths scatter within the band as a Poisson count would, while all-cause deaths rise every winter, with peaks that differ from year to year, and spend long stretches outside the band.
Weekly deaths in Iowa, 2014 to 2019, from all causes (top) and from cancer (bottom). The shaded band is the mean plus or minus two Poisson standard deviations, twice the square root of the mean. Notice that cancer deaths scatter within the band as a Poisson count would, while all-cause deaths rise every winter, with peaks that differ from year to year, and spend long stretches outside the band.
Iowa, 313 weeks Mean Variance Variance/mean $\chi^2$ (312 df) $p$
Cancer deaths 122.67 117.49 0.958 298.8 0.69
All-cause deaths 570.52 2,196.66 3.850 1,201.3 < 0.001

Cancer deaths behave as a Poisson count:

  • their variance is within a few percent of the mean, and the index of dispersion is unremarkable;
  • the weekly counts, from 97 to 160, stay almost entirely within two Poisson standard deviations of the mean;
  • the yearly means barely move (from 120.9 to 124.2), and there is little seasonal pattern.

Each week, a large population faces a small and nearly constant probability of death from cancer. That is the setting of the binomial limit from which The Poisson Distribution derives the distribution.

All-cause deaths do not behave as a Poisson count. Their variance is 3.85 times their mean, and the index of dispersion, 1,201 on 312 degrees of freedom, rules out the Poisson model beyond doubt. The figure shows why. Deaths from all causes rise every winter, the season of influenza and other respiratory infections, and the winter peaks differ in height from year to year. The rate is not constant, which breaks the second assumption of the Poisson process, a constant rate. The counts were also drifting upward, from a yearly mean of 556.3 in 2014 to 587.5 in 2019.

Removing the average seasonal pattern takes away part of the excess. Subtract from each week the mean for that week number over the six years. The residual variance is 1,000.55 (on 260 degrees of freedom), a ratio of 1.754 to the mean. The summer weeks alone (weeks 23 to 35, 78 weeks) have a ratio of 1.566. The remaining excess comes from the year-to-year differences in the winter peaks, the upward trend, and shorter-lived changes in the rate. A fixed seasonal pattern captures none of these.

Why a varying rate inflates the variance

Here is the idea in one sentence. A count whose rate wanders varies for two reasons, the Poisson chance at a given rate and the wandering of the rate itself, and the two variances add.

Suppose the count in each week is Poisson given that week's rate $\Lambda$, but the rate itself varies from week to week, with mean $\mu$ and variance $\operatorname{Var}(\Lambda)$. The law of total variance, proved in Random Variables, Expectation and Variance and used for clusters in The Binomial Distribution, gives

$$\operatorname{Var}(X) = E[\operatorname{Var}(X \mid \Lambda)] + \operatorname{Var}(E[X \mid \Lambda]) = E[\Lambda] + \operatorname{Var}(\Lambda) = \mu + \operatorname{Var}(\Lambda),$$

because a Poisson count with mean $\Lambda$ has $E[X \mid \Lambda] = \Lambda$ and $\operatorname{Var}(X \mid \Lambda) = \Lambda$. (The first of those articles proves the law for a rate with finitely many values, which covers the 313 weekly rates here. For a rate with a continuous distribution, such as the gamma below, the sums become integrals.) So the excess of the variance over the mean measures the variance of the underlying rate. For Iowa's all-cause deaths, $\operatorname{Var}(\Lambda) = 2{,}196.66 - 570.52 = 1{,}626.1$, a standard deviation of 40.3 deaths a week, or 7.1% of the mean.

Why does a rate that varies by only 7% produce a variance nearly four times the Poisson value? The Poisson part of the variance grows like $\mu$, while the part due to the varying rate grows like $\mu^2$. Write $c$ for the coefficient of variation of the rate, its standard deviation divided by its mean, so that $\operatorname{Var}(\Lambda) = c^2\mu^2$. The ratio of variance to mean is then $(\mu + c^2\mu^2)/\mu = 1 + c^2\mu$. For Iowa, $1 + 0.0707^2 \times 570.52$ recovers the observed 3.850.

That relation predicts that larger states have larger ratios. Of the 54 jurisdictions in the file, 51 have no suppressed weeks for these two causes. (Alaska and Wyoming have a few weeks of cancer deaths withheld because they were between 1 and 9, and the national total is excluded.) The 51 are 48 states, the District of Columbia, New York City, which the file reports apart from the rest of New York State, and Puerto Rico.

Variance against mean of weekly deaths, 2014 to 2019, for 51 states and other reporting areas, on logarithmic scales. Cancer deaths (circles) follow the dashed line where variance equals mean, as a Poisson count would, drifting above it in a few areas, mostly the largest. Notice that all-cause deaths (squares) lie above the line everywhere, and further above it the larger the state.
Variance against mean of weekly deaths, 2014 to 2019, for 51 states and other reporting areas, on logarithmic scales. Cancer deaths (circles) follow the dashed line where variance equals mean, as a Poisson count would, drifting above it in a few areas, mostly the largest. Notice that all-cause deaths (squares) lie above the line everywhere, and further above it the larger the state.

For cancer, the ratio of variance to mean has a median of 1.090 across the 51 areas (range 0.807 to 1.829). The largest ratios belong to Florida, Texas, Arizona and California. In each of them except California, weekly cancer deaths rose markedly over the six years; in Texas, from a yearly mean of 755.0 in 2014 to 806.4 in 2019. Removing a straight-line trend lowers the ratio:

  • for Florida, from 1.829 to 1.631;
  • for Arizona, from 1.524 to 1.393;
  • for Texas, from 1.656 to 1.236;
  • for California, only from 1.481 to 1.453.

A slow change in the rate is part of the excess, magnified, as the formula above says, by the size of the mean, but not all of it.

For all-cause deaths the median ratio is 4.728. It ranges from 1.432 to 35.58 in California, which averaged 5,027.7 deaths a week. Since the ratio is $1 + c^2\mu$, a seasonal swing of similar relative size (a coefficient of variation $c$ of 7.1% in Iowa and, as a worked problem below computes, 8.3% in California) makes Iowa's variance four times its mean and California's thirty-six times.

Models for overdispersed counts

Methods that assume Poisson variance understate standard errors, and the confidence intervals for rates from Crude Rates and Their Precision become too narrow. Three remedies are common:

  1. Model the variation in the rate directly, for example by giving each week of the year its own expected value. This is how excess deaths during an epidemic are measured.
  2. Scale the Poisson variance up by an estimated factor, the ratio of variance to mean.
  3. Use a distribution with an extra parameter.

The best known distribution of the third kind is the negative binomial. It is the distribution of a Poisson count whose mean varies according to a gamma distribution with shape $k$. Its variance is $\mu + \mu^2/k$, a special case of the formula above with $\operatorname{Var}(\Lambda) = \mu^2/k$. That is the variance of a gamma distribution with shape $k$ and mean $\mu$. Its density is proportional to $x^{k-1}e^{-x/\theta}$ on $x > 0$, with scale $\theta = \mu/k$ and normalizing constant $\Gamma(k)\theta^k$, where $\Gamma(k) = \int_0^\infty u^{k-1}e^{-u}\,du$ is the gamma function. Substituting $u = x/\theta$ gives $E[\Lambda^r] = \theta^r\,\Gamma(k + r)/\Gamma(k)$, and $\Gamma(k + 1) = k\Gamma(k)$ by integration by parts. So $E[\Lambda] = k\theta = \mu$, $E[\Lambda^2] = k(k + 1)\theta^2$, and $\operatorname{Var}(\Lambda) = k\theta^2 = \mu^2/k$.

Mixing the Poisson probabilities over this density gives the distribution itself:

$$P(X = x) = \int_0^\infty \frac{e^{-\lambda}\lambda^x}{x!}\cdot\frac{\lambda^{k-1}e^{-\lambda/\theta}}{\Gamma(k)\theta^k}\,d\lambda = \frac{1}{x!\,\Gamma(k)\theta^k}\int_0^\infty \lambda^{x+k-1}e^{-\lambda(1 + 1/\theta)}\,d\lambda.$$

The substitution $u = \lambda(1 + 1/\theta)$ turns the last integral into $\Gamma(x + k)\,\big(\theta/(1 + \theta)\big)^{x+k}$, and so

$$P(X = x) = \frac{\Gamma(x + k)}{x!\,\Gamma(k)}\Big(\frac{1}{1 + \theta}\Big)^{k}\Big(\frac{\theta}{1 + \theta}\Big)^{x}, \qquad x = 0, 1, 2, \ldots$$

This is the pmf of the negative binomial distribution with shape $k$ and mean $k\theta = \mu$. The script confirms the formula against numerical integration and against the library's negative binomial.

Example (a common misstatement). "Overdispersion means the events are not independent." Dependence between events is one cause. A rate that varies, with the events still independent given the rate, is another, and it is the one the weekly deaths show: each week's deaths can be independent of one another while the weekly rate rises every winter.

The script checked this by simulation. It drew a gamma rate with shape 2 and mean 10 and then a Poisson count with that rate, 200,000 times for each of 30 seeds. The variance had median 60.07, against $10 + 100/2 = 60$. The probability of exactly 5 events had median 0.0669, against 0.0670 from the negative binomial and 0.0378 from the Poisson with the same mean. For Iowa's all-cause deaths, $k = \mu^2/\operatorname{Var}(\Lambda) = 200.2$.

The information retrieval wiki's 2-Poisson Model meets the same problem in a different setting. The number of times a word occurs in a document is badly overdispersed relative to a single Poisson, because documents differ in how much they are about the word. A mixture of two Poisson distributions is one answer.

Worked problems

The problem is posed on this article's own weekly death counts, not taken from the course text. The script named at the end of the article checks every number.

Problem 1 (the varying rate behind California's deaths). California averaged 5,027.7 deaths a week from all causes, with a ratio of variance to mean of 35.58. Find the variance and the coefficient of variation of the underlying weekly rate, the shape $k$ of the negative binomial with the same mean and variance, and the ratio California would have with Iowa's coefficient of variation.

Solution. The variance of the counts is 178,879, the mean 5,027.7 times the unrounded ratio 35.578 (all values are computed before rounding). By the formula $\operatorname{Var}(X) = \mu + \operatorname{Var}(\Lambda)$,

$$\operatorname{Var}(\Lambda) = 178{,}879 - 5{,}028 = 173{,}851,$$

a standard deviation of 417.0 deaths a week. The coefficient of variation of the rate is $c = 417.0/5{,}027.7 = 0.0829$, or 8.3%. Substituting back, $1 + c^2\mu$ with the unrounded $c$ returns the ratio 35.58. A Poisson count with this mean would have standard deviation $\sqrt{5{,}027.7} = 70.9$; the weekly counts have 422.9.

The negative binomial with variance $\mu + \mu^2/k$ matches when $\mu^2/k = \operatorname{Var}(\Lambda)$, so $k = \mu^2/\operatorname{Var}(\Lambda) = 1/c^2 = 145.4$.

With Iowa's $c = 0.0707$ the ratio would be $1 + 0.0707^2 \times 5{,}027.7 = 26.12$. California's observed 35.58 is larger because its rate varies a little more in relative terms, 8.3% against 7.1%. Most of the difference between the two states' ratios, 3.85 against 35.58, comes from the factor $\mu$, the size of the state.

Summary

Quantity Formula
Varying rate $\operatorname{Var}(X) = \mu + \operatorname{Var}(\Lambda)$; ratio $1 + c^2\mu$
Negative binomial variance $\mu + \mu^2/k$

What we have learned

Weekly cancer deaths in Iowa behave as a Poisson count: their variance is within a few percent of their mean. All-cause deaths do not: a rate that swings by about 7% makes their variance 3.85 times their mean. A count whose rate varies has variance equal to its mean plus the variance of the rate, so the excess grows with the size of the state. The negative binomial, a Poisson count with a gamma-distributed rate, is the standard distribution for such counts.

Heights, blood pressures and the means of samples are not whole-number counts. The Normal Distribution shows how probabilities are assigned to measurements that can take any value in an interval, and why one bell-shaped curve describes so many of them.

Every number in this article is computed by tools/experiments/applied-statistics/poisson-distribution.py in the site's repository (its Parts 5 and 8 and the checks under "Reference pass"), with the NCHS weekly death counts loaded by distdata.py beside it.

References