The Poisson Distribution

The distribution of counts of rare events. The Poisson probability derived as the limit of the binomial, with its mean and variance both equal to lambda; sums of independent Poisson counts; the Poisson process, its time scaling and exponential waiting times; Bortkiewicz's horse-kick deaths in the Prussian army; and overdispersion, tested with Fisher's index of dispersion on weekly deaths from NCHS, where cancer deaths fit the Poisson and all-cause deaths, with their winter peaks, do not.

From 1875 to 1894 the Prussian army recorded, for each of its army corps and each year, the number of soldiers killed by the kick of a horse. In 1898 Ladislaus von Bortkiewicz tabulated the counts for the 200 corps-years of ten comparable corps: 109 years with no such death, 65 with one, 22 with two, 3 with three and 1 with four. Nobody recorded how many men were at risk or how likely each was to be kicked. Can the pattern of counts be predicted from their average alone?

It can. These are counts of a rare event, in a population of thousands of men each with a tiny chance of the accident in a given year. Bortkiewicz showed that they follow the distribution that Siméon-Denis Poisson had published in 1837 for events with a very small chance in a very large number of trials (Bortkiewicz, 1898, §12; O'Connor and Robertson, 2002). Fisher reprinted the table in Statistical Methods for Research Workers as his first example of the Poisson series (Fisher, 1925; 2nd ed. 1928, §15), and it has remained a standard textbook example.

Counts like these are everywhere in public health: deaths in a county in a year, cases of a disease in a week, admissions to an emergency department in an hour. The Rates and Standardization unit modeled the numerator of every rate as a Poisson count and built its confidence intervals on that model. This article supplies the model. It assumes the binomial distribution and the rules for expected values and variances of The Binomial Distribution.

First we derive the Poisson distribution as the limit of the binomial, and prove its central property: the variance equals the mean. Then we describe the Poisson process, which produces the same distribution for events in time. Next we test the model on the horse kicks. Finally we test it on weekly death counts, where the result cuts both ways. Weekly cancer deaths in Iowa fit the Poisson almost exactly, while weekly deaths from all causes vary nearly four times as much as the model allows, and the reason is visible in the data.

From the binomial to the Poisson

Consider a population of $n$ people, each of whom independently has a small probability $p$ of the event in the period of interest. The number of events is $\text{Bin}(n, p)$, with mean $\lambda = np$. When $n$ is in the thousands, the binomial probabilities are awkward to compute by hand. Worse, $n$ and $p$ are often not even known separately: the army's records gave the number of deaths per corps, not the number of men at risk or the risk per man. What is known is the average count $\lambda$.

Here is the plan. Hold $\lambda$ fixed and let $n$ grow, with $p = \lambda/n$. If the binomial probabilities settle down to a limit that depends only on $\lambda$, that limit is the distribution we want.

Write the binomial probability as a product of four factors:

$$\binom{n}{k}\left(\frac{\lambda}{n}\right)^{k}\left(1 - \frac{\lambda}{n}\right)^{n-k} = \frac{\lambda^k}{k!} \cdot \frac{n(n-1)\cdots(n-k+1)}{n^k} \cdot \left(1 - \frac{\lambda}{n}\right)^{n} \cdot \left(1 - \frac{\lambda}{n}\right)^{-k}.$$

Take a fixed $k$ and follow each factor as $n$ grows:

  • The first factor does not depend on $n$.
  • The second is $\frac{n}{n} \cdot \frac{n-1}{n} \cdots \frac{n-k+1}{n}$, a product of $k$ terms each tending to 1.
  • The third tends to $e^{-\lambda}$, by the limit $(1 + x/n)^n \to e^x$ that defines the exponential function.
  • The fourth tends to $1^{-k} = 1$.

So

$$\lim_{n \to \infty} \binom{n}{k}\left(\frac{\lambda}{n}\right)^{k}\left(1 - \frac{\lambda}{n}\right)^{n-k} = \frac{e^{-\lambda}\lambda^k}{k!}, \qquad k = 0, 1, 2, \ldots$$

A random variable with these probabilities has the Poisson distribution with mean $\lambda$, written $X \sim \text{Poisson}(\lambda)$ (NIST/SEMATECH, §1.3.6.6.19; Grinstead and Snell, 1997, §5.1). Unlike the binomial, it has no upper limit: any count $0, 1, 2, \ldots$ is possible. The probabilities add to 1 by the power series of the exponential:

$$\sum_{k=0}^{\infty} \frac{e^{-\lambda}\lambda^k}{k!} = e^{-\lambda} \sum_{k=0}^{\infty} \frac{\lambda^k}{k!} = e^{-\lambda} e^{\lambda} = 1.$$

The parameter $\lambda$ is the expected number of events in the period counted. If deaths occur at 0.7 per corps per year, the count in one corps-year has $\lambda = 0.7$, and the count in one corps over ten years has $\lambda = 7$. A rate becomes a Poisson mean only once the size of the population and the length of the period are fixed.

How fast the limit arrives

How large must $n$ be for the limit to be useful?

Probabilities of k events for a binomial with n = 5 and p = 0.14, a binomial with n = 20 and p = 0.035, and the Poisson distribution with the same mean 0.7. Notice that already at n = 20 the binomial and the Poisson bars are hard to tell apart.
Probabilities of k events for a binomial with n = 5 and p = 0.14, a binomial with n = 20 and p = 0.035, and the Poisson distribution with the same mean 0.7. Notice that already at n = 20 the binomial and the Poisson bars are hard to tell apart.

The script compared $\text{Bin}(n, 0.7/n)$ with $\text{Poisson}(0.7)$ for several $n$. As a single measure of the difference it used the total variation distance: half the sum of the absolute differences between the two sets of probabilities. It equals the largest difference between the probabilities the two distributions give to any one event.

$n$ $p$ $P(0)$ $P(1)$ $P(2)$ $P(3)$ Total variation
5 0.1400 0.4704 0.3829 0.1247 0.0203 0.0383
20 0.0350 0.4904 0.3557 0.1226 0.0267 0.0090
100 0.0070 0.4954 0.3492 0.1219 0.0281 0.0018
1,000 0.0007 0.4965 0.3478 0.1217 0.0284 0.0002
Poisson 0.4966 0.3476 0.1217 0.0284

The distance falls roughly in proportion to $p$; in this table it is a little below $np^2/2$ in every row. What matters for the approximation is that $p$ be small, not that $n$ be huge. A count of events among 20 people, each with a 3.5% chance, is already close to Poisson. This is the regime where the normal approximation to the binomial does worst, as The Normal Approximation to the Binomial shows, and the Poisson takes over.

Mean and variance

The mean of a Poisson count is $\lambda$, and so is its variance. The proof uses the same device as the binomial's second proof: cancel a factor of $k$ against $k!$ and recognize what is left as a sum of Poisson probabilities.

$$E[X] = \sum_{k=1}^{\infty} k\,\frac{e^{-\lambda}\lambda^k}{k!} = \lambda \sum_{k=1}^{\infty} \frac{e^{-\lambda}\lambda^{k-1}}{(k-1)!} = \lambda,$$

since the last sum, with $j = k - 1$, is the sum of all the Poisson probabilities. In the same way,

$$E[X(X - 1)] = \sum_{k=2}^{\infty} k(k-1)\,\frac{e^{-\lambda}\lambda^k}{k!} = \lambda^2 \sum_{k=2}^{\infty} \frac{e^{-\lambda}\lambda^{k-2}}{(k-2)!} = \lambda^2,$$

and so

$$\operatorname{Var}(X) = E[X(X-1)] + E[X] - (E[X])^2 = \lambda^2 + \lambda - \lambda^2 = \lambda. \qquad \square$$

The same result follows from the binomial: the variance $np(1 - p) = \lambda(1 - \lambda/n)$ tends to $\lambda$ as $n$ grows.

Check your understanding. A Poisson count has mean 100. About how far from 100 does it typically fall? And a count with mean 10,000?

The standard deviation is $\sqrt{\lambda}$: about 10 for a mean of 100, and about 100 for a mean of 10,000. The relative spread, $1/\sqrt{\lambda}$, shrinks from 10% to 1%. That is why a rate based on many events is more precise than one based on few, whatever the size of the population. Rates in Public Health turns this into the relative standard error $1/\sqrt{D}$ for $D$ deaths.

Equality of mean and variance is the property that makes the Poisson model testable. Three kinds of count differ here:

  • A binomial count has variance less than its mean, by the factor $1 - p$.
  • A Poisson count has variance equal to its mean.
  • As the last section shows, counts from a population whose rate varies have variance greater than their mean.

Comparing the variance of observed counts with their mean is therefore the first check of the model.

The distribution is skewed to the right, with skewness $1/\sqrt{\lambda}$ (NIST/SEMATECH, §1.3.6.6.19). The skewness is 1.195 for $\lambda = 0.7$, 0.5 for $\lambda = 4$, and 0.090 for $\lambda = 122.67$, each confirmed by summing the series in the script. Its most likely value is the largest whole number below $\lambda$. When $\lambda$ is a whole number, $\lambda$ and $\lambda - 1$ tie, as $P(X = 3) = P(X = 4) = 0.1954$ do for $\lambda = 4$.

Sums of independent Poisson counts

Does a sum of Poisson counts stay Poisson? If $X \sim \text{Poisson}(\lambda)$ and $Y \sim \text{Poisson}(\mu)$ are independent, then $X + Y \sim \text{Poisson}(\lambda + \mu)$. The event $X + Y = n$ splits into the disjoint events $X = k, Y = n - k$ for $k = 0, \ldots, n$, so

$$P(X + Y = n) = \sum_{k=0}^{n} \frac{e^{-\lambda}\lambda^k}{k!}\cdot\frac{e^{-\mu}\mu^{n-k}}{(n-k)!} = \frac{e^{-(\lambda+\mu)}}{n!}\sum_{k=0}^{n}\binom{n}{k}\lambda^k\mu^{n-k} = \frac{e^{-(\lambda+\mu)}(\lambda+\mu)^n}{n!},$$

by the binomial theorem. $\square$

So deaths can be added across days, counties or age groups and remain Poisson, as long as the parts are independent. This is also what makes the indirect standardization of Direct and Indirect Standardization work. The observed deaths in a study population, summed over age groups, form a single Poisson count to compare with its expected value.

Check your understanding. The ten comparable corps averaged 0.61 horse-kick deaths a year. If one corps's yearly counts are independent Poisson counts with that mean, what is the distribution of its total over ten years, and how likely is a decade with no such death?

The ten yearly counts add to a $\text{Poisson}(6.1)$ count, with standard deviation $\sqrt{6.1} = 2.470$. A decade with no death has probability $e^{-6.1} = 0.0022$.

The Poisson process

The binomial limit explains the Poisson distribution for a fixed population. What about events in time, such as arrivals at an emergency department, where there is no population of trials to count? A second derivation covers them. Suppose events occur at random times, and that:

  1. the numbers of events in non-overlapping time intervals are independent;
  2. the rate is constant: in a short interval of length $h$, the probability of an event is approximately $\lambda h$, the same at every time;
  3. events do not coincide: the probability of two or more events in a short interval is negligible compared with the probability of one.

The idea is to turn time into trials. Divide an interval of length $t$ into $n$ equal pieces. By the third assumption each piece holds at most one event. By the second, it holds one with probability about $\lambda t/n$. By the first, the pieces are independent. The number of events is then approximately $\text{Bin}(n, \lambda t/n)$. Letting $n$ grow gives exactly the limit derived above, with mean $\lambda t$:

$$P\big(N(t) = k\big) = \frac{e^{-\lambda t}(\lambda t)^k}{k!}.$$

Events that satisfy the three assumptions form a Poisson process with rate $\lambda$ (Grinstead and Snell, 1997, §5.1). The count in any interval is Poisson with mean equal to the rate times the length, and counts in disjoint intervals are independent. This is consistent with the result on sums: two adjacent intervals of lengths $s$ and $t$ hold $\text{Poisson}(\lambda s)$ and $\text{Poisson}(\lambda t)$ counts, which add to $\text{Poisson}(\lambda(s + t))$.

Waiting times

The waiting time $T$ until the next event has a simple distribution too. There is no event within time $t$ exactly when the count in $[0, t]$ is zero, so

$$P(T > t) = P\big(N(t) = 0\big) = e^{-\lambda t}.$$

$T$ has the exponential distribution, with mean $1/\lambda$: events at a rate of 2 per hour arrive on average half an hour apart. The information retrieval wiki's Crawl Freshness uses the same model for changes to web pages.

Check your understanding. Events arrive at 2 per hour, and an hour has passed with none. Is the next event now "overdue", more likely to arrive within the next half hour than it was at the start?

No. Because counts in disjoint intervals are independent, the empty hour carries no information about what comes next. The chance of waiting more than another half hour is

$$P(T > 1.5 \mid T > 1) = \frac{e^{-2 \times 1.5}}{e^{-2 \times 1}} = e^{-1} = 0.3679,$$

exactly the chance of waiting more than half an hour from the start, $P(T > 0.5)$. This is the memoryless property of the exponential distribution (Grinstead and Snell, 1997, §5.2). In 30 seeded simulations of 200,000 waiting times each, the share of waits longer than 1 hour that also exceeded 1.5 hours had median 0.3666 (interquartile range 0.3652 to 0.3703).

A simulated process

The script simulated a Poisson process at rate 2 per unit time over 100,000 time units, for 30 seeds.

  • Counts in intervals of length 0.5, 1 and 3 had mean and variance both close to 1, 2 and 6: medians across seeds of 1.0003 and 1.0019, 2.0007 and 2.0031, 6.0020 and 6.0000.
  • The mean gap between events had median 0.4998, against $1/\lambda = 0.5$.
  • The share of unit intervals with no event had median 0.1349, against $e^{-2} = 0.1353$.

The second assumption, a constant rate, is the one that real data most often violate. Telephone calls, hospital admissions and deaths all have a daily, weekly or seasonal rhythm. A process whose rate changes over time in a known way is still Poisson in each interval, with mean equal to the integral of the rate over the interval. The trouble comes when the changes are unknown and the counts are pooled as if the rate were constant. The real data below show both cases.

The horse kicks

Back to Bortkiewicz's soldiers. Does a single Poisson distribution, fitted from the mean alone, reproduce his counts?

Deaths from horse kicks per army corps per year in the Prussian army, 1875 to 1894 (Bortkiewicz, 1898), against the Poisson distribution with the observed mean. For the ten corps of standard composition the fit is nearly exact. Notice that for all fourteen corps the model has slightly too few years with no deaths and with three or more, the pattern that variation in the rate between corps produces.
Deaths from horse kicks per army corps per year in the Prussian army, 1875 to 1894 (Bortkiewicz, 1898), against the Poisson distribution with the observed mean. For the ten corps of standard composition the fit is nearly exact. Notice that for all fourteen corps the model has slightly too few years with no deaths and with three or more, the pattern that variation in the rate between corps produces.

Bortkiewicz gave two tables (1898, §12, pp. 23–25). For all 14 corps over the 20 years, 280 corps-years with 196 deaths, the counts were 144, 91, 32, 11 and 2 for 0 to 4 deaths. He then set aside four corps whose composition, he noted, differed considerably from the norm:

  • the Guard corps, with 134 infantry companies and 40 cavalry squadrons;
  • the XI corps, with three divisions;
  • the I and VI corps, with 30 and 26 cavalry squadrons against a norm of 20.

He treated the other ten as a single series of 200 corps-years.

For the ten corps, the 122 deaths in 200 corps-years give a mean of $\bar{x} = 0.61$. The Poisson probabilities with $\lambda = 0.61$, multiplied by 200, give the expected numbers of corps-years:

Deaths 0 1 2 3 4 5 or more
Observed 109 65 22 3 1 0
Expected, Poisson(0.61) 108.7 66.3 20.2 4.1 0.6 0.1

The expected values agree to the decimal with those Bortkiewicz printed and with Fisher's (1925; 2nd ed. 1928, §15, Table 4). The sample variance of the 200 counts, with divisor $n - 1$, is 0.6110, against the mean 0.6100: a ratio of 1.0016.

Check your understanding. Suppose the fourteen corps had different rates of fatal kicks, and their counts were pooled as if they shared one rate. Compared with a single Poisson distribution with the pooled mean, would the pooled counts show more or fewer years with no deaths?

More, and more years with many deaths too. Low-rate corps add zeros and high-rate corps add large counts, so the pooled counts are more spread out than a Poisson with the same mean. The last section proves this.

For all fourteen corps, the data show exactly that pattern. The mean is 0.70 and the variance 0.7627, a ratio of 1.090. The expected counts under a single Poisson with mean 0.70 are 139.0, 97.3, 34.1, 7.9, 1.4 and 0.2, against observed counts of 144, 91, 32, 11 and 2. There are slightly more years with no deaths and with three or more than the model predicts. That is the signature of a rate that varies between the units being pooled, and it fits Bortkiewicz's reason for setting four corps aside.

Bortkiewicz's own expected values

His own expected values for the fourteen-corps table (143.1, 92.1, 33.8, 8.9, 2.0 and 0.6) differ from ours. He says he treated it exactly as he had his two preceding examples. In those, each unit (a state, an industrial accident insurance association) received a Poisson distribution with its own mean, and the expected frequencies were added across units. His table of counts by corps and year is hard to read in the scan. A transcription of it (Siegrist, after Andrews and Herzberg, 1985) gives exactly his two frequency tables, so we can repeat that calculation. With each corps given its own mean over the 20 years, the expected numbers are 143.5, 91.7, 33.2, 9.0, 2.1 and 0.5. These are much closer to his printed values than the single-mean values are, though not identical. His method was probably of this kind; we cannot account for the remaining differences.

Fisher's index of dispersion

How large a ratio of variance to mean is too large for a Poisson model? Fisher (1925; 2nd ed. 1928, §16) gave the test still used. For counts $x_1, \ldots, x_n$ with mean $\bar{x}$, the index of dispersion is

$$\chi^2 = \frac{\sum_{i=1}^{n}(x_i - \bar{x})^2}{\bar{x}} = \frac{(n - 1)s^2}{\bar{x}}.$$

For true Poisson samples it is distributed approximately as chi-square with $n - 1$ degrees of freedom. The reasoning: if the counts are Poisson, $\bar{x}$ estimates the variance as well as the mean. The statistic is then the familiar sum of squared deviations divided by the variance. That sum has the chi-square distribution for normal data, as Confidence Intervals for a Mean shows, and approximately so for Poisson counts that are not too small.

  • Values far above $n - 1$ indicate more variation than the Poisson allows (overdispersion).
  • Values far below it indicate less (underdispersion), as in a binomial with a large $p$ or counts more regular than chance.

For the ten corps, $\chi^2 = 199.31$ on 199 degrees of freedom, with an upper-tail $p$-value of 0.48: no evidence against the Poisson. For all fourteen, $\chi^2 = 304.00$ on 279 degrees of freedom, $p = 0.15$. The excess variation is visible but not, with 280 small counts, statistically clear.

Does the chi-square approximation hold for the larger counts used next? The script drew 2,000 samples of 313 independent Poisson counts with mean 122.67, for each of 30 seeds. The test at the 5% level rejected in a share of samples with median 0.0495 (interquartile range 0.0475 to 0.0534), as it should.

Overdispersion in weekly deaths

Do modern vital statistics behave as Poisson counts too? The National Center for Health Statistics publishes the number of deaths in each state in each week, by the week in which they occurred, for 2014 to 2019 (National Center for Health Statistics, 2021). Iowa recorded a mean of 570.52 deaths a week from all causes over the 313 weeks, and 122.67 a week from cancer (malignant neoplasms, ICD-10 codes C00 to C97).

Weekly deaths in Iowa, 2014 to 2019, from all causes (top) and from cancer (bottom). The shaded band is the mean plus or minus two Poisson standard deviations, twice the square root of the mean. Notice that cancer deaths scatter within the band as a Poisson count would, while all-cause deaths rise every winter, with peaks that differ from year to year, and spend long stretches outside the band.
Weekly deaths in Iowa, 2014 to 2019, from all causes (top) and from cancer (bottom). The shaded band is the mean plus or minus two Poisson standard deviations, twice the square root of the mean. Notice that cancer deaths scatter within the band as a Poisson count would, while all-cause deaths rise every winter, with peaks that differ from year to year, and spend long stretches outside the band.
Iowa, 313 weeks Mean Variance Variance/mean $\chi^2$ (312 df) $p$
Cancer deaths 122.67 117.49 0.958 298.8 0.69
All-cause deaths 570.52 2,196.66 3.850 1,201.3 < 0.001

Cancer deaths behave as a Poisson count:

  • their variance is within a few percent of the mean, and the index of dispersion is unremarkable;
  • the weekly counts, from 97 to 160, stay almost entirely within two Poisson standard deviations of the mean;
  • the yearly means barely move (from 120.9 to 124.2), and there is little seasonal pattern.

Each week, a large population faces a small and nearly constant probability of death from cancer. That is the setting of the binomial limit.

All-cause deaths do not behave as a Poisson count. Their variance is 3.85 times their mean, and the index of dispersion, 1,201 on 312 degrees of freedom, rules out the Poisson model beyond doubt. The figure shows why. Deaths from all causes rise every winter, the season of influenza and other respiratory infections, and the winter peaks differ in height from year to year. The rate is not constant, which breaks the second assumption of the Poisson process. The counts were also drifting upward, from a yearly mean of 556.3 in 2014 to 587.5 in 2019.

Removing the average seasonal pattern takes away part of the excess. Subtract from each week the mean for that week number over the six years. The residual variance is 1,000.55 (on 260 degrees of freedom), a ratio of 1.754 to the mean. The summer weeks alone (weeks 23 to 35, 78 weeks) have a ratio of 1.566. The remaining excess comes from the year-to-year differences in the winter peaks, the upward trend, and shorter-lived changes in the rate. A fixed seasonal pattern captures none of these.

Why a varying rate inflates the variance

Here is the idea in one sentence. A count whose rate wanders varies for two reasons, the Poisson chance at a given rate and the wandering of the rate itself, and the two variances add.

Suppose the count in each week is Poisson given that week's rate $\Lambda$, but the rate itself varies from week to week, with mean $\mu$ and variance $\operatorname{Var}(\Lambda)$. The law of total variance, used for clusters in The Binomial Distribution, gives

$$\operatorname{Var}(X) = E[\operatorname{Var}(X \mid \Lambda)] + \operatorname{Var}(E[X \mid \Lambda]) = E[\Lambda] + \operatorname{Var}(\Lambda) = \mu + \operatorname{Var}(\Lambda).$$

So the excess of the variance over the mean measures the variance of the underlying rate. For Iowa's all-cause deaths, $\operatorname{Var}(\Lambda) = 2{,}196.66 - 570.52 = 1{,}626.1$, a standard deviation of 40.3 deaths a week, or 7.1% of the mean.

Why does a rate that varies by only 7% produce a variance nearly four times the Poisson value? The Poisson part of the variance grows like $\mu$, while the part due to the varying rate grows like $\mu^2$. Writing $c$ for the coefficient of variation of the rate, the ratio of variance to mean is $1 + c^2\mu$. For Iowa, $1 + 0.0707^2 \times 570.52$ recovers the observed 3.850.

That relation predicts that larger states have larger ratios. Of the 54 jurisdictions in the file, 51 have no suppressed weeks for these two causes. (Alaska and Wyoming have a few weeks of cancer deaths withheld because they were between 1 and 9, and the national total is excluded.) The 51 are 48 states, the District of Columbia, New York City, which the file reports apart from the rest of New York State, and Puerto Rico.

Variance against mean of weekly deaths, 2014 to 2019, for 51 states and other reporting areas, on logarithmic scales. Cancer deaths (circles) follow the dashed line where variance equals mean, as a Poisson count would, drifting above it in a few areas, mostly the largest. Notice that all-cause deaths (squares) lie above the line everywhere, and further above it the larger the state.
Variance against mean of weekly deaths, 2014 to 2019, for 51 states and other reporting areas, on logarithmic scales. Cancer deaths (circles) follow the dashed line where variance equals mean, as a Poisson count would, drifting above it in a few areas, mostly the largest. Notice that all-cause deaths (squares) lie above the line everywhere, and further above it the larger the state.

For cancer, the ratio of variance to mean has a median of 1.090 across the 51 areas (range 0.807 to 1.829). The largest ratios belong to Florida, Texas, Arizona and California. In each of them except California, weekly cancer deaths rose markedly over the six years; in Texas, from a yearly mean of 755.0 in 2014 to 806.4 in 2019. Removing a straight-line trend lowers the ratio:

  • for Florida, from 1.829 to 1.631;
  • for Arizona, from 1.524 to 1.393;
  • for Texas, from 1.656 to 1.236;
  • for California, only from 1.481 to 1.453.

A slow change in the rate is part of the excess, magnified, as the formula above says, by the size of the mean, but not all of it.

For all-cause deaths the median ratio is 4.728. It ranges from 1.432 to 35.58 in California, which averaged 5,027.7 deaths a week. Since the ratio is $1 + c^2\mu$, a seasonal swing of similar relative size (a coefficient of variation $c$ of 7.1% in Iowa) makes Iowa's variance four times its mean and California's thirty-six times.

Models for overdispersed counts

Methods that assume Poisson variance understate standard errors, and the confidence intervals for rates from Rates in Public Health become too narrow. Three remedies are common:

  1. Model the variation in the rate directly, for example by giving each week of the year its own expected value. This is how excess deaths during an epidemic are measured.
  2. Scale the Poisson variance up by an estimated factor, the ratio of variance to mean.
  3. Use a distribution with an extra parameter.

The best known distribution of the third kind is the negative binomial. It is the distribution of a Poisson count whose mean varies according to a gamma distribution with shape $k$. Its variance is $\mu + \mu^2/k$, a special case of the formula above with $\operatorname{Var}(\Lambda) = \mu^2/k$.

The script checked this by simulation. It drew a gamma rate with shape 2 and mean 10 and then a Poisson count with that rate, 200,000 times for each of 30 seeds. The variance had median 60.07, against $10 + 100/2 = 60$. The probability of exactly 5 events had median 0.0669, against 0.0670 from the negative binomial and 0.0378 from the Poisson with the same mean. For Iowa's all-cause deaths, $k = \mu^2/\operatorname{Var}(\Lambda) = 200.2$.

The information retrieval wiki's 2-Poisson Model meets the same problem in a different setting. The number of times a word occurs in a document is badly overdispersed relative to a single Poisson, because documents differ in how much they are about the word. A mixture of two Poisson distributions is one answer.

The normal approximation

For large $\lambda$, the Poisson distribution is close to normal with mean $\lambda$ and standard deviation $\sqrt{\lambda}$. The reason: a $\text{Poisson}(\lambda)$ count is the sum of many independent Poisson counts with small means, and the central limit theorem applies to the sum. The skewness $1/\sqrt{\lambda}$ measures how far from normal it still is.

Check your understanding. The Poisson distribution is skewed to the right, and the normal curve is symmetric. Two standard deviations from the mean, which tail will the normal approximation underestimate?

The upper tail. The script compared exact tail probabilities two standard deviations from the mean with the continuity-corrected normal approximation:

$\lambda$ Lower tail Exact Normal Upper tail Exact Normal
4 $X \le 0$ 0.0183 0.0401 $X \ge 8$ 0.0511 0.0401
25 $X \le 15$ 0.0223 0.0287 $X \ge 35$ 0.0338 0.0287
122.67 $X \le 100$ 0.0201 0.0227 $X \ge 145$ 0.0267 0.0244

The normal curve, being symmetric, puts equal probability in both tails. The Poisson puts more in the upper tail at every $\lambda$; even at $\lambda = 122.67$ the two tails differ by a third. This is why the exact Poisson limits of Rates in Public Health, built from the gamma distribution, are preferred for counts below about 100, and they are what NCHS uses.

Corrections to the course notes

The unit's notes give the Poisson pmf, the equality of mean and variance, the limit from the binomial, the assumptions of the process, additivity, thinning and time scaling; the limit is proved above.

  • They describe $\lambda$ as the average rate of events per interval, and caution against reading it as a fixed count. More precisely, $\lambda$ is the expected count for the particular interval (or population and period) being counted. It is a rate per unit time multiplied by the length of the interval, so the same rate gives different Poisson distributions for different intervals.
  • The notes' normal approximation for $\lambda$ above about 10 is written $N(\lambda, \sqrt{\lambda})$, with the standard deviation as the second argument as in the course's formula sheet. It is accurate in the center of the distribution. But, as the last table shows, it leaves the tails unequal well beyond $\lambda = 10$.
  • The notes list overdispersion, time-varying rates and clustering as violations, with the negative binomial as a remedy, and connect overdispersion with events that are not independent. Dependence is one cause. A rate that varies, with the events still independent given the rate, is another, and it is the one the weekly deaths above show in real data.

Summary

Quantity Formula
pmf $P(X = k) = e^{-\lambda}\lambda^k/k!$, $k = 0, 1, 2, \ldots$
Mean and variance both $\lambda$; standard deviation $\sqrt{\lambda}$
Binomial limit $\text{Bin}(n, \lambda/n) \to \text{Poisson}(\lambda)$ as $n \to \infty$
Sums independent $\text{Poisson}(\lambda) + \text{Poisson}(\mu) = \text{Poisson}(\lambda + \mu)$
Poisson process $N(t) \sim \text{Poisson}(\lambda t)$; waiting time $P(T > t) = e^{-\lambda t}$, mean $1/\lambda$
Index of dispersion $\sum (x_i - \bar{x})^2/\bar{x} \approx \chi^2_{n-1}$ under the Poisson
Varying rate $\operatorname{Var}(X) = \mu + \operatorname{Var}(\Lambda)$; ratio $1 + c^2\mu$
Negative binomial variance $\mu + \mu^2/k$

What we have learned

Counts of rare events in a large population follow the Poisson distribution, the limit of the binomial with $np = \lambda$ fixed. Its mean and variance are both $\lambda$, and independent Poisson counts add. Events at a constant rate in time give the same distribution, with memoryless exponential waiting times. Bortkiewicz's ten corps fit it from their mean of 0.61 alone, and so do weekly cancer deaths in Iowa. All-cause deaths do not: a rate that swings by about 7% makes their variance 3.85 times their mean, and the excess grows with the size of the state.

Heights, blood pressures and the means of samples are not whole-number counts. How are probabilities assigned to measurements that can take any value in an interval, and why does one bell-shaped curve describe so many of them? That is The Normal Distribution.

In Python

from scipy import stats
import numpy as np

kicks = np.repeat([0, 1, 2, 3, 4], [109, 65, 22, 3, 1])   # Bortkiewicz, ten corps
lam = kicks.mean()                                          # 0.61
print(200 * stats.poisson.pmf(range(5), lam))               # 108.7, 66.3, 20.2, 4.1, 0.6
chi2 = ((kicks - lam) ** 2).sum() / lam                     # index of dispersion: 199.31
print(chi2, stats.chi2.sf(chi2, len(kicks) - 1))            # upper-tail p about 0.48

Every number in this article is computed by tools/experiments/applied-statistics/poisson-distribution.py in the site's repository, with the horse-kick tables and the NCHS weekly death counts loaded by distdata.py beside it.

References