The Chi-Square Goodness-of-Fit Test

Whether counts in several categories agree with proposed probabilities. The multinomial model and the covariance of its counts; Pearson's statistic shown to be a squared Mahalanobis distance and a score statistic; its chi-squared limit with k - 1 degrees of freedom proved for every number of categories by a projection; the likelihood ratio statistic shown to differ from it by an amount that vanishes; the noncentral chi-squared limit that gives the power; and the loss of one degree of freedom for each parameter estimated from the counts, proved for one parameter and, given the several-parameter linear representation of the estimate, for any number. Worked on self-rated health among older adults in NHANES and on Bortkiewicz's horse kicks. Worked problems on the covariance of the counts, an exact multinomial test, a fully specified Poisson model and the power of a two-category test.

Of 200 adults aged 60 and over drawn at random from the adults examined in the National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018, 7 described their health in general as excellent, 46 as very good, 88 as good, 50 as fair and 9 as poor. Among all 5,242 adults of the survey who answered the question, 11.50% chose excellent, so a sample of 200 drawn from all of them would be expected to hold about 23 people who did. Is the older sample further from the shares of all adults than chance alone would put it? It is: the test of this article gives $p = 0.0030$.

A goodness-of-fit test asks whether counts in $k$ categories are consistent with a proposed set of category probabilities. Karl Pearson (1900) gave the statistic that is still used, the sum over the categories of squared differences between observed and expected counts, each divided by the expected count, and he derived its distribution. The test of a $2 \times 2$ table in Comparing Two Proportions is one use of the same statistic; this article treats the statistic itself, for any number of categories.

First we set out the multinomial model and the covariance of its counts. Then we define Pearson's statistic, show that it is the squared distance of the counts from their expected values in the metric of their own covariance, and prove its chi-squared limit, first for two categories and then for any number by a projection. Next come the likelihood ratio statistic, the question of how large a sample the limit needs, and the power of the test. Finally we show what changes when the probabilities contain a parameter estimated from the data. The article assumes the chi-squared distribution of The Chi-Squared and t Distributions, the projection theorem of Sampling Distributions from the Normal and the central limit theorem of Convergence in Distribution and the Proof of the Central Limit Theorem.

This article is the home of the unit's facts about multinomial counts and of Pearson's theorem. The Chi-Square Test of Independence and Trends in Proportions: the Cochran–Armitage Test build on them.

The multinomial model

Each of $n$ observations falls into exactly one of $k$ categories, independently of the others, with probability $\pi_j$ for category $j$ and $\pi_1 + \cdots + \pi_k = 1$. The counts $O_1, \ldots, O_k$ record how many fall in each category, so $O_1 + \cdots + O_k = n$. The vector of counts has the multinomial distribution,

$$P(O_1 = o_1, \ldots, O_k = o_k) = \frac{n!}{o_1!\cdots o_k!}\,\pi_1^{o_1}\cdots\pi_k^{o_k}, \qquad o_1 + \cdots + o_k = n.$$

The coefficient counts the orderings of the $n$ observations that put $o_j$ of them in category $j$, and each ordering has the probability given by the product. With $k = 2$ this is the binomial distribution.

The moments follow from writing the counts as sums. Let $V_i = (V_{i1}, \ldots, V_{ik})$ be the indicator vector of observation $i$: $V_{ij} = 1$ if it falls in category $j$ and 0 otherwise. Then $O_j = \sum_{i=1}^n V_{ij}$, and the vectors $V_1, \ldots, V_n$ are independent and identically distributed (i.i.d.).

Theorem 1 (moments of the counts). $E[O_j] = n\pi_j$, $\operatorname{Var}(O_j) = n\pi_j(1 - \pi_j)$, and for $j \ne l$, $\operatorname{Cov}(O_j, O_l) = -n\pi_j\pi_l$.

Proof. $V_{ij}$ is an indicator with mean $\pi_j$ and variance $\pi_j(1 - \pi_j)$. For $j \ne l$, an observation cannot fall in both categories, so $V_{ij}V_{il} = 0$ and $\operatorname{Cov}(V_{ij}, V_{il}) = E[V_{ij}V_{il}] - \pi_j\pi_l = -\pi_j\pi_l$. Covariances of sums of independent terms add over the terms, and terms from different observations have covariance 0 (Expectation, Variance and Covariance, the variance of a linear combination). Summing over $i$ gives the three formulas. $\square$

In matrix form the covariance matrix of $(O_1, \ldots, O_k)$ is $n(\Delta_\pi - \boldsymbol{\pi}\boldsymbol{\pi}^{\mathsf T})$, where $\Delta_\pi$ is the diagonal matrix of the $\pi_j$ (written $\Delta$ rather than $D$, which in this unit is the Kullback–Leibler divergence). The negative covariances come from the fixed total: a count that runs high leaves less room for the others. The matrix is singular, since the counts always add to $n$ and so their sum has variance 0.

Example (the sample was drawn without replacement). The 200 older adults were drawn without replacement from the 1,923 adults aged 60 and over, so their counts are not exactly multinomial: each draw changes the shares left for the next. Sampling without replacement multiplies every variance and covariance by $(N - n)/(N - 1)$ for a population of $N$, here $1{,}723/1{,}922$, a little under 0.9 (the finite population correction of The Sampling Distribution of the Mean). The multinomial model, which the rest of the article uses as the course pages do, therefore slightly overstates the variability, and a test built on it is slightly conservative for this sample.

Pearson's statistic

To test the null hypothesis $H_0\colon \pi_j = \pi_{0j}$ for every $j$, with the $\pi_{0j}$ given numbers, compare each count with its expected count under $H_0$, $E_j = n\pi_{0j}$. Pearson's statistic is

$$X^2 = \sum_{j=1}^{k}\frac{(O_j - E_j)^2}{E_j}.$$

It is zero when every count equals its expectation and grows with every departure. Large values count against $H_0$. For the older adults the null shares are those of all 5,242 adults:

Health $O_j$ $\pi_{0j}$ $E_j$ $(O_j - E_j)^2/E_j$ $(O_j - E_j)/\sqrt{E_j}$
Excellent 7 0.1150 23.01 11.136 −3.34
Very good 46 0.2625 52.50 0.805 −0.90
Good 88 0.3737 74.74 2.352 1.53
Fair 50 0.2089 41.78 1.618 1.27
Poor 9 0.0399 7.97 0.132 0.36
Total 200 1 200 $X^2 = 16.043$

The shares are the counts 603, 1,376, 1,959, 1,095 and 209 of all adults divided by 5,242. Two thirds of the statistic comes from one cell: 7 older adults rated their health excellent where 23 were expected. The last column, the Pearson residual $(O_j - E_j)/\sqrt{E_j}$, is the signed square root of each contribution and shows the direction: fewer older adults than expected in the two best categories, more in the three others. The section after next shows that, if $H_0$ were true, $X^2$ would have approximately the chi-squared distribution with $k - 1 = 4$ degrees of freedom. The 95th percentile of that distribution is 9.488, and $P(\chi^2_4 \ge 16.043) = 0.0030$.

Why divide by the expected count

The division puts every cell on the scale of its own variability. A deviation of 10 is large for a cell expecting 8 and small for one expecting 75. But $\operatorname{Var}(O_j) = n\pi_j(1 - \pi_j)$, not $n\pi_j$, and the counts are correlated, so the scaling is not obviously right. Here is the idea of why it is exactly right. The natural distance of a random vector from its mean is measured in the metric of its covariance matrix, $\mathbf{d}^{\mathsf T}\Sigma^{-1}\mathbf{d}$ (a Mahalanobis distance). The $k$ deviations have a singular covariance, but any $k - 1$ of them determine the last, and their distance in their own metric turns out to be $X^2$. That is the route Pearson (1900, equations (vii) to (xv)) took: he wrote down the standard deviations and correlations of the deviations $e_p = m'_p - m_p$, evaluated the determinant of their correlations and its minors, which give the coefficients of the quadratic form, and arrived at $\chi^2 = S(e^2/m)$, the sum over all the groups.

Theorem 2 (Pearson's statistic as a distance). Let $\mathbf{d} = (O_1 - n\pi_1, \ldots, O_{k-1} - n\pi_{k-1})$ be the first $k - 1$ deviations, with all $\pi_j > 0$, and let $\Sigma = n(\Delta - \boldsymbol{\pi}\boldsymbol{\pi}^{\mathsf T})$ be their covariance matrix, where now $\Delta$ and $\boldsymbol{\pi}$ involve only $\pi_1, \ldots, \pi_{k-1}$. Then $\Sigma$ is invertible,

$$\Sigma^{-1} = \frac{1}{n}\Big(\Delta^{-1} + \frac{1}{\pi_k}\mathbf{1}\mathbf{1}^{\mathsf T}\Big), \qquad \text{and} \qquad \mathbf{d}^{\mathsf T}\Sigma^{-1}\mathbf{d} = \sum_{j=1}^{k}\frac{(O_j - n\pi_j)^2}{n\pi_j}.$$

Proof. Multiply, using $\Delta\mathbf{1} = \boldsymbol{\pi}$, $\boldsymbol{\pi}^{\mathsf T}\Delta^{-1} = \mathbf{1}^{\mathsf T}$ and $\boldsymbol{\pi}^{\mathsf T}\mathbf{1} = 1 - \pi_k$:

$$(\Delta - \boldsymbol{\pi}\boldsymbol{\pi}^{\mathsf T})\Big(\Delta^{-1} + \frac{\mathbf{1}\mathbf{1}^{\mathsf T}}{\pi_k}\Big) = I + \frac{\boldsymbol{\pi}\mathbf{1}^{\mathsf T}}{\pi_k} - \boldsymbol{\pi}\mathbf{1}^{\mathsf T} - \frac{(1 - \pi_k)\,\boldsymbol{\pi}\mathbf{1}^{\mathsf T}}{\pi_k} = I + \boldsymbol{\pi}\mathbf{1}^{\mathsf T}\Big(\frac{1}{\pi_k} - 1 - \frac{1 - \pi_k}{\pi_k}\Big) = I.$$

So the inverse is as stated. Then

$$\mathbf{d}^{\mathsf T}\Sigma^{-1}\mathbf{d} = \frac{1}{n}\sum_{j=1}^{k-1}\frac{d_j^2}{\pi_j} + \frac{1}{n\pi_k}\Big(\sum_{j=1}^{k-1}d_j\Big)^2.$$

The deviations of all $k$ counts sum to $n - n = 0$, so $\sum_{j<k} d_j = -(O_k - n\pi_k)$, and the last term is $(O_k - n\pi_k)^2/(n\pi_k)$. $\square$

The same matrix gives a second reading. The multinomial log-likelihood, with $\pi_k = 1 - \pi_1 - \cdots - \pi_{k-1}$ and the first $k - 1$ probabilities as parameters, is $\ell = \sum_j O_j\ln\pi_j$ plus a constant. Its score has components $U_j = O_j/\pi_j - O_k/\pi_k$, which, since $\sum_j O_j = n = n\sum_j \pi_j$, equal $d_j/\pi_j - (O_k - n\pi_k)/\pi_k = d_j/\pi_j + \sum_{l<k}d_l/\pi_k$. In matrix form $\mathbf{U} = (\Delta^{-1} + \mathbf{1}\mathbf{1}^{\mathsf T}/\pi_k)\,\mathbf{d} = n\Sigma^{-1}\mathbf{d}$. The Fisher information is the covariance matrix of the score (Fisher Information and the Cramér–Rao Bound proves the one-parameter identity; here the covariance is computed directly), $n\Sigma^{-1}\,\Sigma\,n\Sigma^{-1} = n^2\Sigma^{-1}$. The score statistic is therefore

$$\mathbf{U}^{\mathsf T}(n^2\Sigma^{-1})^{-1}\mathbf{U} = \mathbf{d}^{\mathsf T}\Sigma^{-1}\mathbf{d} = X^2.$$

Pearson's statistic is the score statistic of the multinomial model. Likelihood Ratio, Wald and Score Tests showed the two-proportion case, where the score statistic is the pooled $z^2$.

A third fact needs no limit at all. Under $H_0$, $E[(O_j - E_j)^2] = \operatorname{Var}(O_j) = n\pi_{0j}(1 - \pi_{0j})$, so

$$E[X^2] = \sum_{j=1}^{k}(1 - \pi_{0j}) = k - 1$$

exactly, at every sample size. The mean of the limiting distribution, $\chi^2_{k-1}$, is $k - 1$ (The Chi-Squared and t Distributions).

Two categories

With two categories the test is the one-proportion test of the course. Write $O_1 = x$ for the count in the first category, $O_2 = n - x$ and $\pi_{01} = p_0$. The two deviations are equal and opposite, $O_2 - E_2 = -(x - np_0)$, so

$$X^2 = (x - np_0)^2\Big(\frac{1}{np_0} + \frac{1}{n(1 - p_0)}\Big) = \frac{(x - np_0)^2}{np_0(1 - p_0)} = z^2,$$

where $z = (x - np_0)/\sqrt{np_0(1 - p_0)}$ is the statistic of the one-sample test in Tests and Sample Size for a Proportion. The limit is then a one-line consequence of the central limit theorem: under $H_0$, $z \xrightarrow{d} Z$ standard normal (The Normal Approximation to the Binomial; it is the theorem of the convergence article applied to the indicators $V_{i1}$), and by the continuous mapping theorem $X^2 = z^2 \xrightarrow{d} Z^2$, which is $\chi^2_1$ by definition.

Example (fewer categories, a different question). Merge the older adults' answers into "fair or poor" and "better". The sample has $x = 59$ in fair or poor, against $p_0 = 0.2488$ among all adults and an expected 49.75. Then $z = 1.513$, $X^2 = 2.288$ and $p = 0.1304$. The two-category test does not reject, although the five-category test did with $p = 0.0030$. Nothing is wrong with either. The merged test asks about one contrast, and the large departure in the sample lies elsewhere, in the excellent category, which the merge folds into "better" together with the good answers, 13 above their expectation, which offset most of the shortfall of 16 in excellent and 6.5 in very good. A goodness-of-fit test answers the question its categories ask. The Cochran–Armitage article returns to the price of spreading a test over many directions.

Pearson's theorem for any number of categories

Here is the idea of the general proof. Scale each deviation by the square root of its expected count. The covariance matrix of the scaled deviations is the identity minus the outer product of one unit vector with itself: a projection onto a subspace of dimension $k - 1$. By the central limit theorem the scaled deviations behave in large samples like that projection applied to $k$ independent standard normal variables, and the squared length of such a projection is chi-squared with $k - 1$ degrees of freedom. $X^2$ is the squared length of the scaled deviations.

Lemma 1 (the scaled deviations). Let all $\pi_j > 0$, let $Z_j = (O_j - n\pi_j)/\sqrt{n\pi_j}$, and let $\mathbf{u} = (\sqrt{\pi_1}, \ldots, \sqrt{\pi_k})$. Then $\mathbf{u}$ is a unit vector, $\mathbf{Z} = (Z_1, \ldots, Z_k)$ has covariance matrix $P = I - \mathbf{u}\mathbf{u}^{\mathsf T}$, and $P$ is a projection matrix of rank $k - 1$.

Proof. $\lVert\mathbf{u}\rVert^2 = \sum_j \pi_j = 1$. By Theorem 1, $\operatorname{Cov}(Z_j, Z_l) = \operatorname{Cov}(O_j, O_l)/(n\sqrt{\pi_j\pi_l})$, which is $1 - \pi_j$ for $j = l$ and $-\sqrt{\pi_j\pi_l}$ otherwise: the entries of $I - \mathbf{u}\mathbf{u}^{\mathsf T}$. That matrix is symmetric, and $P^2 = I - 2\mathbf{u}\mathbf{u}^{\mathsf T} + \mathbf{u}(\mathbf{u}^{\mathsf T}\mathbf{u})\mathbf{u}^{\mathsf T} = P$. Its trace is $k - \lVert\mathbf{u}\rVert^2 = k - 1$, and the trace of a projection matrix is its rank (Lemma 2 of Sampling Distributions from the Normal). $\square$

$P$ removes the component along $\mathbf{u}$. The scaled deviations have none: $\mathbf{u}\cdot\mathbf{Z} = \sum_j (O_j - n\pi_j)/\sqrt{n} = 0$ for every sample, which is the fixed total again.

Theorem 3 (Pearson). If $H_0\colon \pi_j = \pi_{0j}$ holds, with every $\pi_{0j} > 0$, then as $n \to \infty$

$$X^2 = \sum_{j=1}^{k}\frac{(O_j - n\pi_{0j})^2}{n\pi_{0j}} \xrightarrow{d} \chi^2_{k-1}.$$

The proof uses one result that is cited, not proved on this site: the Cramér–Wold device, that random vectors $\mathbf{X}_n$ converge in distribution to $\mathbf{X}$ if $\mathbf{a}\cdot\mathbf{X}_n \xrightarrow{d} \mathbf{a}\cdot\mathbf{X}$ for every fixed vector $\mathbf{a}$ (Durrett, 2019, Theorem 3.10.6). Convergence in distribution of random vectors means $E[h(\mathbf{X}_n)] \to E[h(\mathbf{X})]$ for every bounded continuous function $h$ of $k$ variables, the extension that the convergence article gives in its section on bounded continuous functions.

Proof. With $\pi = \pi_0$, the vector $\mathbf{Z}$ of Lemma 1 is $\mathbf{Z}_n = \sqrt{n}\,\Delta_\pi^{-1/2}(\bar{V}_n - \boldsymbol{\pi})$, where $\bar{V}_n$ is the mean of the indicator vectors, and $X^2 = \lVert\mathbf{Z}_n\rVert^2$.

Step 1 (linear combinations). Fix a vector $\mathbf{a}$. The scalars $\mathbf{a}\cdot \Delta_\pi^{-1/2}V_i$ are i.i.d. with variance $\mathbf{a}^{\mathsf T}P\mathbf{a}$, by Lemma 1. By Corollary 3 of the convergence article (linear combinations of a vector mean), $\mathbf{a}\cdot\mathbf{Z}_n$ converges in distribution to the normal distribution with mean 0 and variance $\mathbf{a}^{\mathsf T}P\mathbf{a}$.

Step 2 (the limit vector). Let $\mathbf{Y} = (Y_1, \ldots, Y_k)$ be independent standard normal variables and $\mathbf{W} = P\mathbf{Y}$. Then $\mathbf{a}\cdot\mathbf{W} = (P\mathbf{a})\cdot\mathbf{Y}$, since $P$ is symmetric. A fixed linear combination $\mathbf{b}\cdot\mathbf{Y}$ of independent standard normal variables is normal with mean 0 and variance $\lVert\mathbf{b}\rVert^2$: rotate so that the first axis points along $\mathbf{b}$, and $\mathbf{b}\cdot\mathbf{Y}$ becomes $\lVert\mathbf{b}\rVert$ times the first rotated coordinate, which is standard normal by Theorem 1 of Sampling Distributions from the Normal. Here $\lVert P\mathbf{a}\rVert^2 = \mathbf{a}^{\mathsf T}P^2\mathbf{a} = \mathbf{a}^{\mathsf T}P\mathbf{a}$. So $\mathbf{a}\cdot\mathbf{Z}_n \xrightarrow{d} \mathbf{a}\cdot\mathbf{W}$ for every $\mathbf{a}$, and by the Cramér–Wold device $\mathbf{Z}_n \xrightarrow{d} \mathbf{W}$.

Step 3 (the squared length). The function $g(\mathbf{z}) = \lVert\mathbf{z}\rVert^2$ is continuous. For a bounded continuous function $h$ of one variable, $h \circ g$ is a bounded continuous function of $k$ variables, so $E[h(g(\mathbf{Z}_n))] \to E[h(g(\mathbf{W}))]$. By the criterion by bounded continuous functions, $X^2 = g(\mathbf{Z}_n) \xrightarrow{d} g(\mathbf{W}) = \lVert P\mathbf{Y}\rVert^2$.

Step 4 (the distribution of the limit). $\mathbf{Y}$ has independent normal coordinates with variance 1 and mean vector $\mathbf{0}$, and $P$ is a projection matrix of rank $k - 1$. By Theorem 2 of Sampling Distributions from the Normal (projections of a normal vector, part 3), $\lVert P\mathbf{Y}\rVert^2$ is $\chi^2_{k-1}$. $\square$

The degrees of freedom are the dimension of the space in which the scaled deviations can move: $k$ coordinates less the one direction, $\mathbf{u}$, that the fixed total forbids. Pearson's paper (1900, section 3, p. 160) makes the same count in his terms, for $n + 1$ groups: "only $n$ of the $n + 1$ errors are variables", the last being determined when the others are known.

Example (the limit at the sample size of the running example). The script drew 10,000 multinomial samples of 200 with the null shares, for each of 30 seeds (5,100,000 + $s$), and applied the test at the 5% level with the cut-off 9.488. The share of samples rejected had median 0.0503 over the seeds (interquartile range 0.0490 to 0.0513), and the mean of $X^2$ had median 3.998, against the exact $k - 1 = 4$. At this sample size, for these shares, the chi-squared distribution with 4 degrees of freedom describes the statistic well.

The likelihood ratio statistic

The multinomial likelihood gives a second statistic. The log-likelihood is $\ell(\boldsymbol\pi) = \sum_j O_j\ln\pi_j$ plus a constant, and its maximum over all probability vectors is at the sample shares $\hat\pi_j = O_j/n$ (Cross-Entropy and Kullback–Leibler Divergence, section "Cross-entropy loss is the average negative log-likelihood", proves this by Gibbs' inequality). The likelihood ratio statistic of Likelihood Ratio, Wald and Score Tests is therefore

$$G^2 = 2[\ell(\hat{\boldsymbol\pi}) - \ell(\boldsymbol\pi_0)] = 2\sum_{j=1}^{k} O_j\ln\frac{O_j}{E_j} = 2n\,D(\hat{\boldsymbol\pi}\,\|\,\boldsymbol\pi_0),$$

with $0\ln 0 = 0$, where $D$ is the Kullback–Leibler divergence in nats. The statistic is the $G$ of Mutual Information, written $G^2$ in this unit. For the older adults, $D(\hat{\boldsymbol\pi}\,\|\,\boldsymbol\pi_0) = 0.0502$ nats and $G^2 = 20.067$, with $P(\chi^2_4 \ge 20.067) = 0.0005$.

The several-parameter form of Wilks's theorem, which that article states and cites, predicts the $\chi^2_{k-1}$ limit, since $H_0$ fixes all $k - 1$ free probabilities. Here it can be proved from Pearson's theorem, because the two statistics differ by an amount that vanishes. The idea: the divergence between two nearby distributions is half their chi-square distance, and under $H_0$ the sample shares come ever closer to the null shares.

Theorem 4 ($G^2$ and $X^2$ agree). Let $\varepsilon = \max_j \lvert O_j/E_j - 1\rvert$. Whenever $\varepsilon < 1$,

$$\frac{X^2}{1 + \varepsilon} \le G^2 \le \frac{X^2}{1 - \varepsilon}.$$

Under $H_0$, with every $\pi_{0j} > 0$, $G^2 - X^2 \xrightarrow{p} 0$, and so $G^2 \xrightarrow{d} \chi^2_{k-1}$.

Proof. Cross-Entropy and Kullback–Leibler Divergence, section "Small differences: chi-square and Fisher information", proves that if the relative errors $t_j = (p_j - q_j)/q_j$ satisfy $\lvert t_j\rvert \le \varepsilon < 1$, then $\chi^2(p, q)/(2(1 + \varepsilon)) \le D(p\,\|\,q) \le \chi^2(p, q)/(2(1 - \varepsilon))$, where $\chi^2(p, q) = \sum_j (p_j - q_j)^2/q_j$. Take $p = \hat{\boldsymbol\pi}$ and $q = \boldsymbol\pi_0$. Then $t_j = O_j/E_j - 1$, $n\,\chi^2(\hat{\boldsymbol\pi}, \boldsymbol\pi_0) = X^2$ and $2nD = G^2$; multiplying the bounds by $2n$ gives the display.

Under $H_0$, each $O_j/n \xrightarrow{p} \pi_{0j}$ by the weak law of large numbers (The Laws of Large Numbers), so each $\lvert O_j/E_j - 1\rvert \xrightarrow{p} 0$ and so does their maximum over the $k$ cells. On the event $\varepsilon \le \tfrac12$, whose probability tends to 1, the display gives

$$G^2 - X^2 \le X^2\Big(\frac{1}{1 - \varepsilon} - 1\Big) = X^2\,\frac{\varepsilon}{1 - \varepsilon} \le 2X^2\varepsilon, \qquad X^2 - G^2 \le X^2\Big(1 - \frac{1}{1 + \varepsilon}\Big) = X^2\,\frac{\varepsilon}{1 + \varepsilon} \le 2X^2\varepsilon,$$

so $\lvert G^2 - X^2\rvert \le 2X^2\varepsilon$. $X^2$ converges in distribution by Theorem 3, so it is bounded in probability (Lemma 2 of the convergence article), and $2X^2\varepsilon \xrightarrow{p} 0$ by Lemma 4 there (bounded times vanishing). Hence $G^2 - X^2 \xrightarrow{p} 0$, and $G^2 = X^2 + (G^2 - X^2) \xrightarrow{d} \chi^2_{k-1}$ by Lemma 3 there (converging together). $\square$

Example (the two statistics are not interchangeable in this sample). For the older adults $X^2 = 16.043$ and $G^2 = 20.067$, a gap of 4. The largest relative error is $\varepsilon = 0.696$, in the excellent cell, where 7 were observed against 23.01 expected, and the theorem's bounds are 9.46 and 52.73: true, and far apart. The theorem says the statistics agree when every cell is close to its expectation, which a sample with one cell 70% below its expectation is not. Both statistics reject $H_0$ here, so the disagreement does not change the conclusion. It does say that the $p$-values, 0.0030 and 0.0005, are less precise than their digits suggest.

How large a sample the limit needs

Theorem 3 is a statement about $n \to \infty$. For a finite sample the exact null distribution of $X^2$ can be found by listing every possible table, since a table of $n$ observations in $k$ categories is one of $\binom{n + k - 1}{k - 1}$ vectors of counts. The script did this for the null shares of the running example and computed the exact probability that each statistic exceeds the 5% cut-off 9.488:

$n$ Possible tables Smallest $E_j$ Exact size of $X^2$ test Exact size of $G^2$ test
10 1,001 0.40 0.0526 0.0595
20 10,626 0.80 0.0484 0.0587
40 135,751 1.59 0.0498 0.0573

These are observations for one set of null shares, not a general rule. Two things stand out. Pearson's statistic keeps close to its nominal level even with an expected count below 1, and the likelihood ratio statistic runs above it at each of these sizes, by about one percentage point. The simulation at $n = 200$ in the section on Pearson's theorem also recorded $G^2$: it rejected in a share with median 0.0524 (interquartile range 0.0511 to 0.0536), against 0.0503 for $X^2$. In this example, then, $X^2$ held its nominal level more closely at every size tried. Pearson's own paper warned against a different excess, too many groups. He showed that as the number of groups grows without limit, the probability he computes for a given value of $\chi$ tends to 1, and concluded: "if we take a very great number of groups our test becomes illusory" (Pearson, 1900, p. 160). Cochran's (1952) expository paper, by its abstract, reviews the research on the minimum expected number per class; we have not read its recommendations and do not quote one.

When a sample is small enough to list its tables, the approximation can be avoided altogether. The exact multinomial test sums the null probabilities of every table at least as extreme as the one observed. Problem 2 carries it out for 20 older adults and shows that "at least as extreme" needs a definition, which changes the answer.

Power and the noncentral chi-squared distribution

How likely was the test to detect the difference it found? In the NHANES population the 1,923 adults aged 60 and over have shares 0.0733, 0.2350, 0.3812, 0.2522 and 0.0582 in the five categories. The power of the test at a sample of 200 is the probability that $X^2$ exceeds 9.488 when the counts come from these shares. The answer is close to one half, and it follows from a limit theorem of the same kind as Theorem 3, in which the true shares approach the null shares as $n$ grows. Here is the idea: a fixed alternative makes $X^2$ grow without bound, so a useful approximation lets the alternative shrink at the rate $1/\sqrt{n}$, at which the scaled deviations acquire a fixed mean and the limit becomes a noncentral chi-squared distribution.

Theorem 5 (local alternatives). Let the true shares be $\pi_{nj} = \pi_{0j} + \delta_j/\sqrt{n}$, with fixed numbers $\delta_j$ summing to 0 and every $\pi_{0j} > 0$ (these are probabilities once $n$ is large enough that every $\pi_{nj}$ lies in $(0, 1)$, and the theorem concerns those $n$). Then $X^2$, computed with the null shares $\pi_{0j}$, converges in distribution to the noncentral chi-squared distribution with $k - 1$ degrees of freedom and noncentrality

$$\lambda = \sum_{j=1}^{k}\frac{\delta_j^2}{\pi_{0j}}.$$

The noncentral chi-squared distribution $\chi^2_r(\lambda)$ is defined, and shown to depend on the means only through the sum of their squares, in Sampling Distributions from the Normal (Lemma 1); its mean is $r + \lambda$. The proof uses a second cited theorem, the Lindeberg–Feller theorem: if for each $n$ the variables $X_{n,1}, \ldots, X_{n,n}$ are independent with mean 0, $\sum_i E[X_{n,i}^2] \to v > 0$, and for every $\eta > 0$ the sum $\sum_i E[X_{n,i}^2\mathbf{1}\{\lvert X_{n,i}\rvert > \eta\}]$ tends to 0, then $\sum_i X_{n,i}$ converges in distribution to the normal with mean 0 and variance $v$ (Durrett, 2019, Theorem 3.4.10). It is needed because the observations' distribution now changes with $n$.

Proof. Write $\Delta_0$ for the diagonal matrix of the $\pi_{0j}$, and split the scaled deviations as

$$\mathbf{Z}_n = \frac{\Delta_0^{-1/2}(\mathbf{O} - n\boldsymbol\pi_0)}{\sqrt{n}} = \frac{\Delta_0^{-1/2}(\mathbf{O} - n\boldsymbol\pi_n)}{\sqrt{n}} + \Delta_0^{-1/2}\boldsymbol\delta.$$

Fix $\mathbf{a}$ and let $X_{n,i} = \mathbf{a}\cdot \Delta_0^{-1/2}(V_i - \boldsymbol\pi_n)/\sqrt{n}$, so that the first term dotted with $\mathbf{a}$ is $\sum_i X_{n,i}$. These are independent with mean 0. Each is at most $c/\sqrt{n}$ in absolute value, with $c = 2\sum_j \lvert a_j\rvert/\sqrt{\pi_{0j}}$, because the entries of $V_i$ and $\boldsymbol\pi_n$ lie in $[0, 1]$. So for every $\eta > 0$ the Lindeberg sum is exactly 0 once $c/\sqrt{n} < \eta$. By Theorem 1 applied with the shares $\boldsymbol\pi_n$, $\sum_i E[X_{n,i}^2] = \mathbf{a}^{\mathsf T}\Delta_0^{-1/2}(\Delta_{\pi_n} - \boldsymbol\pi_n\boldsymbol\pi_n^{\mathsf T})\Delta_0^{-1/2}\mathbf{a}$, which tends to $\mathbf{a}^{\mathsf T}P\mathbf{a}$ as $\boldsymbol\pi_n \to \boldsymbol\pi_0$. If $\mathbf{a}^{\mathsf T}P\mathbf{a} > 0$, the Lindeberg–Feller theorem gives that $\sum_i X_{n,i}$ converges in distribution to the normal with variance $\mathbf{a}^{\mathsf T}P\mathbf{a}$. If it is 0, the variance of the sum tends to 0 and the sum tends to 0 in probability by Chebyshev's inequality, which is convergence in distribution to the normal with variance 0. Either way the limit is the law of $\mathbf{a}\cdot P\mathbf{Y}$ (Step 2 of Theorem 3). Adding the constant $\mathbf{a}\cdot\mathbf{m}$, with $\mathbf{m} = \Delta_0^{-1/2}\boldsymbol\delta$, gives $\mathbf{a}\cdot\mathbf{Z}_n \xrightarrow{d} \mathbf{a}\cdot(P\mathbf{Y} + \mathbf{m})$, and by the Cramér–Wold device $\mathbf{Z}_n \xrightarrow{d} P\mathbf{Y} + \mathbf{m}$.

The mean vector lies in the subspace onto which $P$ projects: $\mathbf{u}\cdot\mathbf{m} = \sum_j \sqrt{\pi_{0j}}\,\delta_j/\sqrt{\pi_{0j}} = \sum_j \delta_j = 0$, so $P\mathbf{m} = \mathbf{m}$ and $P\mathbf{Y} + \mathbf{m} = P(\mathbf{Y} + \mathbf{m})$. As in Step 3, $X^2 = \lVert\mathbf{Z}_n\rVert^2 \xrightarrow{d} \lVert P(\mathbf{Y} + \mathbf{m})\rVert^2$. The vector $\mathbf{Y} + \mathbf{m}$ has independent normal coordinates with variance 1 and mean vector $\mathbf{m}$, so by Theorem 2 of Sampling Distributions from the Normal the limit is noncentral chi-squared with $k - 1$ degrees of freedom and noncentrality $\lVert P\mathbf{m}\rVert^2 = \lVert\mathbf{m}\rVert^2 = \sum_j \delta_j^2/\pi_{0j}$. $\square$

In use, the theorem is read at the sample in hand: with true shares $\pi_j$, take $\delta_j = \sqrt{n}(\pi_j - \pi_{0j})$, so that

$$\lambda = n\sum_{j=1}^{k}\frac{(\pi_j - \pi_{0j})^2}{\pi_{0j}}, \qquad \text{power} \approx P\big(\chi^2_{k-1}(\lambda) > \chi^2_{k-1, 1-\alpha}\big).$$

By the abstract of Mitra (1958), Cochran had suggested this limiting power in an expository article and illustrated it for the simple goodness-of-fit test; Mitra's paper extends it to probabilities that contain parameters estimated from the sample.

For the older adults the chi-square distance of the true shares from the null shares is $\sum_j(\pi_j - \pi_{0j})^2/\pi_{0j} = 0.0356$, so at $n = 200$, $\lambda = 7.12$, and the approximate power is $P(\chi^2_4(7.12) > 9.488) = 0.548$. A simulation of 10,000 multinomial samples of 200 at the true shares, for each of 30 seeds (5,200,000 + $s$), rejected in a share with median 0.5555 (interquartile range 0.5491 to 0.5603). The study had about an even chance of detecting the difference, and this sample happened to detect it. Under the alternative the statistic is centered near $4 + 7.12 = 11.12$, above the cut-off but not by much.

Example (the sample size for 80% power). The power reaches 0.80 when $P(\chi^2_4(\lambda) > 9.488) = 0.8$, which the script solves to give $\lambda = 11.94$. Dividing by the distance 0.0356 gives $n = 335.3$, so 336 adults. At $n = 336$ the simulation (30 seeds 5,250,000 + $s$) rejected in a share with median 0.8098. Sample size grows like $1/\text{distance}$, as it grows like $1/\text{difference}^2$ for one proportion in Tests and Sample Size for a Proportion; the chi-square distance is a sum of squared differences.

Parameters estimated from the data

Often the null hypothesis names a family rather than a distribution: "the counts are Poisson", with the mean unknown. Back to Bortkiewicz's horse kicks of The Poisson Distribution: 200 corps-years of the Prussian army, with 109, 65, 22, 3 and 1 having 0, 1, 2, 3 and 4 deaths from horse kicks, 122 deaths in all. To test the Poisson model, group the counts as 0, 1, 2, and 3 or more, so that $k = 4$, and fill in the mean with an estimate. Which estimate, and how many degrees of freedom?

Pearson (1900, section 5, p. 164) considered the case in which "the theoretical distribution has to be judged from the sample itself", and argued that the theoretical numbers for the population and those deduced from the sample would "only differ by terms of the order of the probable errors" of the sample's constants, which are small in a fairly large sample. Fisher (1922) showed that the degrees of freedom must be reduced, and that Pearson "did not recognise that in all cases linear restrictions imposed upon the frequencies of the sampled population, by our methods of reconstructing that population, have exactly the same effect upon the distributions of $\chi^2$ as have restrictions placed upon the cell contents of the sample". His paper concerns contingency tables, where the expected counts are built from the observed margins, and it closes with the general rule: when the expected frequencies are connected with the observed ones "by a number of linear relations", each restriction costs a degree of freedom, so that for a frequency curve fitted by four moments "the number of degrees of freedom has been reduced by 4". He added that "in the case of restrictions of a non-linear character, Elderton's tables are no longer exactly applicable". An estimate by maximum likelihood, such as the Poisson mean fitted to grouped counts, imposes restrictions of that non-linear kind, so his paper does not settle our case. The independence article proves Fisher's count for contingency tables. Here we prove that the count still holds in the limit for one parameter estimated by maximum likelihood from the counts themselves (Theorem 6), and for $s$ parameters given the several-parameter form of a result that the site proves only for one (Theorem 7).

The plan of the proof: write the deviations from the fitted expected counts as the deviations from the true ones minus a correction, and show that the correction is, in the limit, the projection of the scaled deviations onto one more direction. The limit is then a projection of rank $k - 2$.

Theorem 6 (one estimated parameter). Let the category probabilities be $\pi_j(\theta)$ for a parameter $\theta$ in an open interval, with true value $\theta_0$, $k \ge 3$, every $\pi_j(\theta) > 0$, and each $\pi_j$ three times continuously differentiable with $\pi'(\theta_0) \ne \mathbf{0}$. Let $\hat\theta$ be a root of the likelihood equation of the counts, $\sum_j O_j\,\pi'_j(\theta)/\pi_j(\theta) = 0$, with $\hat\theta \xrightarrow{p} \theta_0$. Then

$$X^2(\hat\theta) = \sum_{j=1}^{k}\frac{(O_j - n\pi_j(\hat\theta))^2}{n\pi_j(\hat\theta)} \xrightarrow{d} \chi^2_{k-2}.$$

Proof. One observation is a category $J$ with probability $\pi_J(\theta)$, so its log-probability is $\ln\pi_J(\theta)$. The conditions C1 to C5 of Asymptotics of the Maximum Likelihood Estimator hold for this model: the support, the $k$ categories, does not depend on $\theta$; the smoothness is assumed; with finitely many outcomes, sums may be differentiated term by term, which gives the score identities of C4 with Fisher information

$$I(\theta_0) = \sum_j \pi_j\Big(\frac{\pi'_j}{\pi_j}\Big)^2 = \lVert\mathbf{b}\rVert^2, \qquad b_j = \frac{\pi'_j(\theta_0)}{\sqrt{\pi_j(\theta_0)}},$$

which is positive because $\pi'(\theta_0) \ne \mathbf{0}$; and the third derivatives of the $k$ functions $\ln\pi_j$ are continuous, hence bounded near $\theta_0$, which gives C5 with a constant $M$. Write $\pi_j$ and $\pi'_j$ for the values at $\theta_0$ and let $\mathbf{Z}_n$ be the scaled deviations at $\theta_0$, as in Lemma 1.

The score is a linear combination of the scaled deviations. Since $\sum_j \pi_j(\theta) = 1$ for all $\theta$, $\sum_j \pi'_j = 0$. Hence

$$\frac{U(\theta_0)}{\sqrt{n}} = \frac{1}{\sqrt{n}}\sum_j O_j\frac{\pi'_j}{\pi_j} = \frac{1}{\sqrt{n}}\sum_j (O_j - n\pi_j)\frac{\pi'_j}{\pi_j} = \sum_j b_j Z_{nj} = \mathbf{b}\cdot\mathbf{Z}_n.$$

By equation (2) of that article (Theorem 2, the linear representation),

$$\sqrt{n}(\hat\theta - \theta_0) = \frac{\mathbf{b}\cdot\mathbf{Z}_n}{\lVert\mathbf{b}\rVert^2} + R_n, \qquad R_n \xrightarrow{p} 0.$$

The fitted deviations. By the mean value theorem, $\pi_j(\hat\theta) - \pi_j = \pi'_j(\theta_j^*)(\hat\theta - \theta_0)$ for a point $\theta_j^*$ between $\theta_0$ and $\hat\theta$; since $\pi'_j$ is continuous and $\theta_j^* \xrightarrow{p} \theta_0$, $\pi'_j(\theta_j^*) \xrightarrow{p} \pi'_j$. Let $\mathbf{v} = \mathbf{b}/\lVert\mathbf{b}\rVert$. Then

$$\frac{O_j - n\pi_j(\hat\theta)}{\sqrt{n\pi_j}} = Z_{nj} - \frac{\pi'_j(\theta_j^*)}{\sqrt{\pi_j}}\sqrt{n}(\hat\theta - \theta_0) = Z_{nj} - v_j\,(\mathbf{v}\cdot\mathbf{Z}_n) + R'_{nj},$$

where $R'_{nj} = -[\pi'_j(\theta_j^*) - \pi'_j]\sqrt{n}(\hat\theta - \theta_0)/\sqrt{\pi_j} - b_jR_n$. Both terms of $R'_{nj}$ tend to 0 in probability: $\sqrt{n}(\hat\theta - \theta_0)$ converges in distribution (Theorem 2 of the asymptotics article), so it is bounded in probability, and Lemma 4 of the convergence article applies. So the vector of fitted deviations is $(I - \mathbf{v}\mathbf{v}^{\mathsf T})\mathbf{Z}_n + \mathbf{R}'_n$ with $\mathbf{R}'_n \xrightarrow{p} \mathbf{0}$.

The limit. By the proof of Theorem 3, $\mathbf{Z}_n \xrightarrow{d} P\mathbf{Y}$ with $P = I - \mathbf{u}\mathbf{u}^{\mathsf T}$. The vector $\mathbf{v}$ is perpendicular to $\mathbf{u}$, since $\mathbf{u}\cdot\mathbf{b} = \sum_j \pi'_j = 0$. Hence $(I - \mathbf{v}\mathbf{v}^{\mathsf T})(I - \mathbf{u}\mathbf{u}^{\mathsf T}) = I - \mathbf{u}\mathbf{u}^{\mathsf T} - \mathbf{v}\mathbf{v}^{\mathsf T} = P_2$, a symmetric matrix with $P_2^2 = P_2$ (expand and use $\mathbf{u}\cdot\mathbf{v} = 0$) and trace $k - 2$. As in Step 3 of Theorem 3, $\lVert(I - \mathbf{v}\mathbf{v}^{\mathsf T})\mathbf{Z}_n\rVert^2 \xrightarrow{d} \lVert P_2\mathbf{Y}\rVert^2$, which is $\chi^2_{k-2}$ by the projection theorem. Adding $\mathbf{R}'_n$ changes the squared length by $2(I - \mathbf{v}\mathbf{v}^{\mathsf T})\mathbf{Z}_n\cdot\mathbf{R}'_n + \lVert\mathbf{R}'_n\rVert^2$, which tends to 0 in probability by Lemma 4, and Lemma 3 (converging together) keeps the limit. Finally,

$$X^2(\hat\theta) = \sum_j\Big(\frac{O_j - n\pi_j(\hat\theta)}{\sqrt{n\pi_j}}\Big)^2\frac{\pi_j}{\pi_j(\hat\theta)},$$

and each ratio $\pi_j/\pi_j(\hat\theta) \xrightarrow{p} 1$, by continuity. Each squared fitted deviation is bounded in probability, so replacing the ratios by 1 changes the sum by an amount that tends to 0 in probability (Lemma 4 once more), and the limit is $\chi^2_{k-2}$. $\square$

The estimate spends one direction. The score of the counts is, up to the factor $\lVert\mathbf{b}\rVert$, the component of the scaled deviations along $\mathbf{v}$, and maximum likelihood chooses $\hat\theta$ to make the fitted deviations have no component along it.

With $s$ parameters the same argument works with a matrix in place of $\mathbf{b}$, once one ingredient is available: the several-parameter form of the linear representation (2). The asymptotics article proves (2) for one parameter and, for several, states and cites only the normal limit of the estimate (section "Several parameters"), not the representation. The next theorem therefore takes the representation as a hypothesis.

Theorem 7 ($s$ estimated parameters). Let the category probabilities be $\pi_j(\boldsymbol\theta)$ for $\boldsymbol\theta$ in an open set of $\mathbb{R}^s$, with true value $\boldsymbol\theta_0$, $k - 1 - s \ge 1$, every $\pi_j > 0$, each $\pi_j$ continuously differentiable, and let $B$ be the $k \times s$ matrix with entries $B_{jm} = (\partial\pi_j/\partial\theta_m)(\boldsymbol\theta_0)/\sqrt{\pi_j(\boldsymbol\theta_0)}$, of rank $s$. Let $\hat{\boldsymbol\theta}$ be an estimate with values in the parameter set, with $\hat{\boldsymbol\theta} \xrightarrow{p} \boldsymbol\theta_0$, that satisfies

$$\text{(L)} \qquad \sqrt{n}(\hat{\boldsymbol\theta} - \boldsymbol\theta_0) = (B^{\mathsf T}B)^{-1}\,\frac{\mathbf{U}(\boldsymbol\theta_0)}{\sqrt{n}} + \mathbf{R}_n, \qquad \mathbf{R}_n \xrightarrow{p} \mathbf{0},$$

where $\mathbf{U}$ is the score vector of the counts. Then $X^2(\hat{\boldsymbol\theta}) \xrightarrow{d} \chi^2_{k-1-s}$.

For $s = 1$ and $B = \mathbf{b}$, (L) is equation (2) of the asymptotics article, with $I(\theta_0) = \lVert\mathbf{b}\rVert^2 = B^{\mathsf T}B$; under Theorem 6's hypotheses that article proves (2), so Theorem 6 is the case $s = 1$ of Theorem 7. For $s \ge 2$ it is the several-parameter form of (2), which this site does not prove; $B^{\mathsf T}B$ is then the information matrix of one observation, by the same calculation as in Theorem 6, entry by entry.

Proof. The score. As in Theorem 6, $\sum_j \partial\pi_j/\partial\theta_m = 0$ for each $m$, so $\mathbf{u}^{\mathsf T}B = \mathbf{0}$, and component $m$ of the score is $\sum_j (O_j - n\pi_j)(\partial\pi_j/\partial\theta_m)/\pi_j$, so that $\mathbf{U}(\boldsymbol\theta_0)/\sqrt{n} = B^{\mathsf T}\mathbf{Z}_n$. By (L), $\sqrt{n}(\hat{\boldsymbol\theta} - \boldsymbol\theta_0) = (B^{\mathsf T}B)^{-1}B^{\mathsf T}\mathbf{Z}_n + \mathbf{R}_n$, which converges in distribution and so is bounded in probability (each component, by Lemma 2 of the convergence article).

The fitted deviations. Fix a ball about $\boldsymbol\theta_0$ contained in the parameter set. The event that $\hat{\boldsymbol\theta}$ lies in it has probability tending to 1, so what happens off it does not affect the limit; on it the whole segment from $\boldsymbol\theta_0$ to $\hat{\boldsymbol\theta}$ lies in the ball. For each $j$, the mean value theorem applied to $t \mapsto \pi_j(\boldsymbol\theta_0 + t(\hat{\boldsymbol\theta} - \boldsymbol\theta_0))$ on $[0, 1]$ gives a point $\boldsymbol\theta_j^*$ on the segment from $\boldsymbol\theta_0$ to $\hat{\boldsymbol\theta}$ with $\pi_j(\hat{\boldsymbol\theta}) - \pi_j(\boldsymbol\theta_0) = \nabla\pi_j(\boldsymbol\theta_j^*)\cdot(\hat{\boldsymbol\theta} - \boldsymbol\theta_0)$, and $\nabla\pi_j(\boldsymbol\theta_j^*) \xrightarrow{p} \nabla\pi_j(\boldsymbol\theta_0)$ by continuity. Hence, with $Q = B(B^{\mathsf T}B)^{-1}B^{\mathsf T}$,

$$\frac{O_j - n\pi_j(\hat{\boldsymbol\theta})}{\sqrt{n\pi_j}} = Z_{nj} - \big(B\sqrt{n}(\hat{\boldsymbol\theta} - \boldsymbol\theta_0)\big)_j + R'_{nj} = \big((I - Q)\mathbf{Z}_n\big)_j + R'_{nj},$$

where $R'_{nj} = -[\nabla\pi_j(\boldsymbol\theta_j^*) - \nabla\pi_j(\boldsymbol\theta_0)]\cdot\sqrt{n}(\hat{\boldsymbol\theta} - \boldsymbol\theta_0)/\sqrt{\pi_j} - (B\mathbf{R}_n)_j$ tends to 0 in probability, by Lemma 4 of the convergence article applied to each of the $s$ products.

The limit. $Q$ is symmetric, $Q^2 = B(B^{\mathsf T}B)^{-1}B^{\mathsf T}B(B^{\mathsf T}B)^{-1}B^{\mathsf T} = Q$, and its trace is the trace of $(B^{\mathsf T}B)^{-1}B^{\mathsf T}B = I_s$, which is $s$. Since $\mathbf{u}^{\mathsf T}B = \mathbf{0}$, $Q\mathbf{u} = \mathbf{0}$, so $(I - Q)(I - \mathbf{u}\mathbf{u}^{\mathsf T}) = I - \mathbf{u}\mathbf{u}^{\mathsf T} - Q = P_s$, which is symmetric, satisfies $P_s^2 = P_s$ (expand, using $Q\mathbf{u}\mathbf{u}^{\mathsf T} = \mathbf{u}\mathbf{u}^{\mathsf T}Q = 0$) and has trace $k - 1 - s$. From here the proof of Theorem 6 applies word for word with $I - Q$ in place of $I - \mathbf{v}\mathbf{v}^{\mathsf T}$ and $P_s$ in place of $P_2$: $\lVert(I - Q)\mathbf{Z}_n\rVert^2 \xrightarrow{d} \lVert P_s\mathbf{Y}\rVert^2$, which is $\chi^2_{k-1-s}$ by the projection theorem; the remainders and the ratios $\pi_j(\boldsymbol\theta_0)/\pi_j(\hat{\boldsymbol\theta})$ do not change the limit. $\square$

Chernoff and Lehmann (1954) call the result for estimates from the cell frequencies "the well known result" (from their abstract).

Example (the horse kicks). The maximum likelihood estimate from the raw counts is the mean, $\hat\lambda = 122/200 = 0.61$ (Maximum Likelihood Estimation). The estimate from the grouped counts, the root of Theorem 6's likelihood equation, is slightly different, 0.6083, because the grouping hides whether the four corps-years in the top cell had 3 deaths or 4. With the grouped estimate the expected counts are 108.85, 66.22, 20.14 and 4.79, against observed 109, 65, 22 and 4. Then $X^2 = 0.324$ on $k - 1 - 1 = 2$ degrees of freedom, $p = 0.850$; the raw-data estimate gives the same $X^2$ to three decimals and $p = 0.851$. The Poisson model fits the ten corps closely, as the index of dispersion in The Poisson Distribution also found. For all fourteen corps, 280 corps-years with counts 144, 91, 32 and 13 in the four cells, the grouped estimate is 0.7019, the expected counts are 138.78, 97.41, 34.19 and 9.62, and $X^2 = 1.942$ on 2 degrees of freedom, $p = 0.379$: the excess of years with no deaths and with three or more is visible and not significant.

Estimating from the raw data instead

Theorem 6 requires the estimate from the grouped counts. In practice the estimate usually comes from the raw observations, the sample mean of the horse-kick counts or the mean and standard deviation of a sample before it is binned. Chernoff and Lehmann (1954) settled this case. By their abstract, with the maximum likelihood estimate based on the original observations "the test statistic does not have a limiting $\chi^2$-distribution, but ... it is stochastically larger than would be expected under the $\chi^2$ theory"; their examples "indicate that the error is not serious in the case of fitting a Poisson distribution, but may be so for the fitting of a normal". The raw-data estimate uses information the cells do not carry, so it fits the cells less closely than the grouped estimate, which is chosen to fit them.

The script measured the size of each version at the 5% level, with 10,000 samples for each of 30 seeds:

Model and estimate Reference distribution Median rejection rate Interquartile range
Poisson, $\lambda = 0.61$ known $\chi^2_3$ 0.0506 0.0487 to 0.0526
Poisson, $\lambda$ from the four cells $\chi^2_2$ 0.0479 0.0468 to 0.0489
Poisson, $\lambda$ = sample mean $\chi^2_2$ 0.0481 0.0472 to 0.0493
Poisson, $\lambda$ = sample mean $\chi^2_3$ 0.0189 0.0182 to 0.0196
Normal, mean and SD from the raw data $\chi^2_5$ 0.0587 0.0575 to 0.0614
Normal, mean and SD from the raw data $\chi^2_7$ 0.0180 0.0178 to 0.0187

The Poisson rows use samples of 200 counts with mean 0.61, grouped as for the horse kicks (seeds 5,300,000 + $s$). The normal rows use samples of 200 standard normal observations in 8 cells fixed in advance, equally likely under the true distribution, with the mean and the maximum likelihood standard deviation (divisor $n$) of the raw data (seeds 5,400,000 + $s$). As observations of these two settings, they agree with the abstract. For the Poisson, the raw and grouped estimates give nearly the same rate. For the normal, the raw estimates push the rate above 5%, to 0.0587, with the degrees of freedom of Theorem 6's rule, $8 - 1 - 2 = 5$. Not subtracting the estimated parameters at all, the other error, makes either test far too cautious: 0.0189 and 0.0180 in place of 0.05.

What is proved here, what is cited, and what the experiments show

Proved here: the moments of multinomial counts (Theorem 1); Pearson's statistic as the Mahalanobis distance of $k - 1$ deviations and as the score statistic, and $E[X^2] = k - 1$ exactly (Theorem 2 and what follows it); $X^2 = z^2$ for two categories and its $\chi^2_1$ limit; Pearson's theorem for any $k$ (Theorem 3); the bounds relating $G^2$ to $X^2$ and the limit of their difference (Theorem 4); the noncentral limit under local alternatives (Theorem 5); the loss of one degree of freedom for one parameter estimated from the counts (Theorem 6), and of $s$ for $s$ parameters given the hypothesis (L) (Theorem 7).

Cited: the Cramér–Wold device and the Lindeberg–Feller theorem (Durrett, 2019), and through the linked articles the continuity theorem for characteristic functions; the several-parameter form of Wilks's theorem; Fisher (1922) for the history of the degrees of freedom, whose rule covers linear restrictions only; the behavior with raw-data estimates (Chernoff and Lehmann, 1954, from their abstract).

Assumed, neither proved nor cited: the several-parameter linear representation (L) of the maximum likelihood estimate, which Theorem 7 takes as a hypothesis. Chernoff and Lehmann (1954) call the resulting limit for estimates from the cell frequencies "the well known result".

What the experiments show: for the null shares of the running example, the exact sizes at 10, 20 and 40 observations and the simulated sizes at 200; the power at the true shares of the older adults; and the sizes with estimated parameters in one Poisson and one normal setting. Each is an observation about those settings.

Worked problems

The problems are posed on this article's two running examples and on small exact cases, to exercise the covariance of the counts, the exact test, a fully specified model and the power calculation. Every number is computed by the script named in the last section.

Problem 1 (the covariance of the counts by enumeration). Four observations fall into three categories with probabilities 0.5, 0.3 and 0.2. Find $\operatorname{Cov}(O_1, O_2)$ from Theorem 1, check it against $\operatorname{Var}(O_1 + O_2) = \operatorname{Var}(n - O_3)$, and say why it is negative.

Solution. $\operatorname{Cov}(O_1, O_2) = -4 \times 0.5 \times 0.3 = -0.60$. Check: $\operatorname{Var}(O_1 + O_2) = \operatorname{Var}(O_1) + \operatorname{Var}(O_2) + 2\operatorname{Cov}(O_1, O_2) = 4(0.5)(0.5) + 4(0.3)(0.7) - 1.2 = 1 + 0.84 - 1.2 = 0.64$, and $\operatorname{Var}(4 - O_3) = \operatorname{Var}(O_3) = 4(0.2)(0.8) = 0.64$. The two agree. The script also lists all 15 possible tables of four observations in three categories, with their multinomial probabilities, and finds the same means and covariance matrix exactly. The covariance is negative because the counts share a fixed total: every observation in category 1 is one that is not in category 2.

Problem 2 (an exact multinomial test). The script drew 20 adults aged 60 and over (seed 51020). Their answers were 1 excellent, 6 very good, 6 good, 5 fair and 2 poor. Test them against the shares of all adults by the chi-squared approximation and exactly.

Solution. The expected counts are 2.30, 5.25, 7.47, 4.18 and 0.80, one of them below 1. Pearson's statistic is

$$X^2 = \frac{(1 - 2.30)^2}{2.30} + \frac{(6 - 5.25)^2}{5.25} + \frac{(6 - 7.47)^2}{7.47} + \frac{(5 - 4.18)^2}{4.18} + \frac{(2 - 0.80)^2}{0.80} = 3.109,$$

and $P(\chi^2_4 \ge 3.109) = 0.5398$. For the exact test, the script listed all 10,626 tables of 20 observations in 5 categories with their null probabilities. The observed table has probability 0.001267. "At least as extreme" can be defined in two ways, and they give different answers. Ordering the tables by $X^2$, the probability of a table with $X^2 \ge 3.109$ is 0.5276. Ordering them by their own probability, the probability of a table no more likely than the observed one is 0.4621. Every version agrees that 20 older adults show no evidence of a difference: the true difference, which needed about 336 observations for 80% power, is far beyond what 20 can detect. The ordering by $X^2$ is the exact version of the test the chi-squared approximation approximates, and here it differs from it by 0.012.

Problem 3 (a fully specified Poisson model). Suppose the mean had been fixed in advance at 0.70 deaths per corps-year, the mean of all fourteen corps, and the ten corps tested against the Poisson distribution with that mean. Find $X^2$, its degrees of freedom and the $p$-value.

Solution. $X^2 = 2.633$ on 3 degrees of freedom, $p = 0.452$. The Poisson probabilities with mean 0.7 are $e^{-0.7} = 0.4966$, $0.7e^{-0.7} = 0.3476$, $0.49e^{-0.7}/2 = 0.1217$ and the remainder $0.0341$, so the expected counts among 200 corps-years are 99.32, 69.52, 24.33 and 6.83. Then

$$X^2 = \frac{(109 - 99.32)^2}{99.32} + \frac{(65 - 69.52)^2}{69.52} + \frac{(22 - 24.33)^2}{24.33} + \frac{(4 - 6.83)^2}{6.83} = 2.633.$$

Treated as fixed in advance, the mean is not estimated from the counts being tested, so Theorem 3 applies with $k - 1 = 3$ degrees of freedom. (It is not quite fixed in advance: the fourteen corps include these ten, whose 122 deaths are part of the 196. The problem sets that aside.) The 5% cut-off of $\chi^2_3$ is 7.815, far above 2.633. The test does not reject a mean of 0.7, although the ten corps' own mean is 0.61. A test aimed at the mean alone does not reject either. For $n$ Poisson counts with total $t$ the score is $U(\lambda_0) = t/\lambda_0 - n$ and the information is $n/\lambda_0$, so the score statistic of Likelihood Ratio, Wald and Score Tests is the square of $z = (t - n\lambda_0)/\sqrt{n\lambda_0} = (122 - 140)/\sqrt{140} = -1.52$, with two-sided $p = 0.128$. It uses one degree of freedom for the mean, where the goodness-of-fit test spreads its power over three.

Problem 4 (the power of the two-category test). In the population, a share 0.3105 of adults aged 60 and over rate their health fair or poor, against 0.2488 of all adults. What power does the two-category test of the section "Two categories" have with 200 older adults, by Theorem 5 and exactly?

Solution. With two categories the noncentrality is $\lambda = n[(\pi_1 - p_0)^2/p_0 + (\pi_1 - p_0)^2/(1 - p_0)] = n(\pi_1 - p_0)^2/(p_0(1 - p_0))$, which is 4.073 here. The 5% cut-off of $\chi^2_1$ is $1.96^2 = 3.841$, and $P(\chi^2_1(4.073) > 3.841) = 0.523$. The exact power sums the binomial probabilities, with $n = 200$ and $p = 0.3105$, of the counts $x$ for which $(x - 200 \times 0.2488)^2/(200 \times 0.2488 \times 0.7512)$ exceeds 3.841. It is 0.532. So the two-category test had about the same chance of rejecting as the five-category test, 0.548 by the approximation. In this sample it did not reject and the five-category test did. Over repeated samples the two tests reject in different samples, because they respond to different departures: the merged test only to the share in the bottom two categories, the full test to all of them.

Summary

Quantity Formula or result
Moments of the counts $E[O_j] = n\pi_j$, $\operatorname{Var}(O_j) = n\pi_j(1 - \pi_j)$, $\operatorname{Cov}(O_j, O_l) = -n\pi_j\pi_l$
Pearson's statistic $X^2 = \sum_j (O_j - E_j)^2/E_j$, $E_j = n\pi_{0j}$; the score statistic; $E[X^2] = k - 1$
Null limit $X^2 \xrightarrow{d} \chi^2_{k-1}$ (Theorem 3)
Likelihood ratio statistic $G^2 = 2\sum_j O_j\ln(O_j/E_j) = 2nD(\hat{\boldsymbol\pi}\,\|\,\boldsymbol\pi_0)$; $G^2 - X^2 \xrightarrow{p} 0$
Power $\approx P(\chi^2_{k-1}(\lambda) > \chi^2_{k-1,1-\alpha})$, $\lambda = n\sum_j(\pi_j - \pi_{0j})^2/\pi_{0j}$
Parameters estimated from the counts one: $\chi^2_{k-2}$ (Theorem 6); $s$: $\chi^2_{k-1-s}$ under (L) (Theorem 7)
Parameters estimated from raw data stochastically larger than $\chi^2_{k-1-s}$ (Chernoff and Lehmann, 1954)

What we have learned

Pearson's statistic is the squared length of the deviations of the counts from their expected values, each scaled by the square root of the expected count. The fixed total removes one direction, the scaled deviations live in the remaining $k - 1$ dimensions, and in large samples they behave like a projected standard normal vector, so their squared length is chi-squared with $k - 1$ degrees of freedom. The likelihood ratio statistic is twice $n$ times the divergence of the sample shares from the null shares, and it agrees with Pearson's when every cell is near its expectation. Each parameter fitted from the counts removes one more direction (proved here for one parameter, and for several given the linear representation (L)). For 200 older adults the test found their self-rated health different from that of all adults, with a test that had about an even chance of doing so.

The same statistic tests a hypothesis about two classifications at once, whether a person's health and age group are independent, with expected counts built from the table's own margins. The Chi-Square Test of Independence counts the directions those margins remove and proves that $(r - 1)(c - 1)$ remain.

In Python

import numpy as np
from scipy import stats

O = np.array([7, 46, 88, 50, 9])
pi0 = np.array([603, 1376, 1959, 1095, 209]) / 5242
print(stats.chisquare(O, 200 * pi0))                                   # X2 = 16.043, p = 0.0030
print(stats.power_divergence(O, 200 * pi0, lambda_='log-likelihood'))  # G2 = 20.067, p = 0.0005

scipy.stats.chisquare takes a ddof argument for estimated parameters, which lowers the degrees of freedom from $k - 1$ to $k - 1 - \text{ddof}$. Every number in this article is computed and checked against the page by tools/experiments/statistics/chi-square-goodness-of-fit.py in the site's repository.

References