A sample of 300 adults drawn at random from the 5,242 adults of the National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018 who rated their health can be cross-classified by that rating and by age group:
| Health | 18–39 | 40–59 | 60 and over | Total |
|---|---|---|---|---|
| Excellent | 21 | 15 | 14 | 50 |
| Very good | 39 | 17 | 16 | 72 |
| Good | 46 | 28 | 37 | 111 |
| Fair | 14 | 21 | 24 | 59 |
| Poor | 1 | 1 | 6 | 8 |
| Total | 121 | 82 | 97 | 300 |
Is self-rated health independent of age group, or does the distribution of answers differ between the age groups by more than sampling would explain? It differs: the test of this article gives $p = 0.0095$, and its residuals show where the difference lies.
A test of independence asks whether two classifications of the same individuals are statistically independent, using a table of counts with $r$ rows and $c$ columns (a contingency table). The statistic is Pearson's, built from expected counts that the table's own margins supply. Comparing Two Proportions proves for $2 \times 2$ tables that the statistic is the square of the two-proportion $z$ statistic, and treats Yates' correction and Fisher's exact test; none of that is repeated here. This article handles tables of any size.
First we describe the three ways a table can be sampled and how conditioning connects them. Then we derive the expected counts and count the degrees of freedom, and prove the chi-squared limit with $(r - 1)(c - 1)$ degrees of freedom. Next come the likelihood ratio statistic, the residuals that locate a departure, the exact test for small counts and a measure of strength. The article builds on The Chi-Square Goodness-of-Fit Test, whose Lemma 1 and Theorem 3 it uses, and on the projection theorem of Sampling Distributions from the Normal.
Three ways to sample a table
Write $n_{ij}$ for the count in row $i$ and column $j$, $n_{i+} = \sum_j n_{ij}$ for the row totals, $n_{+j} = \sum_i n_{ij}$ for the column totals and $n$ for the grand total. The probabilities are $\pi_{ij}$, with marginal probabilities $\pi_{i+}$ and $\pi_{+j}$. Three sampling designs produce such a table.
- Multinomial sampling. A sample of $n$ individuals is drawn and each is classified both ways. Only $n$ is fixed, and the $rc$ counts are multinomial with probabilities $\pi_{ij}$. The hypothesis is independence, $\pi_{ij} = \pi_{i+}\pi_{+j}$ for all $i$ and $j$. The running example is of this kind.
- Independent multinomial samples. A sample of fixed size $n_{i+}$ is drawn from each of $r$ populations, such as 100 adults from each age group, and each is classified into one of $c$ categories. Row $i$ is multinomial with probabilities $\pi_{j \mid i}$, which sum to 1 over $j$. The hypothesis is homogeneity: the same column probabilities $\pi_{j \mid i} = \pi_j$ in every row.
- Both margins fixed. Both sets of totals are fixed by the design or by conditioning. The classic case is Fisher's exact test of a $2 \times 2$ table in Comparing Two Proportions.
The three are connected by conditioning, and the connection is why one statistic serves all three.
Theorem 1 (conditioning on the row totals). Under multinomial sampling, given the row totals, the rows are independent multinomial samples of sizes $n_{i+}$ with probabilities $\pi_{j \mid i} = \pi_{ij}/\pi_{i+}$. Under independence these are $\pi_{+j}$ in every row, so independence becomes homogeneity.
Proof. The row totals are multinomial with probabilities $\pi_{i+}$ (each individual falls in row $i$ with probability $\pi_{i+}$). For a table with row totals $m_i$,
$$P(\text{table} \mid \text{row totals } m) = \frac{\dfrac{n!}{\prod_{i,j} n_{ij}!}\prod_{i,j}\pi_{ij}^{n_{ij}}}{\dfrac{n!}{\prod_i m_i!}\prod_i \pi_{i+}^{m_i}} = \prod_i\Big[\frac{m_i!}{\prod_j n_{ij}!}\prod_j\Big(\frac{\pi_{ij}}{\pi_{i+}}\Big)^{n_{ij}}\Big],$$
using $\pi_{i+}^{m_i} = \prod_j \pi_{i+}^{n_{ij}}$. The right side is a product over the rows of multinomial probabilities, which is the joint probability of independent multinomial rows. Under independence $\pi_{ij}/\pi_{i+} = \pi_{+j}$. $\square$
Theorem 2 (conditioning on both margins). Under homogeneity, given the row and column totals, the table has the multiple hypergeometric distribution
$$P(\text{table} \mid \text{both margins}) = \frac{\prod_i n_{i+}!\,\prod_j n_{+j}!}{n!\,\prod_{i,j} n_{ij}!},$$
which does not depend on the common probabilities. By Theorem 1 the same holds under independence with multinomial sampling.
Proof. Under homogeneity the rows are independent multinomials with common probabilities $\pi_j$, so the probability of the table is $\prod_i [n_{i+}!/\prod_j n_{ij}!]\prod_j \pi_j^{n_{+j}}$, after collecting the powers of each $\pi_j$ over the rows. The column totals are then multinomial with $n$ trials and probabilities $\pi_j$, since every individual falls in column $j$ with probability $\pi_j$ independently of the others, with probability $[n!/\prod_j n_{+j}!]\prod_j\pi_j^{n_{+j}}$. Dividing, the powers of $\pi_j$ cancel and the stated ratio remains. $\square$
The distribution of Theorem 2 has a direct description: list the $n$ individuals with their row labels, and assign the column labels, $n_{+1}$ of the first kind, $n_{+2}$ of the second and so on, at random to the individuals, every arrangement being equally likely. The number of arrangements giving a particular table is $\prod_i [n_{i+}!/\prod_j n_{ij}!]$ out of $n!/\prod_j n_{+j}!$, which is the formula. For two rows and two columns it is the hypergeometric distribution of Fisher's exact test. Because it involves no unknown parameter, it supports an exact test, which the section on small counts uses.
The expected counts
Under independence the probabilities have the form $\pi_{ij} = \alpha_i\beta_j$, with $\sum_i\alpha_i = \sum_j\beta_j = 1$, and the log-likelihood of a multinomial table separates:
$$\ell = \sum_{i,j} n_{ij}\ln(\alpha_i\beta_j) = \sum_i n_{i+}\ln\alpha_i + \sum_j n_{+j}\ln\beta_j.$$
Each sum is the log-likelihood of a single multinomial sample, maximized at the shares, $\hat\alpha_i = n_{i+}/n$ and $\hat\beta_j = n_{+j}/n$ (Cross-Entropy and Kullback–Leibler Divergence proves that the shares maximize it). The fitted probabilities are $\hat\pi_{ij} = (n_{i+}/n)(n_{+j}/n)$, and the estimated expected counts are
$$E_{ij} = n\hat\pi_{ij} = \frac{n_{i+}\,n_{+j}}{n}.$$
Under homogeneity the common probabilities are estimated by pooling the rows, $\hat\pi_j = n_{+j}/n$, and the expected count in row $i$ is $n_{i+}\hat\pi_j$: the same formula. The Pearson statistic for independence is
$$X^2 = \sum_{i=1}^{r}\sum_{j=1}^{c}\frac{(n_{ij} - E_{ij})^2}{E_{ij}}.$$
For the running sample:
| Health | 18–39 | 40–59 | 60 and over |
|---|---|---|---|
| Excellent | 20.17 | 13.67 | 16.17 |
| Very good | 29.04 | 19.68 | 23.28 |
| Good | 44.77 | 30.34 | 35.89 |
| Fair | 23.80 | 16.13 | 19.08 |
| Poor | 3.23 | 2.19 | 2.59 |
For example, $E_{11} = 50 \times 121/300 = 20.17$. Then $X^2 = 20.222$.
Degrees of freedom
The full multinomial model has $rc - 1$ free probabilities, and independence leaves $(r - 1) + (c - 1)$ of them free, the $\alpha_i$ and $\beta_j$ less one constraint each. The difference is
$$(rc - 1) - (r - 1) - (c - 1) = (r - 1)(c - 1),$$
8 for the running table. The same number appears in the deviations. Every row of deviations $n_{ij} - E_{ij}$ sums to $n_{i+} - n_{i+} = 0$, and every column to 0. Fix the deviations in the first $r - 1$ rows and $c - 1$ columns, and the last column of each of those rows, the last row, and its last cell are all determined. That leaves $(r - 1)(c - 1)$ deviations free.
The count is Fisher's. Pearson had referred the statistic for a contingency table to the chi-squared distribution with one fewer degree of freedom than the number of cells. Fisher (1922, p. 88) argued that the deviations are confined by the restrictions on the margins to a space whose dimension is the number of degrees of freedom, and concluded that "for a contingency table of $r$ rows and $c$ columns we should take $n' = (c - 1)(r - 1) + 1$ instead of $n' = cr$", where $n'$ is one more than the degrees of freedom, as Elderton's tables were entered. He noted that the correction "often makes a very great difference to the probability". For a $2 \times 2$ table it changes 3 degrees of freedom to 1, and Fisher pointed out that this resolved a puzzle reported by Greenwood and Yule: their fourfold tables had looked much less significant by the chi-square test than by comparing the two proportions with their probable errors. With one degree of freedom the two tests agree, as Comparing Two Proportions proves.
The limit theorem
Here is the idea of the proof. Condition on the row totals, so that the rows are independent multinomial samples with common probabilities. Within each row, the scaled deviations from the true probabilities behave like a projection of independent normal variables that removes the row's fixed total, as in the goodness-of-fit article. Replacing the true probabilities by the pooled estimates subtracts from each row a share of the column sums, which is a second projection acting across the rows. The two projections act on different indices, and their product has rank $(r - 1)(c - 1)$.
It is convenient to treat a table of numbers $M = (M_{ij})$ as a vector of $rc$ coordinates, with squared length $\lVert M\rVert^2 = \sum_{i,j}M_{ij}^2$ and inner product $\langle M, N\rangle = \sum_{i,j}M_{ij}N_{ij}$. For an $r \times r$ matrix $A$ and a $c \times c$ matrix $B$, $AMB$ is the table obtained by applying $A$ to every column of $M$ and $B$ to every row.
Lemma 1 (a product of projections). Let $A$ and $B$ be projection matrices of ranks $a$ and $b$, and let $Y$ be an $r \times c$ table of independent standard normal variables. Then $\lVert AYB\rVert^2$ has the chi-squared distribution with $ab$ degrees of freedom, and the entry $(AYB)_{ij}$ is normal with mean 0 and variance $A_{ii}B_{jj}$.
Proof. By Lemma 2 of Sampling Distributions from the Normal, $A = UU^{\mathsf T}$ and $B = VV^{\mathsf T}$, where $U$ ($r \times a$) and $V$ ($c \times b$) have orthonormal columns $\mathbf{u}_1, \ldots, \mathbf{u}_a$ and $\mathbf{v}_1, \ldots, \mathbf{v}_b$. Let $M = U^{\mathsf T}YV$, an $a \times b$ table. Then $AYB = UMV^{\mathsf T}$, and
$$\lVert UMV^{\mathsf T}\rVert^2 = \operatorname{tr}(VM^{\mathsf T}U^{\mathsf T}UMV^{\mathsf T}) = \operatorname{tr}(M^{\mathsf T}M\,V^{\mathsf T}V) = \operatorname{tr}(M^{\mathsf T}M) = \lVert M\rVert^2,$$
using $U^{\mathsf T}U = I$, $V^{\mathsf T}V = I$ and the invariance of the trace under cyclic reordering. The entries of $M$ are $M_{kl} = \mathbf{u}_k^{\mathsf T}Y\mathbf{v}_l = \langle Y, \mathbf{u}_k\mathbf{v}_l^{\mathsf T}\rangle$. The $ab$ tables $\mathbf{u}_k\mathbf{v}_l^{\mathsf T}$ are orthonormal in the inner product of tables, because $\langle\mathbf{u}_k\mathbf{v}_l^{\mathsf T}, \mathbf{u}_{k'}\mathbf{v}_{l'}^{\mathsf T}\rangle = (\mathbf{u}_k\cdot\mathbf{u}_{k'})(\mathbf{v}_l\cdot\mathbf{v}_{l'})$, which is 1 if $k = k'$ and $l = l'$ and 0 otherwise. Complete them to an orthonormal basis of the $rc$-dimensional space; by Theorem 1 of that article (rotation), the coordinates of $Y$ in this basis are independent standard normal variables. So the $M_{kl}$ are $ab$ independent standard normal variables, and $\lVert AYB\rVert^2 = \lVert M\rVert^2$ is $\chi^2_{ab}$. For the entry, $(AYB)_{ij} = \sum_{k,l}A_{ik}Y_{kl}B_{lj}$ is a fixed linear combination of independent standard normal variables, so it is normal with mean 0 and variance $\sum_{k,l}A_{ik}^2B_{lj}^2 = (A^2)_{ii}(B^2)_{jj} = A_{ii}B_{jj}$, using the symmetry and idempotence of $A$ and $B$. $\square$
The proof also needs a fact about independent sequences, which follows from the cited continuity theorem for characteristic functions (Moment Generating and Characteristic Functions, section "The continuity theorem").
Lemma 2 (independent pieces). For each $n$ let $S_n$ and $T_n$ be independent random variables, with $S_n \xrightarrow{d} S$ and $T_n \xrightarrow{d} T$. Then $S_n + T_n \xrightarrow{d} S + T$, with $S$ and $T$ independent.
Proof. By independence the characteristic function of $S_n + T_n$ is the product $\varphi_{S_n}(t)\varphi_{T_n}(t)$. Each factor converges, because $\cos(tx)$ and $\sin(tx)$ are bounded continuous functions of $x$; so the product converges to $\varphi_S(t)\varphi_T(t)$, the characteristic function of the sum of independent copies of $S$ and $T$. By the continuity theorem, $S_n + T_n$ converges in distribution to that sum. $\square$
Theorem 3 (independent rows). Let the rows be independent multinomial samples of sizes $n_{i+}$ with common probabilities $\pi_1, \ldots, \pi_c$, all positive, and let $n \to \infty$ with $n_{i+}/n \to \omega_i > 0$ for every row. Then $X^2 \xrightarrow{d} \chi^2_{(r-1)(c-1)}$.
Proof. The scaled deviations from the truth. Let $Z_{ij} = (n_{ij} - n_{i+}\pi_j)/\sqrt{n_{i+}\pi_j}$, and let $\mathbf{u} = (\sqrt{\pi_1}, \ldots, \sqrt{\pi_c})$ and $B = I - \mathbf{u}\mathbf{u}^{\mathsf T}$. By the proof of Pearson's theorem in the goodness-of-fit article (Theorem 3, steps 1 and 2), for each row $i$ and each vector $\mathbf{a}_i$, $\mathbf{a}_i\cdot\mathbf{Z}_i \xrightarrow{d} \mathbf{a}_i\cdot B\mathbf{Y}_i$, where $\mathbf{Z}_i$ is row $i$ of $Z$ and $\mathbf{Y}_i$ is a vector of independent standard normal variables. For a fixed table of coefficients $\mathfrak{a}$ with rows $\mathbf{a}_i$, $\langle\mathfrak{a}, Z\rangle = \sum_i \mathbf{a}_i\cdot\mathbf{Z}_i$ is a sum of independent terms, so by Lemma 2, applied $r - 1$ times, it converges in distribution to $\sum_i \mathbf{a}_i\cdot B\mathbf{Y}_i$ with independent $\mathbf{Y}_i$, which is $\langle\mathfrak{a}, YB\rangle$ for a table $Y$ of independent standard normal variables. By the Cramér–Wold device (cited in the goodness-of-fit article), $Z \xrightarrow{d} YB$.
Subtracting the pooled estimate. Let $\hat\pi_j = n_{+j}/n$. Since $\sum_k n_{k+} = n$,
$$\hat\pi_j - \pi_j = \frac{1}{n}\sum_k (n_{kj} - n_{k+}\pi_j) = \frac{1}{n}\sum_k\sqrt{n_{k+}\pi_j}\,Z_{kj}.$$
So, with $w_i = \sqrt{n_{i+}/n}$,
$$D_{ij} = \frac{n_{ij} - E_{ij}}{\sqrt{n_{i+}\pi_j}} = Z_{ij} - \frac{n_{i+}(\hat\pi_j - \pi_j)}{\sqrt{n_{i+}\pi_j}} = Z_{ij} - w_i\sum_k w_kZ_{kj}.$$
That is, $D = A_nZ$ with $A_n = I - \mathbf{w}\mathbf{w}^{\mathsf T}$, an exact identity for every table. The vector $\mathbf{w}$ is a unit vector, since $\sum_i n_{i+}/n = 1$, so $A_n$ is a projection of rank $r - 1$ by the calculation of Lemma 1 in the goodness-of-fit article. As $n \to \infty$, $\mathbf{w} \to \boldsymbol\omega^{1/2} = (\sqrt{\omega_1}, \ldots, \sqrt{\omega_r})$, a unit vector, and $A_n \to A = I - \boldsymbol\omega^{1/2}(\boldsymbol\omega^{1/2})^{\mathsf T}$, also a projection of rank $r - 1$.
The limit. $A_nZ = AZ + (A_n - A)Z$. The table $Z$ converges in distribution, so each of its entries is bounded in probability, and the entries of $(A_n - A)Z$ are sums of entries of $Z$ multiplied by constants tending to 0; they tend to 0 in probability (Lemma 4 of the convergence article). The map $M \mapsto \lVert AM\rVert^2$ is continuous, so $\lVert AZ\rVert^2 \xrightarrow{d} \lVert AYB\rVert^2$, as in Step 3 of Pearson's theorem. Expanding $\lVert AZ + R\rVert^2 - \lVert AZ\rVert^2 = 2\langle AZ, R\rangle + \lVert R\rVert^2$ with $R = (A_n - A)Z$, which tends to 0 in probability, shows that $\lVert D\rVert^2$ has the same limit (Lemma 3 there, converging together). By Lemma 1 with ranks $r - 1$ and $c - 1$, the limit is $\chi^2_{(r-1)(c-1)}$.
The denominators. $X^2 = \sum_{i,j}D_{ij}^2\,\pi_j/\hat\pi_j$. Each $\hat\pi_j \xrightarrow{p} \pi_j$ by the weak law of large numbers, so $\pi_j/\hat\pi_j \xrightarrow{p} 1$, and each $D_{ij}^2$ is bounded in probability. Replacing the ratios by 1 changes $X^2$ by an amount that tends to 0 in probability, and the limit is unchanged. $\square$
The proof shows where the degrees of freedom go. Each row's fixed total removes the direction $\mathbf{u}$ within the row ($B$), and estimating the column probabilities from the pooled rows removes the direction $\mathbf{w}$ across the rows ($A$). What remains has dimension $(r - 1)(c - 1)$.
Theorem 4 (multinomial sampling). Under multinomial sampling with independence and all $\pi_{i+}$ and $\pi_{+j}$ positive, $X^2 \xrightarrow{d} \chi^2_{(r-1)(c-1)}$.
Proof. Let $F$ be the $\chi^2_{(r-1)(c-1)}$ distribution function and fix $x$. For a vector $\mathbf{m}$ of possible row totals let $g_n(\mathbf{m}) = P(X^2 \le x \mid \text{row totals} = \mathbf{m})$. By Theorem 1, given the row totals the rows are independent multinomials with the common probabilities $\pi_{+j}$, which is the setting of Theorem 3. Let $G_n$ be the set of $\mathbf{m}$ with $\lvert m_i/n - \pi_{i+}\rvert \le n^{-1/4}$ for every $i$.
Claim: $\sup_{\mathbf{m} \in G_n}\lvert g_n(\mathbf{m}) - F(x)\rvert \to 0$. If not, there are $\varepsilon > 0$, infinitely many $n$ and vectors $\mathbf{m}^{(n)} \in G_n$ with $\lvert g_n(\mathbf{m}^{(n)}) - F(x)\rvert > \varepsilon$. Along those $n$, $m_i^{(n)}/n \to \pi_{i+} > 0$, so Theorem 3 applies to independent rows of sizes $m_i^{(n)}$ and gives $g_n(\mathbf{m}^{(n)}) \to F(x)$ ($F$ is continuous), a contradiction.
By the law of total probability, $P(X^2 \le x) = \sum_{\mathbf{m}} P(\text{row totals} = \mathbf{m})\,g_n(\mathbf{m})$, so
$$\lvert P(X^2 \le x) - F(x)\rvert \le \sup_{\mathbf{m} \in G_n}\lvert g_n(\mathbf{m}) - F(x)\rvert + P(\text{row totals} \notin G_n).$$
The first term tends to 0 by the claim. For the second, $n_{i+}$ is binomial with mean $n\pi_{i+}$, so by Chebyshev's inequality $P(\lvert n_{i+}/n - \pi_{i+}\rvert > n^{-1/4}) \le \pi_{i+}(1 - \pi_{i+})/(n \cdot n^{-1/2})$, which tends to 0; a union over the $r$ rows keeps it so. $\square$
The statistic is defined only when every row and column total is positive. In Theorems 3 and 4 that event has probability tending to 1, and how $X^2$ is defined off it does not affect the limit.
The limit when both margins are fixed is not proved here; the next sections use the exact conditional distribution of Theorem 2 numerically instead.
Example (the running table). For the 300 adults, $X^2 = 20.222$ on 8 degrees of freedom. The 5% point of $\chi^2_8$ is 15.507, and $P(\chi^2_8 \ge 20.222) = 0.0095$. Health and age group are not independent in the population from which the sample came. The association is visible in the population itself: among all 5,242 adults, the share rating their health excellent falls from 0.166 in the youngest group to 0.110 and 0.073, and the share rating it poor rises from 0.018 to 0.041 and 0.058.
Example (how well the limit holds here). The script drew tables of 300 from the multinomial distribution whose probabilities are the products of the population's row and column shares, so that independence holds exactly, 10,000 tables for each of 30 seeds (5,600,000 + $s$). The test at the 5% level rejected in a share with median 0.0474 (interquartile range 0.0463 to 0.0490). The smallest cell probability is 0.0120, an expected count of 3.6 at this sample size, and the test held its level a little conservatively.
The likelihood ratio statistic and mutual information
The likelihood ratio statistic compares the maximized log-likelihood of the full multinomial, at the cell shares $n_{ij}/n$, with that under independence, at $E_{ij}/n$:
$$G^2 = 2\sum_{i,j} n_{ij}\ln\frac{n_{ij}}{E_{ij}} = 2n\sum_{i,j}\hat p_{ij}\ln\frac{\hat p_{ij}}{\hat p_{i+}\hat p_{+j}} = 2n\,\hat I,$$
where $\hat p_{ij} = n_{ij}/n$ and $\hat I$ is the mutual information, in nats, of the table's shares. Mutual Information works this identity in its fifth problem. For the running table $\hat I = 0.0336$ nats and $G^2 = 20.186$, with $p = 0.0097$ on 8 degrees of freedom.
The proof of Theorem 4 of the goodness-of-fit article carries over word for word with $p = (\hat p_{ij})$ and $q = (\hat p_{i+}\hat p_{+j})$: the bounds $X^2/(1 + \varepsilon) \le G^2 \le X^2/(1 - \varepsilon)$ hold with $\varepsilon = \max_{i,j}\lvert n_{ij}/E_{ij} - 1\rvert$, and under independence $\varepsilon \xrightarrow{p} 0$ because $n_{ij}/n$ and $E_{ij}/n$ both tend to $\pi_{i+}\pi_{+j}$. So $G^2 - X^2 \xrightarrow{p} 0$ and $G^2$ has the same limit.
In this table the two agree closely, 20.186 against 20.222. That they agree does not mean either is accurate. In the simulation of the last section $G^2$ rejected in a share with median 0.0597 (interquartile range 0.0582 to 0.0608), against 0.0474 for $X^2$. In a second simulation with tables of 40 (seeds 5,650,000 + $s$), the medians were 0.0837 for $G^2$ (0.0820 to 0.0851) and 0.0387 for $X^2$ (0.0369 to 0.0401). At 40 the smallest expected count under independence is 0.48, and a share 0.201 of the simulated tables had an empty row or column; those were tested on the rows and columns that remained, with correspondingly fewer degrees of freedom. As observations of this table's margins: Pearson's statistic held its level from below, and the likelihood ratio statistic exceeded it, by more in the smaller samples.
Where the dependence lies: residuals
A significant $X^2$ says that the table departs from independence, not where. The Pearson residuals $e_{ij} = (n_{ij} - E_{ij})/\sqrt{E_{ij}}$, whose squares sum to $X^2$, are the obvious place to look.
Example (Pearson residuals are not standard normal). It is tempting to read a Pearson residual of 2 as two standard deviations from expectation. It is less than that. Theorem 3's proof gives the limit of $D_{ij}$, which differs from $e_{ij}$ only by the factor $\sqrt{\hat\pi_j/\pi_j} \xrightarrow{p} 1$: it is the entry $(AYB)_{ij}$ of Lemma 1, normal with variance $A_{ii}B_{jj} = (1 - \omega_i)(1 - \pi_j)$. Under multinomial sampling, $\omega_i = \pi_{i+}$ and $\pi_j = \pi_{+j}$, so a Pearson residual has limiting variance $(1 - \pi_{i+})(1 - \pi_{+j})$, below 1, and well below it for a large row or column. In the running table, with the margins estimated from the sample, a cell in the good row and the youngest column has limiting variance about $(1 - 111/300)(1 - 121/300) = 0.38$, a standard deviation near 0.6.
The exact variance comes from Theorem 2.
Theorem 5 (the conditional moments of a cell). Under homogeneity, or independence, given both margins,
$$E[n_{ij}] = \frac{n_{i+}n_{+j}}{n} = E_{ij}, \qquad \operatorname{Var}(n_{ij}) = \frac{n_{i+}\,n_{+j}\,(n - n_{i+})(n - n_{+j})}{n^2(n - 1)} = E_{ij}\Big(1 - \frac{n_{i+}}{n}\Big)\Big(1 - \frac{n_{+j}}{n}\Big)\frac{n}{n - 1}.$$
Proof. By the description after Theorem 2, the column labels are assigned at random to the $n$ individuals. Let $I_l = 1$ if individual $l$ of row $i$ receives label $j$, so that $n_{ij} = \sum_{l=1}^{n_{i+}} I_l$. Each individual receives label $j$ with probability $q = n_{+j}/n$, so $E[n_{ij}] = n_{i+}q = E_{ij}$. Two different individuals both receive it with probability $q(n_{+j} - 1)/(n - 1)$, so
$$\operatorname{Cov}(I_l, I_{l'}) = q\,\frac{n_{+j} - 1}{n - 1} - q^2 = -\frac{q(1 - q)}{n - 1}, \qquad l \ne l'.$$
Hence $\operatorname{Var}(n_{ij}) = n_{i+}q(1 - q) - n_{i+}(n_{i+} - 1)\,q(1 - q)/(n - 1) = n_{i+}q(1 - q)(n - n_{i+})/(n - 1)$, which is the stated formula. $\square$
Dividing by the square root of this variance, with $n/(n - 1)$ dropped, gives the adjusted residual
$$r_{ij} = \frac{n_{ij} - E_{ij}}{\sqrt{E_{ij}(1 - n_{i+}/n)(1 - n_{+j}/n)}}.$$
Under multinomial sampling and independence it converges in distribution to the standard normal: it is $D_{ij}$ divided by the square root of $(1 - n_{i+}/n)(1 - n_{+j}/n)$ times $\hat\pi_j/\pi_j$, both of which converge in probability to their limits, so Slutsky's theorem applies to the limit of $D_{ij}$ found above. That limit was found in Theorem 3, for fixed row totals. Under multinomial sampling the conditional limit given the row totals is standard normal whatever the row shares, and the argument of Theorem 4, with the standard normal distribution function in place of $F$, removes the conditioning. Haberman (1973) applied residual analysis of this kind, developed in his earlier work on loglinear models, to the hypothesis of independence in an $r \times c$ table, by his abstract; the derivation above is our own.
For the running table:
| Health | 18–39 | 40–59 | 60 and over |
|---|---|---|---|
| Excellent | 0.26 | 0.46 | −0.72 |
| Very good | 2.74 | −0.81 | −2.10 |
| Good | 0.30 | −0.63 | 0.28 |
| Fair | −2.90 | 1.59 | 1.53 |
| Poor | −1.63 | −0.95 | 2.62 |
The Pearson residuals are smaller throughout; for example, 1.85 against 2.74 for the young and very good, and −2.01 against −2.90 for the young and fair. Four adjusted residuals exceed 2 in absolute value: more young adults than independence predicts rate their health very good and fewer rate it fair, and more adults of 60 and over rate it poor and fewer very good. The pattern is the population's gradient seen through a sample of 300.
Two cautions. The 15 residuals are 15 looks at one table, and with 15 looks some large values are expected by chance even under independence; Multiple Comparisons and the Bonferroni Correction treats the problem. They are also not independent: within each row and column they are tied by the zero sums of the deviations.
Example (one cell exactly). The cell for poor health at 60 and over has row total 8, column total 97, and $E = 8 \times 97/300 = 2.59$. By Theorem 5 its conditional variance is $8 \times 97 \times 292 \times 203/(300^2 \times 299) = 1.7093$, against $1.7036$ without the factor $n/(n - 1) = 1.0033$. The observed 6 lies $(6 - 2.59)/1.307 = 2.61$ conditional standard deviations above its mean, against the adjusted residual of 2.62.
Small expected counts: the permutation test
The poor row of the running table has expected counts 3.23, 2.19 and 2.59, all below 5. When the chi-squared approximation is in doubt, Theorem 2 supplies an exact alternative. Given both margins, the null distribution of the table is the multiple hypergeometric, free of unknown parameters, so the probability that $X^2$ is at least its observed value can be computed exactly. Listing every table with the given margins is impractical here, but the description after Theorem 2 gives a simulation: shuffle the age labels among the 300 individuals, keeping their health ratings, and recompute $X^2$. The share of shuffles at least as large as the observed 20.222 estimates the exact conditional $p$-value.
The script made 10,000 shuffles for each of 30 seeds (5,500,000 + $s$). The share had median 0.0086 (interquartile range 0.0079 to 0.0092), against 0.0095 from the chi-squared approximation. The two agree to within the play of the simulation's seeds and the approximation's error; the small counts in the poor row did not mislead the approximation here. Problem 2 shows a table where they part.
How strong is the association?
The $p$-value measures evidence, not strength. Fix the cell shares and let $n$ grow: $X^2/n = \sum_{i,j}(\hat p_{ij} - \hat p_{i+}\hat p_{+j})^2/(\hat p_{i+}\hat p_{+j})$ does not change, so $X^2$ grows in proportion to $n$ and any fixed association, however weak, becomes significant in a large enough sample. A measure of strength must remove $n$, and a scale is supplied by the largest value $X^2$ can take.
Theorem 6 (the largest value of X²). For any table with all margins positive, $X^2 \le n(\min(r, c) - 1)$. If $c \le r$, equality holds exactly when every row has all its observations in a single column; if $r \le c$, when every column has all its observations in a single row.
Proof. Expand the square: $\sum (n_{ij} - E_{ij})^2/E_{ij} = \sum n_{ij}^2/E_{ij} - 2\sum n_{ij} + \sum E_{ij}$, and both of the last sums are $n$. With $E_{ij} = n_{i+}n_{+j}/n$,
$$X^2 = n\Big(\sum_{i,j}\frac{n_{ij}^2}{n_{i+}\,n_{+j}} - 1\Big).$$
Since $n_{ij} \le n_{i+}$, each term satisfies $n_{ij}^2/(n_{i+}n_{+j}) \le n_{ij}/n_{+j}$, and $\sum_{i,j}n_{ij}/n_{+j} = \sum_j 1 = c$. So $X^2 \le n(c - 1)$, with equality exactly when $n_{ij} = n_{i+}$ for every cell with $n_{ij} > 0$, that is, when each row lies in one column. Using $n_{ij} \le n_{+j}$ in the same way gives $X^2 \le n(r - 1)$, with equality when each column lies in one row. The smaller bound is $n(\min(r, c) - 1)$. $\square$
The measure
$$V = \sqrt{\frac{X^2}{n(\min(r, c) - 1)}}$$
therefore lies between 0 (the sample shares are exactly independent) and 1 (each category of the shorter classification determines the other). It is the measure scipy calls Cramér's V (scipy.stats.contingency.association with method='cramer'); for a $2 \times 2$ table it is the absolute value of the phi coefficient, since $X^2/n = \phi^2$ there.
For the running sample $V = 0.184$, and for the whole population of 5,242 adults, with $X^2 = 172.88$, $V = 0.128$. The association is real and weak. The sample's $V$ exceeds the population's, as it tends to: under independence $X^2$ is about $(r - 1)(c - 1)$ on average rather than 0, so $V$ estimated from a sample is pushed up by noise.
Ordered categories
Both classifications of the running table are ordered, and the test ignores the order: permuting the rows or the columns of the table leaves $X^2$ unchanged. It spends its 8 degrees of freedom on every pattern of departure, including those no one would expect, such as excellent and poor health both rising with age. When the alternative of interest is a trend, a test aimed at it has more power. Trends in Proportions: the Cochran–Armitage Test develops the test for an ordered classification against a binary one and splits the $r \times 2$ statistic into a trend part and the rest. Mantel (1963), by his abstract, extended the idea to tables with "arbitrarily many rows and/or columns, where rows and columns are orderable", with a score assigned to each row or column and the variance "conditioned on all marginal totals".
What is proved here, what is cited, and what the experiments show
Proved here: the conditional distributions given the row totals and given both margins (Theorems 1 and 2); the expected counts as maximum likelihood estimates under independence and under homogeneity; the count of degrees of freedom; the chi-squared distribution of a product of two projections (Lemma 1); the limit $\chi^2_{(r-1)(c-1)}$ for independent rows (Theorem 3) and for multinomial sampling (Theorem 4); the identity $G^2 = 2n\hat I$ and the vanishing of $G^2 - X^2$; the limiting variance of a Pearson residual; the exact conditional mean and variance of a cell (Theorem 5) and the limit of the adjusted residual; the bound on $X^2$ (Theorem 6).
Cited: the Cramér–Wold device and the continuity theorem for characteristic functions, through the linked articles; Fisher (1922) for the history of the degrees of freedom; Haberman (1973) and Mantel (1963) from their abstracts. The limit when both margins are fixed is stated, not proved.
What the experiments show: for multinomial tables with this example's margins under independence, the sizes of the $X^2$ and $G^2$ tests at 300 and 40 observations; the permutation $p$-values of the running table and of the table in Problem 2. Each is an observation about those settings.
Worked problems
The problems are posed on samples from this unit's NHANES population and on a small constructed table, to exercise the test for an $r \times 2$ table and its residuals, the permutation test and the bound of Theorem 6. Every number is computed by the script named in the last section.
Problem 1 (smoking and diabetes). The script drew 400 adults (seed 52101) from the 5,107 NHANES adults of the course's risk-factor population (nonpregnant adults with blood pressure readings, a body mass index, an answer on diagnosed diabetes, borderline counted as no, and a known smoking status). Of 220 who never smoked, 20 had diabetes; of 100 former smokers, 25; of 80 current smokers, 14. Test independence, show that for a table with two columns $X^2 = \sum_i n_{i+}(\hat p_i - \hat p)^2/(\hat p(1 - \hat p))$, and say where the departure lies.
Solution. The diabetes shares are $\hat p_i = 0.0909$, $0.2500$ and $0.1750$, and the pooled share is $\hat p = 59/400 = 0.1475$. The expected counts with diabetes are $220 \times 0.1475 = 32.45$, $14.75$ and $11.80$, and without, $187.55$, $85.25$ and $68.20$. Then $X^2 = 14.440$ on $(3 - 1)(2 - 1) = 2$ degrees of freedom, $p = 0.0007$, and $G^2 = 13.975$.
For the formula: in row $i$ the deviation in the diabetes column is $n_{i1} - n_{i+}\hat p = n_{i+}(\hat p_i - \hat p)$, and in the other column it is the negative of that, since the row's deviations sum to 0. So row $i$ contributes
$$n_{i+}^2(\hat p_i - \hat p)^2\Big(\frac{1}{n_{i+}\hat p} + \frac{1}{n_{i+}(1 - \hat p)}\Big) = \frac{n_{i+}(\hat p_i - \hat p)^2}{\hat p(1 - \hat p)}.$$
Numerically, $[220(0.0909 - 0.1475)^2 + 100(0.25 - 0.1475)^2 + 80(0.175 - 0.1475)^2]/(0.1475 \times 0.8525)$ gives the same 14.440. It is the statistic for comparing $r$ proportions, the $r$-group form of the two-proportion test.
The adjusted residuals in the diabetes column are $-3.53$ for never smokers, $3.34$ for former smokers and $0.78$ for current smokers (Pearson residuals $-2.19$, $2.67$ and $0.64$). The departure is a lower rate among those who never smoked and a higher rate among former smokers. In the population the three rates are 0.131, 0.229 and 0.123. That current smokers have about the rate of never smokers, and former smokers nearly twice it, is a warning against reading the table causally: former smokers are older on average, and diabetes rises steeply with age. The Mantel–Haenszel Test tests an association within strata of a third variable such as age. Cramér's $V$ for this table is 0.190.
Problem 2 (a small table, by permutation). The script drew 40 adults (seed 52201). By health (excellent to poor) and age group (18–39, 40–59, 60 and over) the counts were (3, 2, 2), (3, 1, 7), (4, 6, 5), (0, 5, 0) and (1, 1, 0). Test independence by the chi-squared approximation and by the permutation test.
Solution. The row totals are 7, 11, 15, 5 and 2, the column totals 11, 15 and 14. Of the 15 expected counts, 13 are below 5 and the smallest is 0.55. The statistic is $X^2 = 15.302$ on 8 degrees of freedom, and the chi-squared approximation gives $p = 0.0535$. Shuffling the age labels among the 40 adults, 10,000 times for each of 30 seeds (5,700,000 + $s$), gave a share of shuffled tables with $X^2 \ge 15.302$ with median 0.0421 (interquartile range 0.0405 to 0.0430). The margins admit 239,162 tables, few enough to list: summing the multiple hypergeometric probabilities of those with $X^2 \ge 15.302$ gives the exact conditional $p = 0.0417$, and the simulation's median agrees with it. The exact conditional test rejects at the 5% level and the approximation does not. More than half of the statistic, 8.33, comes from the fair row, where all 5 adults were 40 to 59. With expected counts this small the approximation's $p$-value is off in its second digit, and the exact conditional $p$-value is the one to report.
Problem 3 (attaining the bound). Find a $3 \times 3$ table of 100 observations with $V = 1$, and verify Theorem 6 on it.
Solution. Put each row in its own column: counts 30, 50 and 20 on the diagonal and zeros elsewhere. Then $n_{i+} = n_{+i}$ and every nonzero cell has $n_{ij} = n_{i+} = n_{+j}$, so $\sum n_{ij}^2/(n_{i+}n_{+j}) = 3$ and $X^2 = 100(3 - 1) = 200.0$, the bound $n(\min(r, c) - 1)$. So $V = \sqrt{200/(100 \times 2)} = 1$. Knowing the row tells the column exactly. A $3 \times 3$ table can also have each row in one column without being diagonal, for example rows that all fall in the first column; but then two columns are empty, the margins are not all positive, and $X^2$ is not defined.
Summary
| Quantity | Formula or result |
|---|---|
| Expected counts | $E_{ij} = n_{i+}n_{+j}/n$, the maximum likelihood fit under independence or homogeneity |
| Statistic | $X^2 = \sum_{i,j}(n_{ij} - E_{ij})^2/E_{ij}$ |
| Null limit | $\chi^2_{(r-1)(c-1)}$, for fixed row totals and for a fixed grand total |
| Likelihood ratio statistic | $G^2 = 2\sum n_{ij}\ln(n_{ij}/E_{ij}) = 2n\hat I$; $G^2 - X^2 \xrightarrow{p} 0$ |
| Conditional variance of a cell | $E_{ij}(1 - n_{i+}/n)(1 - n_{+j}/n)\,n/(n - 1)$ |
| Adjusted residual | $(n_{ij} - E_{ij})/\sqrt{E_{ij}(1 - n_{i+}/n)(1 - n_{+j}/n)}$, approximately standard normal |
| Exact test | shuffle one classification: the multiple hypergeometric distribution given both margins |
| Strength | $V = \sqrt{X^2/(n(\min(r, c) - 1))}$, between 0 and 1 |
What we have learned
The test of independence is Pearson's statistic with expected counts fitted from the margins. Whether the table came from one sample, from separate samples for each row, or with both margins fixed, conditioning turns each design into the next and the same statistic serves, with the chi-squared reference proved here for the first two designs and the exact conditional distribution available for the third. The fitted margins remove one direction within the rows and one across them, and the deviations keep $(r - 1)(c - 1)$ dimensions, which is Fisher's count. For 300 NHANES adults the test found self-rated health related to age group, $p = 0.0095$, through an association that Cramér's $V$ shows to be weak, and the adjusted residuals located it at the ends of the age range.
The test treats both classifications as unordered. When one of them is a yes-or-no outcome and the other is ordered, a test aimed at a trend can find what this test spreads over many degrees of freedom. That is the subject of Trends in Proportions: the Cochran–Armitage Test.
In Python
import numpy as np
from scipy import stats
from scipy.stats import contingency
t = np.array([[21, 15, 14], [39, 17, 16], [46, 28, 37], [14, 21, 24], [1, 1, 6]])
res = stats.chi2_contingency(t) # no Yates correction for tables larger than 2 x 2
print(res.statistic, res.dof, res.pvalue) # 20.222, 8, 0.0095
print(stats.chi2_contingency(t, lambda_='log-likelihood').statistic) # G2 = 20.186
print(contingency.association(t, method='cramer')) # V = 0.184
Every number in this article is computed and checked against the page by tools/experiments/statistics/chi-square-test-of-independence.py in the site's repository.
References
- Fisher, R. A. (1922). On the interpretation of $\chi^2$ from contingency tables, and the calculation of P. Journal of the Royal Statistical Society, 85(1), 87–94. https://doi.org/10.2307/2340521 record · 3 pages
- Haberman, S. J. (1973). The analysis of residuals in cross-classified tables. Biometrics, 29(1), 205 (first page). https://doi.org/10.2307/2529686 record
- Mantel, N. (1963). Chi-square tests with one degree of freedom; extensions of the Mantel–Haenszel procedure. Journal of the American Statistical Association, 58(303), 690–700. https://doi.org/10.1080/01621459.1963.10500879 record · 3 pages
- National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018: DEMO_J, HUQ_J, BPX_J, BPQ_J, BMX_J, DIQ_J and SMQ_J data files. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 75 pages
- SciPy documentation.
scipy.stats.chi2_contingencyandscipy.stats.contingency.association. https://docs.scipy.org/doc/scipy/reference/stats.html record