A study samples 200 obese and 200 non-obese adults and finds diabetes in 35 of the first group and 16 of the second, a relative risk of 2.188. The course page Intervals and Adjustment for Relative Risks and Odds Ratios gives the logarithm of that ratio the standard error $\sqrt{1/35 - 1/200 + 1/16 - 1/200} = 0.2847$, found by multiplying the variance of each sample proportion by the squared derivative of the logarithm. Why may a variance be carried through a function by its derivative? Because of a limit theorem, the delta method, which the course page states and cites and this article proves: if an estimate is asymptotically normal, so is any differentiable function of it, with the limiting standard deviation multiplied by the absolute value of the derivative.
The delta method is the name for that theorem and its relatives. The method is precise about limiting distributions and says nothing directly about the variance of the transformed estimate in a finite sample, a distinction that this article takes care to keep.
First we prove the first-order theorem, for one estimate, and its studentized form. Then we look at what it does not say about moments. Next we prove the second-order theorem, for the case in which the derivative is zero. Then we prove the theorem for functions of several estimates and apply it to two samples, to a table of four cells and to the sample variance. Finally we use it to construct variance-stabilizing transformations. The article assumes Convergence in Distribution and the Proof of the Central Limit Theorem, whose lemmas, continuous mapping theorem and Slutsky's theorem are used by name throughout.
The idea, and what has to be proved
The population of the running example is the 5,107 adults of the National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018 used on the course pages (National Center for Health Statistics, 2020). Of them 2,099 are obese, a share 0.4110. Diagnosed diabetes occurs in 432 of the obese, a risk $p_1 = 0.2058$, and in 345 of the other 3,008, a risk $p_2 = 0.1147$. The relative risk is 1.794 and the odds ratio 2.000.
Take a sample of $n$ obese adults, with replacement, and let $\hat{p}$ be the share with diabetes. By the central limit theorem $\sqrt{n}\,(\hat{p} - p_1)$ converges in distribution to the normal distribution with mean 0 and variance $p_1(1 - p_1)$. What can be said about $\ln\hat{p}$?
Here is the idea in one sentence: for large $n$ the estimate $\hat{p}$ is close to $p_1$, near $p_1$ the logarithm is almost a straight line of slope $1/p_1$, and a straight line multiplies deviations by its slope. Written as an approximation, $\ln\hat{p} - \ln p_1 \approx (\hat{p} - p_1)/p_1$. Two things have to be made exact. The first is the word "almost": the straight line is only an approximation to the function, and the error must be shown not to matter in the limit. The second is the word "close": the error of the linear approximation is small only where $\hat{p}$ is near $p_1$, and $\hat{p}$ is random. Slutsky's theorem takes care of both.
The first-order theorem
Theorem 1 (delta method). Let $Y_1, Y_2, \ldots$ be random variables, $\theta$ a number and $r_n$ positive constants with $r_n \to \infty$, such that
$$r_n\,(Y_n - \theta) \xrightarrow{d} X$$
for some random variable $X$. Let $g$ be a function defined on the whole real line and differentiable at $\theta$. Then
$$r_n\,\big(g(Y_n) - g(\theta)\big) \xrightarrow{d} g'(\theta)\,X, \qquad\text{and}\qquad r_n\,\big(g(Y_n) - g(\theta)\big) - g'(\theta)\,r_n\,(Y_n - \theta) \xrightarrow{p} 0.$$
In particular, if $\sqrt{n}\,(Y_n - \theta)$ converges in distribution to the normal distribution with mean 0 and variance $\sigma^2$, then $\sqrt{n}\,(g(Y_n) - g(\theta))$ converges in distribution to the normal distribution with mean 0 and variance $g'(\theta)^2\sigma^2$.
Here is the plan of the proof. First, $Y_n$ converges in probability to $\theta$, because its distance from $\theta$ is a bounded quantity divided by $r_n$. Second, the error of the straight-line approximation, measured relative to $Y_n - \theta$, is a function that is continuous and zero at $\theta$, so it converges in probability to 0. Third, the scaled error is that vanishing quantity times a sequence that is bounded in probability, and Slutsky's theorem removes it.
Proof. Step 1: $Y_n \xrightarrow{p} \theta$. The sequence $r_n(Y_n - \theta)$ converges in distribution, so it is bounded in probability (Lemma 2 of the article on convergence). The constants $1/r_n$ tend to 0. So the product $Y_n - \theta = (1/r_n) \cdot r_n(Y_n - \theta)$ converges in probability to 0 (Lemma 4 there).
Step 2: the relative error vanishes. Define
$$h(y) = \frac{g(y) - g(\theta)}{y - \theta} - g'(\theta) \quad\text{for } y \ne \theta, \qquad h(\theta) = 0.$$
That $g$ is differentiable at $\theta$ means exactly that $h(y) \to 0$ as $y \to \theta$, so $h$ is continuous at $\theta$. By Step 1 and the continuous mapping theorem for a function continuous at a constant limit (part 3), $h(Y_n) \xrightarrow{p} h(\theta) = 0$.
Step 3: Slutsky. By the definition of $h$, for every value of $Y_n$, including $Y_n = \theta$,
$$g(Y_n) - g(\theta) = g'(\theta)\,(Y_n - \theta) + h(Y_n)\,(Y_n - \theta).$$
Multiplying by $r_n$,
$$r_n\,\big(g(Y_n) - g(\theta)\big) = g'(\theta)\,r_n(Y_n - \theta) + h(Y_n)\,r_n(Y_n - \theta).$$
The last term is a sequence converging in probability to 0 times a sequence bounded in probability, so it converges in probability to 0 (Lemma 4 of the article on convergence). That is the second claim of the theorem. The first term converges in distribution to $g'(\theta)X$, by the continuous mapping theorem with the function $u \mapsto g'(\theta)u$. By the converging together lemma (Lemma 3 of the article on convergence), the sum converges in distribution to $g'(\theta)X$.
For the particular case, a normal variable with mean 0 and variance $\sigma^2$ multiplied by the constant $g'(\theta)$ is normal with mean 0 and variance $g'(\theta)^2\sigma^2$. $\square$
Four remarks on the statement.
The rate and the limit are general. The theorem does not need $\sqrt{n}$ or a normal limit. In the second worked problem of the article on convergence, the largest $M_n$ of $n$ uniform draws satisfies $n(1 - M_n) \xrightarrow{d} X$ with $X$ exponential. Theorem 1 with $r_n = n$ and $g(y) = y^2$ gives $n(1 - M_n^2) \xrightarrow{d} 2X$.
The function must be defined wherever the estimate can fall. The logarithm is not defined at 0, and a sample can have no cases. The theorem is stated for a function on the whole line, so the function has to be completed: let $g(y) = \ln y$ for $y > 0$ and $g(y) = 0$ otherwise. This $g$ is differentiable at every $\theta > 0$, which is all the theorem asks. The value chosen at 0 does not affect the limit, as the theorem shows by not mentioning it. This is the repair described on the course page.
Differentiability is needed only at the one point $\theta$. Nothing is assumed about $g$ elsewhere, not even continuity. The proof looks at $g$ only through $h$, near $\theta$.
If $g'(\theta) = 0$ the theorem is true and empty. The limit is then the constant 0: the scaling $r_n$ is too weak to show the variation of $g(Y_n)$. The second-order theorem below finds the right scaling.
Casella and Berger (2002) and van der Vaart (1998, Chapter 3) are the textbook references that the course page gives for the theorem. Ver Hoef (2012) reviews the modern versions and the history; its abstract reports that the earliest article on the method "was found in an obscure journal, and the author is rarely cited for his contribution".
The check on the running example
For the share $\hat{p}$ of diabetes among $n$ obese adults, $\sigma^2 = p_1(1 - p_1)$ and $g'(p_1) = 1/p_1$, so the theorem gives $\sqrt{n}\,(\ln\hat{p} - \ln p_1)$ a normal limit with variance
$$\frac{1}{p_1^2} \cdot p_1(1 - p_1) = \frac{1 - p_1}{p_1} = 3.859,$$
and standard deviation 1.964. Because the count is binomial, the distribution of $\ln\hat{p}$ can be computed exactly, with the completed logarithm. The table gives the largest gap between the distribution function of the standardized statistic and $\Phi$, for the transformed estimate and, for comparison, for the standardized $\hat{p}$ itself.
| $n$ | Largest gap for $\hat{p}$ | Largest gap for $\ln\hat{p}$ |
|---|---|---|
| 50 | 0.0833 | 0.0905 |
| 200 | 0.0417 | 0.0432 |
| 1,000 | 0.0186 | 0.0189 |
Both columns tend to 0, as the central limit theorem and Theorem 1 say they must. Most of each gap is the discreteness of a count: a distribution function that moves in jumps cannot stay closer to a continuous one than half its largest jump. The logarithm adds a little at $n = 50$ and almost nothing at $n = 1{,}000$. In these calculations the transformation costs little.
The studentized form
The limiting variance $g'(\theta)^2\sigma^2$ depends on the unknown $\theta$. An interval needs it estimated.
Corollary 1 (studentized delta method). Suppose $\sqrt{n}\,(Y_n - \theta)$ converges in distribution to the normal distribution with mean 0 and variance $\sigma^2 > 0$. Suppose $g$ is differentiable on an interval around $\theta$, with $g'$ continuous at $\theta$ and $g'(\theta) \ne 0$, and $\hat\sigma_n$ are random variables with $\hat\sigma_n \xrightarrow{p} \sigma$. Then
$$\frac{\sqrt{n}\,\big(g(Y_n) - g(\theta)\big)}{\lvert g'(Y_n)\rvert\,\hat\sigma_n} \xrightarrow{d} Z, \qquad Z \text{ standard normal},$$
where $g'$ is given the value 0 outside the interval and the ratio any fixed value when its denominator is 0. Consequently the interval $g(Y_n) \pm z^*\,\lvert g'(Y_n)\rvert\,\hat\sigma_n/\sqrt{n}$, with $z^* = \Phi^{-1}(1 - \alpha/2)$, contains $g(\theta)$ with a probability that tends to $1 - \alpha$.
Proof. By Step 1 of Theorem 1, $Y_n \xrightarrow{p} \theta$. The function $\lvert g'\rvert$, completed by 0 outside the interval, is continuous at $\theta$, so $\lvert g'(Y_n)\rvert \xrightarrow{p} \lvert g'(\theta)\rvert$. A product of two sequences that converge in probability to constants converges in probability to the product of the constants: with $A_n \xrightarrow{p} a$ and $B_n \xrightarrow{p} b$, write $A_nB_n - ab = (A_n - a)B_n + a(B_n - b)$, where $B_n$ is bounded in probability, and apply Lemma 4 of the article on convergence to the first term. So the denominator $D_n = \lvert g'(Y_n)\rvert\,\hat\sigma_n$ converges in probability to $\lvert g'(\theta)\rvert\,\sigma > 0$. By Theorem 1 the numerator converges in distribution to $g'(\theta)\sigma Z$. By the quotient form of Slutsky's theorem the ratio converges in distribution to $g'(\theta)\sigma Z/(\lvert g'(\theta)\rvert\sigma)$, which is $Z$ or $-Z$, and both are standard normal. The statement about the interval follows as in Corollary 2 of the article on convergence: the interval contains $g(\theta)$ exactly when the ratio lies between $-z^*$ and $z^*$, apart from the event $D_n = 0$, whose probability tends to 0. $\square$
For the logarithm of a proportion, $\hat\sigma_n^2 = \hat{p}(1 - \hat{p})$ converges in probability to $p(1 - p)$ by the continuous mapping theorem, and $\lvert g'(\hat{p})\rvert\,\hat\sigma_n/\sqrt{n} = \sqrt{(1 - \hat{p})/(n\hat{p})} = \sqrt{1/x - 1/n}$ for a count $x$. That is the standard error of the course page, and the corollary is the proof that it may be used with an estimated $p$.
What the theorem does not say: moments
The theorem is often paraphrased as "the variance of $g(Y_n)$ is approximately $g'(\theta)^2\operatorname{Var}(Y_n)$". That paraphrase is a statement about a moment, and the theorem is a statement about a limiting distribution. The two can differ without limit.
Example (a limiting variance where no variance exists). Let $\bar{X}_n$ be the mean of $n$ independent normal observations with mean $\mu \ne 0$ and standard deviation $\sigma$, and let $g(y) = 1/y$ for $y \ne 0$, $g(0) = 0$. Theorem 1 gives $\sqrt{n}\,(1/\bar{X}_n - 1/\mu)$ a normal limit with variance $\sigma^2/\mu^4$. But $1/\bar{X}_n$ has no variance, and not even a mean, for any $n$. The density of $\bar{X}_n$ is normal, so it is at least some positive number $m$ on an interval $(-\delta, \delta)$ around 0, and $E[\lvert 1/\bar{X}_n\rvert] \ge m\int_{-\delta}^{\delta} \lvert x\rvert^{-1}\,dx = \infty$. The limiting distribution describes where $1/\bar{X}_n$ falls with overwhelming probability. The mean is destroyed by the rare samples whose mean is nearly 0, which the limiting distribution ignores because their probability vanishes.
The same thing happens to $\ln\hat{p}$ in a milder form: with the completed logarithm its variance exists but depends on the arbitrary value assigned at 0. The quantity $g'(\theta)^2\sigma^2/n$ is best called the limiting variance or asymptotic variance of $g(Y_n)$: the variance of the normal distribution that approximates it.
When moments do exist, the limiting variance is usually close to the actual one, and the next term of the Taylor expansion suggests the bias. Expanding to second order and taking expected values formally,
$$E[g(Y_n)] - g(\theta) \approx \frac{g''(\theta)}{2}\,\frac{\sigma^2}{n}.$$
This is a heuristic, not a theorem; it needs conditions on the tails that the example above violates. For the logarithm of a proportion, $g''(p) = -1/p^2$ and the suggested bias is $-(1 - p)/(2np)$. The exact calculation, over samples with at least one case, compares as follows at $p_1 = 0.2058$.
| $n$ | $n \times$ exact variance of $\ln\hat{p}$ | Limiting value | Exact bias | Suggested bias |
|---|---|---|---|---|
| 50 | 4.540 | 3.859 | $-0.0422$ | $-0.0386$ |
| 200 | 3.997 | 3.859 | $-0.0098$ | $-0.0096$ |
| 1,000 | 3.885 | 3.859 | $-0.0019$ | $-0.0019$ |
At $n = 200$ the exact variance is 0.0200 against the limiting 0.0193, the two numbers quoted on the course page. The estimate $\ln\hat{p}$ is biased downward, since the logarithm is concave, and by about the suggested amount. In this example the heuristic works; the table is evidence for these three sample sizes and this $p$, not a proof.
The second-order theorem
Take the estimated variance of a single trial, $\hat{p}(1 - \hat{p})$. The function $g(y) = y(1 - y)$ has derivative $1 - 2y$, which is 0 at $p = \tfrac12$. There Theorem 1 says only that $\sqrt{n}\,(\hat{p}(1 - \hat{p}) - \tfrac14)$ converges to 0, which is true and uninformative. When the slope vanishes, the function is nearly a parabola near $\theta$, and a parabola turns deviations into their squares. Squares of deviations are of order $1/r_n^2$, so the scaling must be $r_n^2$, and the limit is the square of the original limit.
Theorem 2 (second-order delta method). Let $r_n(Y_n - \theta) \xrightarrow{d} X$ with $r_n \to \infty$. Let $g$ be defined on the whole line and differentiable on an interval around $\theta$, with a second derivative at $\theta$, and suppose $g'(\theta) = 0$. Then
$$r_n^2\,\big(g(Y_n) - g(\theta)\big) \xrightarrow{d} \frac{g''(\theta)}{2}\,X^2.$$
In particular, if $\sqrt{n}\,(Y_n - \theta)$ converges in distribution to the normal distribution with mean 0 and variance $\sigma^2$, then $n\,(g(Y_n) - g(\theta))$ converges in distribution to $\tfrac12 g''(\theta)\,\sigma^2$ times a chi-squared variable with 1 degree of freedom.
Proof. Step 1: the second-order remainder. Let $k(y) = g(y) - g(\theta) - \tfrac12 g''(\theta)(y - \theta)^2$, and define $h_2(y) = k(y)/(y - \theta)^2$ for $y \ne \theta$ and $h_2(\theta) = 0$. The claim is that $h_2$ is continuous at $\theta$. On the interval, $k$ is differentiable with $k'(y) = g'(y) - g''(\theta)(y - \theta)$. Since $g'(\theta) = 0$,
$$\frac{k'(y)}{y - \theta} = \frac{g'(y) - g'(\theta)}{y - \theta} - g''(\theta) \to 0 \quad\text{as } y \to \theta,$$
by the definition of the second derivative. So for any $\varepsilon > 0$ there is a $\delta > 0$, small enough that the points within $\delta$ of $\theta$ lie in the interval, with $\lvert k'(u)\rvert \le \varepsilon\,\lvert u - \theta\rvert$ whenever $\lvert u - \theta\rvert < \delta$. For such a $y \ne \theta$, the mean value theorem gives a point $\xi$ between $\theta$ and $y$ with $k(y) = k(y) - k(\theta) = k'(\xi)(y - \theta)$, so
$$\lvert k(y)\rvert \le \varepsilon\,\lvert\xi - \theta\rvert\,\lvert y - \theta\rvert \le \varepsilon\,(y - \theta)^2.$$
Hence $\lvert h_2(y)\rvert \le \varepsilon$ within $\delta$ of $\theta$, which is the claim.
Step 2: Slutsky. As in Theorem 1, $Y_n \xrightarrow{p} \theta$, so $h_2(Y_n) \xrightarrow{p} 0$. By the definitions,
$$r_n^2\,\big(g(Y_n) - g(\theta)\big) = \frac{g''(\theta)}{2}\,\big[r_n(Y_n - \theta)\big]^2 + h_2(Y_n)\,\big[r_n(Y_n - \theta)\big]^2.$$
By the continuous mapping theorem with the function $u \mapsto u^2$, $[r_n(Y_n - \theta)]^2 \xrightarrow{d} X^2$; in particular that sequence is bounded in probability. So the second term converges in probability to 0, the first converges in distribution to $\tfrac12 g''(\theta)X^2$, and Lemma 3 of the article on convergence gives the result.
For the particular case, $X = \sigma Z$ with $Z$ standard normal, and $Z^2$ is chi-squared with 1 degree of freedom by the definition in The Chi-Squared and t Distributions. $\square$
The check. At $p = \tfrac12$, $g(y) = y(1 - y)$ has $g'' = -2$ and $\sigma^2 = \tfrac14$, so the theorem gives $n\,(\hat{p}(1 - \hat{p}) - \tfrac14) \xrightarrow{d} -\tfrac14\chi^2_1$. Here the algebra is exact, $\hat{p}(1 - \hat{p}) = \tfrac14 - (\hat{p} - \tfrac12)^2$, so the statement says that $4n(\hat{p} - \tfrac12)^2$ is approximately $\chi^2_1$. The chi-squared distribution with 1 degree of freedom puts probability 0.95 below $1.96^2 = 3.841$. The exact binomial probability that $4n(\hat{p} - \tfrac12)^2 \le 3.841$ is 0.9586 for $n = 20$, 0.9431 for $n = 100$, 0.9463 for $n = 1{,}000$ and 0.9488 for $n = 10{,}000$. The estimate $\hat{p}(1 - \hat{p})$ can only fall below $\tfrac14$, and its limiting distribution is one-sided, as the negative sign in the limit says.
Example (a nearly vanishing slope). At the obesity share $p = 0.4110$ the slope of $g(y) = y(1 - y)$ is $1 - 2p = 0.178$, small but not zero. Theorem 1 applies: $\sqrt{n}\,(\hat{p}(1 - \hat{p}) - p(1 - p))$ has a normal limit with standard deviation $0.178\sqrt{0.2421} = 0.0876$. But the statistic can never exceed $\tfrac14$, which is only 0.0079 above $p(1 - p) = 0.2421$. Standardized, the statistic $W_n$ is bounded above by $\sqrt{n} \times 0.0079/0.0876$, which is 0.90 at $n = 100$, 2.86 at $n = 1{,}000$ and 9.04 at $n = 10{,}000$. A variable that cannot exceed 0.90 is far from normal.
| $n$ | Largest gap to $\Phi$ | $P(W_n > 0)$ | Upper bound of $W_n$ |
|---|---|---|---|
| 100 | 0.1829 | 0.4650 | 0.90 |
| 1,000 | 0.0333 | 0.4865 | 2.86 |
| 10,000 | 0.0099 | 0.4961 | 9.04 |
| 100,000 | 0.0031 | 0.4998 |
The first-order limit is correct and arrives slowly, because the second-order term of Theorem 2 competes with a first-order term whose coefficient is small. A reading consistent with the table is that the first-order approximation becomes accurate once the typical deviation of $\hat{p}$, about $\sqrt{p(1-p)/n}$, is small compared with the distance from $p$ to the point where the slope vanishes; that is a description of these numbers, not a theorem.
Functions of several estimates
A relative risk is a function of two proportions, an odds ratio of four cell shares, a sample variance of a mean and a mean of squares. The theorem extends to functions of several variables with the derivative replaced by the gradient. For a point $\boldsymbol{\theta}$ in $k$ dimensions, a function $g$ of $k$ variables is differentiable at $\boldsymbol{\theta}$ with gradient $\nabla g(\boldsymbol{\theta})$ if
$$g(\mathbf{y}) = g(\boldsymbol{\theta}) + \nabla g(\boldsymbol{\theta})\cdot(\mathbf{y} - \boldsymbol{\theta}) + \lVert\mathbf{y} - \boldsymbol{\theta}\rVert\,h(\mathbf{y}), \qquad h(\mathbf{y}) \to 0 \text{ as } \mathbf{y} \to \boldsymbol{\theta}, \quad h(\boldsymbol{\theta}) = 0,$$
where $\lVert\cdot\rVert$ is the Euclidean length. The gradient is then the vector of partial derivatives. Two kinds of function, which cover every example below, are differentiable in this sense by a one-line check.
- A sum of differentiable functions of single coordinates, $g(\mathbf{y}) = g_1(y_1) + \cdots + g_k(y_k)$. With $h_j$ the one-variable relative error of Theorem 1 for $g_j$, the remainder is $\sum_j h_j(y_j)(y_j - \theta_j)$, at most $\lVert\mathbf{y} - \boldsymbol{\theta}\rVert\sum_j\lvert h_j(y_j)\rvert$ in absolute value.
- The function $g(y_1, y_2) = y_2 - y_1^2$. Its gradient at $\boldsymbol{\theta}$ is $(-2\theta_1, 1)$, and the remainder is exactly $-(y_1 - \theta_1)^2$, at most $\lVert\mathbf{y} - \boldsymbol{\theta}\rVert^2$ in absolute value.
Theorem 3 (delta method for several variables). Let $\mathbf{Y}_n$ be random vectors with $k$ components, $\boldsymbol{\theta}$ a point, $r_n \to \infty$ positive constants and $\mathbf{X}$ a random vector, such that for every vector of constants $\mathbf{a}$,
$$r_n\;\mathbf{a}\cdot(\mathbf{Y}_n - \boldsymbol{\theta}) \xrightarrow{d} \mathbf{a}\cdot\mathbf{X}.$$
Let $g$ be a function of $k$ variables, defined everywhere and differentiable at $\boldsymbol{\theta}$. Then
$$r_n\,\big(g(\mathbf{Y}_n) - g(\boldsymbol{\theta})\big) \xrightarrow{d} \nabla g(\boldsymbol{\theta})\cdot\mathbf{X}.$$
In particular, if there is a matrix $\Sigma$ such that every $\mathbf{a}\cdot\mathbf{X}$ is normal with mean 0 and variance $\mathbf{a}^{\mathsf{T}}\Sigma\,\mathbf{a}$, the limit is normal with mean 0 and variance $\nabla g(\boldsymbol{\theta})^{\mathsf{T}}\,\Sigma\,\nabla g(\boldsymbol{\theta})$.
Proof. Taking $\mathbf{a}$ to be each coordinate vector in turn, every coordinate $r_n(Y_{n,j} - \theta_j)$ converges in distribution, so each is bounded in probability. The length satisfies $\lVert r_n(\mathbf{Y}_n - \boldsymbol{\theta})\rVert \le \sum_j \lvert r_n(Y_{n,j} - \theta_j)\rvert$, and a sum of $k$ sequences bounded in probability is bounded in probability: if each exceeds $M$ with probability below $\eta/k$, the sum exceeds $kM$ with probability below $\eta$. Dividing by $r_n$, $\lVert\mathbf{Y}_n - \boldsymbol{\theta}\rVert \xrightarrow{p} 0$ by Lemma 4 of the article on convergence. Since $h$ is continuous at $\boldsymbol{\theta}$ with $h(\boldsymbol{\theta}) = 0$, for every $\varepsilon > 0$ there is a $\delta > 0$ with $\lvert h(\mathbf{y})\rvert \le \varepsilon$ whenever $\lVert\mathbf{y} - \boldsymbol{\theta}\rVert \le \delta$, so $P(\lvert h(\mathbf{Y}_n)\rvert > \varepsilon) \le P(\lVert\mathbf{Y}_n - \boldsymbol{\theta}\rVert > \delta) \to 0$. By the definition of differentiability,
$$r_n\,\big(g(\mathbf{Y}_n) - g(\boldsymbol{\theta})\big) = r_n\,\nabla g(\boldsymbol{\theta})\cdot(\mathbf{Y}_n - \boldsymbol{\theta}) + h(\mathbf{Y}_n)\,\lVert r_n(\mathbf{Y}_n - \boldsymbol{\theta})\rVert.$$
The first term converges in distribution to $\nabla g(\boldsymbol{\theta})\cdot\mathbf{X}$, by the hypothesis with $\mathbf{a} = \nabla g(\boldsymbol{\theta})$. The second converges in probability to 0 by the same Lemma 4, and Lemma 3 of the article on convergence gives the result. The particular case is the general one with $\mathbf{a} = \nabla g(\boldsymbol{\theta})$. $\square$
The hypothesis is stated for linear combinations so that the proof needs only the convergence of real random variables. It is equivalent to the convergence in distribution of the vectors $r_n(\mathbf{Y}_n - \boldsymbol{\theta})$ to $\mathbf{X}$: the vectors converge if the linear combinations do, by the Cramér–Wold device, which is cited in the article on convergence. The other direction is immediate from the definition of convergence in distribution for vectors given there: if $h$ is a bounded continuous function on the line, then $\mathbf{y} \mapsto h(\mathbf{a}\cdot\mathbf{y})$ is a bounded continuous function of $k$ variables. A random vector all of whose linear combinations are normal in this way is, by definition, multivariate normal with covariance matrix $\Sigma$; Multivariate Statistics is the home of that distribution, and nothing about it is needed here beyond the linear combinations. For a mean of i.i.d. vectors with finite variances the hypothesis holds with $r_n = \sqrt{n}$ and $\Sigma$ the covariance matrix of one observation: that is Corollary 3 of the article on convergence, proved there. So for functions of means, Theorem 3 rests on nothing that is cited except the two theorems on characteristic functions behind the central limit theorem.
In coordinates, the limiting variance is
$$\nabla g^{\mathsf{T}}\,\Sigma\,\nabla g = \sum_{j}\sum_{l} \frac{\partial g}{\partial y_j}\,\frac{\partial g}{\partial y_l}\,\Sigma_{jl},$$
and when the components of $\mathbf{Y}_n$ are independent, $\Sigma$ is diagonal and the variance is the sum of the one-variable terms $(\partial g/\partial y_j)^2\,\Sigma_{jj}$.
The log relative risk
Two samples of the same size $n$, one from each exposure group, can be laid side by side as $n$ pairs $(X_{1i}, X_{2i})$ of indicators of the outcome. The pairs are i.i.d. vectors whose components are independent, with means $(p_1, p_2)$ and covariance matrix $\Sigma$ with diagonal entries $p_1(1 - p_1)$ and $p_2(1 - p_2)$ and zeros elsewhere. Their mean is $(\hat{p}_1, \hat{p}_2)$. The function $g(y_1, y_2) = \ln y_1 - \ln y_2$, with each logarithm completed at 0, is a sum of differentiable functions of single coordinates, with gradient $(1/p_1, -1/p_2)$. Theorem 3 gives
$$\sqrt{n}\,\big(\ln\widehat{\text{RR}} - \ln\text{RR}\big) \xrightarrow{d} \text{normal, mean 0, variance } \frac{1 - p_1}{p_1} + \frac{1 - p_2}{p_2}.$$
For the diabetes risks the two terms are 3.859 and 7.719, and the variance is 11.578. Estimating each $p$ by its sample share changes the variance by a factor that converges in probability to 1, so by Slutsky's theorem the standardized statistic keeps its normal limit, as in Corollary 1. The estimate turns the variance of $\ln\widehat{\text{RR}}$ into $1/a - 1/n + 1/c - 1/n$ for $a$ and $c$ cases: the Katz formula of the opening (Katz et al., 1978). The exact largest gap between the distribution function of the standardized $\ln\widehat{\text{RR}}$ and $\Phi$, computed from the two binomial distributions, is 0.0492 for $n = 50$ per group, 0.0249 for $n = 200$ and 0.0101 for $n = 1{,}000$.
Unequal sample sizes need one more fact, since the two samples can no longer be paired.
Lemma (independent sums). If $U_n \xrightarrow{d} U$ and $V_n \xrightarrow{d} V$, and $U_n$ and $V_n$ are independent for each $n$, then $U_n + V_n$ converges in distribution to the sum of independent variables distributed as $U$ and $V$.
Proof. By the converse of the continuity theorem, proved in the article on convergence, $\varphi_{U_n}(t) \to \varphi_U(t)$ and $\varphi_{V_n}(t) \to \varphi_V(t)$ for every $t$. By independence $\varphi_{U_n + V_n}(t) = \varphi_{U_n}(t)\,\varphi_{V_n}(t) \to \varphi_U(t)\,\varphi_V(t)$, which is the characteristic function of the sum of independent copies of $U$ and $V$. The continuity theorem, cited there, gives the convergence in distribution. $\square$
Now let the sample sizes $n_1$ and $n_2$ both tend to infinity with $n_1/n_2 \to \rho$, a positive number. Write $v_j = (1 - p_j)/p_j$ and $s^2 = v_1/n_1 + v_2/n_2$. By Theorem 1, $\sqrt{n_1}\,(\ln\hat{p}_1 - \ln p_1)$ converges in distribution to a normal variable with variance $v_1$, and the constants $1/(\sqrt{n_1}\,s) = (v_1 + v_2 n_1/n_2)^{-1/2}$ tend to $(v_1 + v_2\rho)^{-1/2}$. By Slutsky's theorem, $U_n = (\ln\hat{p}_1 - \ln p_1)/s$ has a normal limit with variance $v_1/(v_1 + v_2\rho)$. In the same way $V_n = -(\ln\hat{p}_2 - \ln p_2)/s$ has a normal limit with variance $v_2\rho/(v_1 + v_2\rho)$. The two samples are independent, so by the lemma $U_n + V_n = (\ln\widehat{\text{RR}} - \ln\text{RR})/s$ converges in distribution to a normal variable with variance $(v_1 + v_2\rho)/(v_1 + v_2\rho) = 1$, a sum of independent normal variables being normal. With each $p_j$ estimated by its sample share, the estimated standard error $\hat{s}$ satisfies $\hat{s}/s \xrightarrow{p} 1$: each estimated $v_j$ converges in probability to $v_j$ by the continuous mapping theorem, and $\hat{s}^2/s^2$ lies between the smaller and the larger of the two ratios of estimated to true $v_j$. The quotient form of Slutsky's theorem then keeps the limit. So the Katz standard error is right for unequal groups as well.
The log odds ratio from a single sample
The course page derives Woolf's standard error for the log odds ratio, $\sqrt{1/a + 1/b + 1/c + 1/d}$, for two independent samples, one per exposure group, by adding the variances of two log odds. A cross-sectional study has a different design: one sample of $n$ people, each classified by exposure and outcome into one of four cells. The four counts are then not independent, since they add to $n$. Does the same formula hold? It does, and Theorem 3 shows why.
Let the cell probabilities be $\pi_1, \pi_2, \pi_3, \pi_4$ for exposed with the outcome, exposed without, unexposed with, and unexposed without. Each person contributes a vector of four indicators, exactly one of which is 1. These vectors are i.i.d. with mean $\boldsymbol{\pi}$. The variance of the $j$th indicator is $\pi_j(1 - \pi_j)$, and since two different indicators are never both 1, the covariance of the $j$th and $l$th is $0 - \pi_j\pi_l$. So $\Sigma_{jj} = \pi_j - \pi_j^2$ and $\Sigma_{jl} = -\pi_j\pi_l$: in matrix form, $\Sigma$ is the diagonal matrix of the $\pi_j$ minus $\boldsymbol{\pi}\boldsymbol{\pi}^{\mathsf{T}}$. The sample shares $\hat{\boldsymbol{\pi}}$ are the mean of the vectors, and the log odds ratio is
$$g(\hat{\boldsymbol{\pi}}) = \ln\hat\pi_1 - \ln\hat\pi_2 - \ln\hat\pi_3 + \ln\hat\pi_4,$$
a sum of functions of single coordinates with gradient $\nabla g = (1/\pi_1, -1/\pi_2, -1/\pi_3, 1/\pi_4)$. The limiting variance is
$$\nabla g^{\mathsf{T}}\Sigma\,\nabla g = \sum_j \Big(\frac{\partial g}{\partial y_j}\Big)^2\pi_j - \big(\nabla g\cdot\boldsymbol{\pi}\big)^2 = \sum_{j=1}^{4}\frac{1}{\pi_j} - (1 - 1 - 1 + 1)^2 = \sum_{j=1}^{4}\frac{1}{\pi_j}.$$
The correction for the dependence among the cells is $(\nabla g\cdot\boldsymbol{\pi})^2$, and it vanishes because the signs in the log odds ratio cancel. So $\sqrt{n}\,(\ln\widehat{\text{OR}} - \ln\text{OR})$ has a normal limit with variance $\sum_j 1/\pi_j$. With each $\pi_j$ estimated by its cell count over $n$, the variance of $\ln\widehat{\text{OR}}$ becomes $1/a + 1/b + 1/c + 1/d$, Woolf's formula (Woolf, 1955), now for a single multinomial sample.
For the population's four cells, 432, 1,667, 345 and 2,663 out of 5,107, the sum $\sum_j 1/\pi_j$ is 31.606, so for a cross-sectional sample of 400 the limiting standard error of $\ln\widehat{\text{OR}}$ is $\sqrt{31.606/400} = 0.2811$. The script drew 10,000 such samples for each of 30 seeds. The standard deviation of $\ln\widehat{\text{OR}}$ across samples had median 0.2873 over the seeds (interquartile range 0.2865 to 0.2887), about 2% above the limiting value; no sample had an empty cell.
The sample variance of a population that is not normal
For a normal sample, $(n - 1)s_n^2/\sigma^2$ has a chi-squared distribution (The Chi-Squared and t Distributions), whose variance $2(n - 1)$ gives $\operatorname{Var}(s_n^2) = 2\sigma^4/(n - 1)$. For any other population the delta method gives the limit.
Corollary 2 (the sample variance). Let $X_1, X_2, \ldots$ be i.i.d. with mean $\mu$, variance $\sigma^2 > 0$ and finite fourth central moment $\mu_4 = E[(X - \mu)^4]$, and let $\kappa = \mu_4/\sigma^4$. Then
$$\sqrt{n}\,(s_n^2 - \sigma^2) \xrightarrow{d} \text{normal, mean 0, variance } \mu_4 - \sigma^4 = (\kappa - 1)\,\sigma^4,$$
and $\sqrt{n}\,(\ln s_n^2 - \ln\sigma^2)$ has a normal limit with variance $\kappa - 1$.
Proof. The sample variance is unchanged when a constant is subtracted from every observation, so assume $\mu = 0$. The vectors $(X_i, X_i^2)$ are i.i.d. with mean $(0, \sigma^2)$ and finite variances, the variance of $X_i^2$ being $\mu_4 - \sigma^4$. Their mean is $(\bar{X}_n, V_n)$ with $V_n = \frac1n\sum X_i^2$. With $g(y_1, y_2) = y_2 - y_1^2$, whose gradient at $(0, \sigma^2)$ is $(0, 1)$, Theorem 3 gives $\sqrt{n}\,(V_n - \bar{X}_n^2 - \sigma^2)$ a normal limit with variance $\Sigma_{22} = \mu_4 - \sigma^4$. Finally $s_n^2 = \frac{n}{n-1}(V_n - \bar{X}_n^2)$, so
$$\sqrt{n}\,(s_n^2 - \sigma^2) = \frac{n}{n - 1}\,\sqrt{n}\,(V_n - \bar{X}_n^2 - \sigma^2) + \frac{\sqrt{n}}{n - 1}\,\sigma^2,$$
and Slutsky's theorem, with the constants $n/(n-1) \to 1$ and $\sqrt{n}\,\sigma^2/(n - 1) \to 0$, gives the first claim. The second follows from Theorem 1 with $g = \ln$, completed at 0, whose derivative at $\sigma^2$ is $1/\sigma^2$. $\square$
For a normal population $\kappa = 3$ (The Chi-Squared and t Distributions computes $E[Z^4] = 3$), and the limiting variance $2\sigma^4$ agrees with the chi-squared result. For any other $\kappa$ the chi-squared interval for a variance has the wrong width, however large the sample.
Corollary 3 (the chi-squared interval under a non-normal population). Under the hypotheses of Corollary 2 with $\kappa > 1$, the 95% interval for $\sigma^2$ built from the chi-squared distribution, $[(n-1)s_n^2/b_n,\ (n-1)s_n^2/a_n]$ with $a_n$ and $b_n$ the points with probability 0.025 below and above under $\chi^2_{n-1}$, contains $\sigma^2$ with a probability that tends to
$$2\,\Phi\Big(1.96\,\sqrt{\frac{2}{\kappa - 1}}\Big) - 1.$$
Proof. Write $\nu = n - 1$. The interval contains $\sigma^2$ exactly when $a_n \le \nu s_n^2/\sigma^2 \le b_n$, that is, when
$$\frac{a_n - \nu}{\sqrt{2\nu}} \le W_n \le \frac{b_n - \nu}{\sqrt{2\nu}}, \qquad W_n = \sqrt{\frac{\nu}{2}}\,\Big(\frac{s_n^2}{\sigma^2} - 1\Big).$$
By Corollary 2 and Slutsky's theorem, $W_n$ converges in distribution to a normal variable $W$ with mean 0 and variance $(\kappa - 1)/2$. The end points converge to $\mp 1.96$. To see it, let $V$ be a $\chi^2_\nu$ variable. It is a sum of $\nu$ i.i.d. squared standard normal variables with mean 1 and variance 2, so by the central limit theorem and the theorem on uniform convergence to a continuous limit, the distribution function of $(V - \nu)/\sqrt{2\nu}$ converges to $\Phi$ uniformly. Its value at $(a_n - \nu)/\sqrt{2\nu}$ is 0.025 for every $n$, so $\Phi((a_n - \nu)/\sqrt{2\nu}) \to 0.025$, and since $\Phi$ has a continuous inverse, $(a_n - \nu)/\sqrt{2\nu} \to -1.96$. The upper end point is treated in the same way. Subtracting the end points from $W_n$ and applying Slutsky's theorem to each difference, $P(W_n \ge (a_n - \nu)/\sqrt{2\nu}) \to P(W \ge -1.96)$ and $P(W_n > (b_n - \nu)/\sqrt{2\nu}) \to P(W > 1.96)$, by Lemma 1 of the article on convergence. The probability of the interval's event is the difference, which tends to $P(-1.96 \le W \le 1.96) = 2\Phi(1.96/\sqrt{(\kappa - 1)/2}) - 1$. $\square$
The check on the blood pressures. The 5,247 adults with a blood pressure reading in the same survey are the unit's other population. Sampled with replacement, they have $\sigma^2 = 405.01$ and $\kappa = 4.680$, heavier tails than the normal's 3. The limiting variance of $\sqrt{n}\,(s_n^2 - \sigma^2)$ is $\mu_4 - \sigma^4 = 603{,}580$, against the $2\sigma^4 = 328{,}058$ of normal theory, a ratio $(\kappa - 1)/2 = 1.840$. The sample variance has a standard deviation $\sqrt{1.840} = 1.356$ times what the chi-squared distribution assumes. By Corollary 3 the chi-squared interval's coverage tends to 0.852, not 0.95.
The script drew 10,000 samples of 400 for each of 30 seeds. Three results, each a median over the seeds with its interquartile range:
- $n$ times the variance of $s_n^2$ across samples was 607,725 (598,906 to 610,111). The exact value for $n = 400$, from the formula in One-Sided Confidence Bounds and Sample Size, is 604,402, and the limit is 603,580.
- The chi-squared interval for $\sigma^2$ contained it in a share 0.8516 of samples (0.8500 to 0.8540), in line with the limit 0.852.
- The interval $\ln s_n^2 \pm 1.96\sqrt{(\hat\kappa - 1)/n}$, from Corollary 2 with the sample's own fourth-moment ratio $\hat\kappa$ in place of $\kappa$, contained $\ln\sigma^2$ in a share 0.9378 of samples (0.9359 to 0.9387).
So for these blood pressures, at a sample size where the $t$ interval for the mean is accurate (5.07% two-sided misses at $n = 400$ in the simulation of Moment Generating Functions and Slutsky's Theorem), the chi-squared interval for the variance misses about three times as often as it claims, and it will not improve with more data. The delta-method interval is close to its nominal level, a little below it, with the kurtosis estimated from the same 400 values. This is the precise form of the warning in the second worked problem of The Chi-Squared and t Distributions.
Variance-stabilizing transformations
In the examples so far the function $g$ was given and the theorem supplied the limiting variance. The theorem can be run the other way: choose $g$ to make the limiting variance come out as wanted. Suppose the limiting variance of $\sqrt{n}\,(Y_n - \theta)$ is a known function $\sigma^2(\theta)$ of the parameter, as it is for a proportion, where $\sigma^2(p) = p(1 - p)$. Then the precision of $Y_n$ depends on the unknown $\theta$, which complicates intervals and comparisons between groups. A variance-stabilizing transformation is a function $g$ for which the limiting variance of $\sqrt{n}\,(g(Y_n) - g(\theta))$ is a constant $c^2$, the same for every $\theta$. By Theorem 1 that requires
$$g'(\theta)\,\sigma(\theta) = c, \qquad\text{that is,}\qquad g(\theta) = \int \frac{c}{\sigma(\theta)}\,d\theta.$$
A proportion. With $\sigma(p) = \sqrt{p(1 - p)}$ and $c = \tfrac12$, $g(p) = \int \frac{dp}{2\sqrt{p(1 - p)}} = \arcsin\sqrt{p}$. To confirm, the derivative of $\arcsin u$ is $1/\sqrt{1 - u^2}$, so by the chain rule the derivative of $\arcsin\sqrt{p}$ is $\frac{1}{\sqrt{1 - p}} \cdot \frac{1}{2\sqrt{p}}$. Theorem 1 then gives, for every $p$ strictly between 0 and 1,
$$\sqrt{n}\,\big(\arcsin\sqrt{\hat{p}} - \arcsin\sqrt{p}\big) \xrightarrow{d} \text{normal, mean 0, variance } \tfrac14.$$
A Poisson count. The mean $\bar{X}_n$ of $n$ i.i.d. Poisson counts with mean $\lambda$ has $\sigma(\lambda) = \sqrt{\lambda}$. With $c = \tfrac12$, $g(\lambda) = \int \frac{d\lambda}{2\sqrt{\lambda}} = \sqrt{\lambda}$, so $\sqrt{n}\,(\sqrt{\bar{X}_n} - \sqrt{\lambda})$ has a normal limit with variance $\tfrac14$. The total $n\bar{X}_n$ is itself a Poisson count with mean $n\lambda$, since independent Poisson counts add (The Poisson Distribution), and $\sqrt{n\bar{X}_n} - \sqrt{n\lambda} = \sqrt{n}\,(\sqrt{\bar{X}_n} - \sqrt{\lambda})$. So the square root of a single Poisson count with a large mean is approximately normal with limiting variance $\tfrac14$, whatever the mean. That its actual variance is also close to $\tfrac14$ is a separate matter, as the section on moments explained; the exact calculation below shows it for means from 10 up. Bartlett (1936) discussed the square root transformation in the analysis of variance; the review of the paper in the Jahrbuch über die Fortschritte der Mathematik (JFM 63.1085.01) reports that he found a variance near $\tfrac14$ for the square root of a Poisson count with a large mean, and supported it with numerical tables in place of a proof. Anscombe (1948) treated transformations of Poisson, binomial and negative binomial data.
A variance. By Corollary 2 the limiting standard deviation of $s_n^2$ is $\sqrt{\kappa - 1}\,\sigma^2$, proportional to the quantity estimated. With $\sigma(\theta) = \sqrt{\kappa - 1}\,\theta$, $g(\theta) = \int d\theta/\theta = \ln\theta$: the logarithm stabilizes the variance of a sample variance, which is why Corollary 2 and the interval above work on the log scale. The same holds for any estimate whose standard deviation is proportional to its target, such as a rate (Crude Rates and Their Precision).
The check. Stabilization is a statement about limits. The table gives exact variances. For the proportion, the unstabilized $n\operatorname{Var}(\hat{p}) = p(1 - p)$ varies fivefold across the rows, while $4n\operatorname{Var}(\arcsin\sqrt{\hat{p}})$, whose limit is 1 in every row, stays near 1 except for small $np$.
| $p$ | $n\operatorname{Var}(\hat{p})$ | $4n\operatorname{Var}(\arcsin\sqrt{\hat{p}})$, $n = 50$ | $n = 200$ |
|---|---|---|---|
| 0.05 | 0.0475 | 1.444 | 1.044 |
| 0.1 | 0.0900 | 1.132 | 1.020 |
| 0.2 | 0.1600 | 1.043 | 1.009 |
| 0.3 | 0.2100 | 1.028 | 1.007 |
| 0.4 | 0.2400 | 1.022 | 1.005 |
| 0.5 | 0.2500 | 1.021 | 1.005 |
For a Poisson count $X$ with mean $\lambda$, the variance of $X$ is $\lambda$, and $4\operatorname{Var}(\sqrt{X})$ is 1.609 at $\lambda = 1$, 1.560 at $\lambda = 2$, 1.144 at $\lambda = 5$, 1.045 at $\lambda = 10$, 1.020 at $\lambda = 20$, 1.008 at $\lambda = 50$ and 1.004 at $\lambda = 100$. In this table the transformation is within 5% of its limit once the mean is 10 or more, and it fails for means near 1, where a count is too coarse for any smooth function to help. With 50 trials and $p = 0.05$, an expected count of 2.5, the arcsine transformation is off by 44%.
What is proved, what is cited, what is computed
Proved here. Theorems 1, 2 and 3 and the three corollaries, from the lemmas, the continuous mapping theorem and Slutsky's theorem of Convergence in Distribution and the Proof of the Central Limit Theorem. The limiting variances of the log relative risk, for equal and unequal samples, of the log odds ratio under multinomial sampling, and of the sample variance. The construction of the arcsine, square root and logarithmic transformations.
Cited. Through the article on convergence: the continuity theorem for characteristic functions, on which the central limit theorem and the lemma on independent sums rest, and the Cramér–Wold device, used only to relate the hypothesis of Theorem 3 to convergence of vectors. This unit avoids measure theory, as the article on convergence explains; nothing in the present article needs more than that one does.
Computed. The tables for $\ln\hat{p}$, for $\hat{p}(1 - \hat{p})$, for the log relative risk and for the transformations are exact calculations from binomial and Poisson probabilities. The results for the log odds ratio and for the sample variance of blood pressures are seeded simulations, and are observations about those populations at those sample sizes. The second-order approximation to the bias is a heuristic, checked in one case.
Worked problems
These problems were written for this article, on the course pages' own samples: the 200 obese and 200 non-obese adults of the opening, and the ten sleep differences of Student (1908). The script tools/experiments/statistics/delta-method.py recomputes every number.
Problem 1 (the relative risk with and without the logarithm). For the sample of the opening, 35 cases among 200 obese adults and 16 among 200 others, find the limiting standard error of the relative risk itself, build the interval $\widehat{\text{RR}} \pm 1.96\,\text{SE}$, and compare it with the interval built on the log scale. Which should be preferred?
Solution. The untransformed interval is $(0.967,\ 3.408)$ and the log-scale interval is $(1.252,\ 3.822)$. The log-scale interval should be preferred.
The relative risk is the exponential of its logarithm, so Theorem 1 applies with $Y_n = \ln\widehat{\text{RR}}$, $\theta = \ln\text{RR}$ and $g(y) = e^y$, whose derivative at $\theta$ is $e^\theta = \text{RR}$. Since $\sqrt{n}\,(\ln\widehat{\text{RR}} - \ln\text{RR})$ has a normal limit with variance $v = 11.578$, as found above, $\sqrt{n}\,(\widehat{\text{RR}} - \text{RR})$ has a normal limit with variance $\text{RR}^2 v$. The standard error of $\widehat{\text{RR}}$ is therefore $\widehat{\text{RR}}$ times the standard error of its logarithm: $2.188 \times 0.2847 = 0.6228$. The interval is $2.188 \pm 1.96 \times 0.6228 = (0.967,\ 3.408)$. The log-scale interval of the course page is $2.188 \times e^{\pm 1.96 \times 0.2847} = (1.252,\ 3.822)$.
Both intervals are justified by the same theorem and have the same limiting coverage, 95%. They disagree on a matter of substance: the first contains 1, no association, and the second does not. The theorem cannot choose between them; a finite-sample calculation can. With 200 adults per group at the population risks, the exact coverage of the log-scale interval is 0.954, as on the course page, and that of the untransformed interval is 0.946. More telling is where the misses fall. The log-scale interval lies entirely below the true ratio with probability 0.026 and entirely above it with probability 0.020, nearly balanced. The untransformed interval lies entirely below the true ratio with probability 0.053 and entirely above it with probability 0.000 to three places. The sampling distribution of a ratio is skewed to the right, and a symmetric interval around it reaches too far down and not far enough up. The logarithm removes most of the skew, which is the reason the course page gives for working on that scale. The delta method justifies an interval on any smooth scale; it is silent on which scale makes the normal approximation good.
Problem 2 (the arcsine interval for a proportion). For 35 cases among 200, build a 95% interval for the risk with the arcsine transformation, and compare it with the interval $\hat{p} \pm 1.96\sqrt{\hat{p}(1 - \hat{p})/n}$.
Solution. The arcsine interval is $(0.1256,\ 0.2306)$ and the usual interval is $(0.1223,\ 0.2277)$.
By the section on variance stabilization, $\arcsin\sqrt{\hat{p}}$ is approximately normal with mean $\arcsin\sqrt{p}$ and standard deviation $1/(2\sqrt{n})$, which does not involve $p$. With $\hat{p} = 0.175$, $\arcsin\sqrt{0.175} = 0.4316$ radians and $1.96/(2\sqrt{200}) = 0.0693$. So with limiting probability 0.95,
$$0.4316 - 0.0693 \le \arcsin\sqrt{p} \le 0.4316 + 0.0693.$$
The function $\sin^2$ is increasing on the interval from 0 to $\pi/2$, where these numbers lie, so applying it to all three parts preserves the inequalities: $\sin^2(0.4316 - 0.0693) = 0.1256$ and $\sin^2(0.4316 + 0.0693) = 0.2306$. The usual interval has half-width $1.96\sqrt{0.175 \times 0.825/200} = 0.0527$, giving $(0.1223,\ 0.2277)$.
The arcsine interval needs no estimate of the standard error, and it is not symmetric about $\hat{p}$: it reaches further up than down, as the binomial distribution for a small $p$ suggests it should. Its exact coverage, by summing binomial probabilities, is not uniformly better. At $n = 200$ and the population risk 0.2058 the usual interval covers with probability 0.9402 and the arcsine interval with 0.9460. At $n = 50$ and the same risk both cover with probability 0.9419. At $n = 50$ and $p = 0.05$ the usual interval covers with probability 0.9199 and the arcsine interval with 0.9113. These are three cases, not a comparison of the two methods; Confidence Intervals for a Proportion treats the intervals for a proportion and their coverage at length. What the problem shows is how a stabilizing transformation turns a limit theorem into an interval without a plug-in variance.
Problem 3 (a second-order limit by hand). A fair coin is tossed 100 times and $\hat{p}$ is the share of heads. Approximate the probability that the estimated variance $\hat{p}(1 - \hat{p})$ is at least 0.24, and compare with the exact value.
Solution. The second-order approximation gives 0.9545, or 0.9643 with a correction for discreteness; the exact probability is 0.9648.
Since $\hat{p}(1 - \hat{p}) = \tfrac14 - (\hat{p} - \tfrac12)^2$, the event is $(\hat{p} - \tfrac12)^2 \le 0.01$, or $4n(\hat{p} - \tfrac12)^2 \le 4 \times 100 \times 0.01 = 4$. By Theorem 2, as worked out in its check, $4n(\hat{p} - \tfrac12)^2$ is approximately chi-squared with 1 degree of freedom. A $\chi^2_1$ variable is the square of a standard normal $Z$, so $P(\chi^2_1 \le 4) = P(\lvert Z\rvert \le 2) = 2\Phi(2) - 1 = 0.9545$.
Exactly, the event is $\lvert X - 50\rvert \le 10$ for the binomial count $X$ of heads, with probability $P(40 \le X \le 60) = 0.9648$. The approximation is low by 0.0103. Most of the difference is discreteness: the event $\lvert X - 50\rvert \le 10$ is the same as $\lvert X - 50\rvert \le 10.5$, and using 10.5 gives $4 \times 100 \times 0.105^2 = 4.41$ and $P(\chi^2_1 \le 4.41) = 2\Phi(2.1) - 1 = 0.9643$. That is the continuity correction of The Normal Approximation to the Binomial, carried through the square.
A first-order calculation would have failed outright here. The slope of $y(1 - y)$ at $\tfrac12$ is 0, so Theorem 1 assigns $\hat{p}(1 - \hat{p})$ a limiting standard deviation of 0 and offers no probability at all.
Problem 4 (an interval for a standard deviation from the logarithm of the variance). Student's ten sleep differences have $s^2 = 1.513$, $s = 1.230$ hours. Assuming a normal population, build a 95% interval for $\sigma$ from Corollary 2, compare it with the exact chi-squared interval $[0.846,\ 2.245]$ of The Chi-Squared and t Distributions, and find its true coverage.
Solution. The delta-method interval is $(0.794,\ 1.907)$, and for normal samples of 10 its true coverage is 0.917, not 0.95.
For a normal population $\kappa = 3$, so by Corollary 2, $\sqrt{n}\,(\ln s_n^2 - \ln\sigma^2)$ has a normal limit with variance 2. The approximate standard error of $\ln s^2$ is $\sqrt{2/n} = \sqrt{2/10} = 0.4472$, and the interval for $\ln\sigma^2$ is $\ln s^2 \pm 1.96 \times 0.4472 = \ln s^2 \pm 0.8765$. Since $\ln\sigma = \tfrac12\ln\sigma^2$, the interval for $\sigma$ is $s\,e^{\pm 0.8765/2}$, from $1.230\,e^{-0.4383} = 0.794$ to $1.230\,e^{0.4383} = 1.907$.
The exact interval, $[0.846,\ 2.245]$, lies higher at both ends. The delta-method interval is symmetric on the log scale, and the distribution of $\ln s^2$ for 10 observations is not: the chi-squared distribution with 9 degrees of freedom is skewed, and the logarithm removes only part of the skew. Because the distribution is known here, the coverage of the delta-method interval can be found exactly. It contains $\sigma$ when $e^{-0.8765} \le s^2/\sigma^2 \le e^{0.8765}$, and $9s^2/\sigma^2$ is $\chi^2_9$, so the coverage is the $\chi^2_9$ probability of the interval from $9e^{-0.8765}$ to $9e^{0.8765}$, which is 0.917. The same calculation gives 0.940 for $n = 30$ and 0.947 for $n = 100$.
So for a normal population the exact chi-squared interval is the one to use, and the delta method only approaches it. The roles reverse for a population that is not normal: there, by Corollary 3 and the blood pressure simulation, the chi-squared interval has the wrong coverage at every sample size, and the delta-method interval with an estimated $\kappa$ has the right limit.
Summary
| Result | Statement |
|---|---|
| First order | $r_n(Y_n - \theta) \xrightarrow{d} X$ and $g$ differentiable at $\theta$ imply $r_n(g(Y_n) - g(\theta)) \xrightarrow{d} g'(\theta)X$ |
| Normal case | limiting variance $g'(\theta)^2\sigma^2$ |
| Second order | if $g'(\theta) = 0$: $r_n^2(g(Y_n) - g(\theta)) \xrightarrow{d} \tfrac12 g''(\theta)X^2$ |
| Several variables | limiting variance $\nabla g^{\mathsf{T}}\Sigma\,\nabla g$ |
| Log relative risk | $(1 - p_1)/(n_1p_1) + (1 - p_2)/(n_2p_2)$ |
| Log odds ratio, one sample | $\sum_j 1/(n\pi_j)$ |
| Sample variance | $\sqrt{n}\,(s_n^2 - \sigma^2)$: limiting variance $\mu_4 - \sigma^4$ |
| Stabilizing transformation | $g(\theta) = \int c/\sigma(\theta)\,d\theta$: $\arcsin\sqrt{p}$, $\sqrt{\lambda}$, $\ln\sigma^2$ |
What we have learned
If a scaled estimate converges in distribution, so does any function of it that is differentiable at the true value, with the limit multiplied by the derivative. The proof is three lines once Slutsky's theorem is in hand: the estimate converges in probability to the true value, the relative error of the linear approximation is continuous there, and a vanishing factor times a bounded sequence vanishes. When the derivative is zero the scaling is squared and the limit is the square of the original limit, chi-squared in the normal case. For several estimates the derivative becomes the gradient. The theorem concerns limiting distributions: it gives a limiting variance even where no variance exists, and it does not say on which scale the normal approximation is good. The exact calculations of this article show both sides: the log-scale interval for a relative risk balances its misses where the untransformed interval does not, and the chi-squared interval for the variance of blood pressures covers 85% of the time where the delta-method interval covers 94%.
The delta method assumed an estimate with a normal limit and asked about its functions. Sampling Distributions from the Normal: Chi-Squared, t and F turns from limits to exact distributions: what can be said for every sample size when the population is normal.
References
- Anscombe, F. J. (1948). The transformation of Poisson, binomial and negative-binomial data. Biometrika, 35(3–4), 246–254. record
- Bartlett, M. S. (1936). The square root transformation in analysis of variance. Supplement to the Journal of the Royal Statistical Society, 3(1), 68–78. record
- Casella, G., and Berger, R. L. (2002). Statistical Inference (2nd ed.). Duxbury. As cited on the course page for Slutsky's theorem and the limiting distribution of a differentiable function of an estimator. record · 11 pages
- Katz, D., Baptista, J., Azen, S. P., and Pike, M. C. (1978). Obtaining confidence intervals for the risk ratio in cohort studies. Biometrics, 34(3), 469–474. record · 2 pages
- National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018: DEMO_J, BPX_J, BPQ_J, BMX_J, DIQ_J and SMQ_J data files. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 67 pages
- Student (1908). The probable error of a mean. Biometrika, 6(1), 1–25. record · 14 pages
- van der Vaart, A. W. (1998). Asymptotic Statistics. Cambridge University Press. Chapter 3, "Delta Method", pp. 25–34, as cited on the course page for functions of several variables. record · 8 pages
- Ver Hoef, J. M. (2012). Who invented the delta method? The American Statistician, 66(2), 124–127. record
- Woolf, B. (1955). On estimating the relation between blood group and disease. Annals of Human Genetics, 19(4), 251–253. record · 2 pages