Fisher Information and the Cramér–Rao Bound

How much a sample can say about a parameter, and the limit that sets on every unbiased estimator. The score and its two properties proved under stated regularity conditions, both forms of the Fisher information, its additivity, its behavior under a change of parameter and under a reduction of the data; the Cramér–Rao bound proved by the Cauchy–Schwarz inequality, with the exact condition for equality and the matrix form; and the uniform distribution, where the conditions fail and an estimator beats the formal bound. Worked problems on weekly death counts, the Prussian horse-kick deaths, the exponential distribution and a sample variance.

A random sample of 40 adults aged 18 to 39 from the National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018 contains 10 with hypertension. The sample proportion $\hat{p} = 10/40 = 0.25$ estimates the proportion $p$ in the population of 1,696 young adults, which is $389/1{,}696 = 0.2294$. Over repeated samples $\hat{p}$ is unbiased, and its variance is $p(1-p)/n = 0.00442$. Could a cleverer unbiased estimator, computed from the same 40 observations, have a smaller variance? No. For independent observations that are each a case with probability $p$, no unbiased estimator of $p$ has variance below $p(1-p)/n$, and this article proves it.

The proof runs through a single quantity. The Fisher information $I(\theta)$ of an observation measures how sharply the logarithm of its probability changes when the parameter $\theta$ changes. The Cramér–Rao bound says that, under regularity conditions, the variance of an unbiased estimator of $\theta$ from $n$ independent observations is at least $1/(nI(\theta))$. For the hypertension indicator $I(p) = 1/(p(1-p))$, and the bound is the variance of $\hat{p}$.

First we define the score and prove its two properties, stating the conditions they need. Then we define the information in both of its forms, show that it adds over independent observations, compute it for the standard families, and see how it changes with the parameter and with a reduction of the data. Next we prove the bound, find exactly when it is attained, and extend it to several parameters. Finally we take a model in which the conditions fail and the formal bound is beaten. The article assumes the likelihood function and maximum likelihood estimation, and uses the Cauchy–Schwarz inequality of Probability Inequalities.

This article is the home of the score and the Fisher information for the site. Asymptotics of the Maximum Likelihood Estimator and Likelihood Ratio, Wald and Score Tests build on the identities proved here.

The score

The notation is that of the unit. One observation $X$ has probability mass function or density $f(x; \theta)$, where the parameter $\theta$ lies in an open interval $\Theta$. A sample $X_1, \ldots, X_n$ of independent observations has log-likelihood $\ell(\theta) = \sum_i \ln f(x_i; \theta)$. Expected values, variances and probabilities computed under the parameter value $\theta$ are written $E_\theta$, $\operatorname{Var}_\theta$ and $P_\theta$.

The score of one observation is the derivative of its log-probability with respect to the parameter,

$$u(x; \theta) = \frac{\partial}{\partial\theta} \ln f(x; \theta) = \frac{\partial f(x;\theta)/\partial\theta}{f(x;\theta)},$$

and the score of the sample is the derivative of the log-likelihood, $U(\theta) = \ell'(\theta) = \sum_{i=1}^n u(x_i; \theta)$. The score is the relative rate at which the probability of what was observed changes as $\theta$ moves. A maximum likelihood estimate in the interior of $\Theta$ is a point where the score of the sample is zero.

For the hypertension indicator, $X = 1$ for a case and 0 otherwise, so $f(x; p) = p^x (1-p)^{1-x}$ and

$$u(x; p) = \frac{x}{p} - \frac{1-x}{1-p} = \frac{x - p}{p(1-p)}.$$

A case pushes the score up by $1/p$ and a non-case pushes it down by $1/(1-p)$. For the sample of 40 with 10 cases,

$$U(p) = \frac{10}{p} - \frac{30}{1-p}.$$

It is 12.5 at $p = 0.20$, exactly 0 at $p = 0.25$ and −9.52 at $p = 0.30$. Below the estimate the data ask for a larger $p$, and above it for a smaller one.

The score is a function of the data, so before the data are seen it is a random variable. Its value at the true parameter is the one that matters for the theory. At the true $p = 0.2294$ the sample of 40 has score 4.67. Is that large? The next section shows that at the true value the score has mean 0, and the section after it that its standard deviation here is 15.04. The observed 4.67 is 0.31 standard deviations from 0, an unremarkable draw.

Two properties of the score

Here is the idea in plain words. The probabilities of all possible observations add to 1 for every value of the parameter. When $\theta$ moves, some observations gain probability and others lose it, and the gains and losses cancel. The score is the relative gain, so its probability-weighted average is zero. Differentiating the same identity once more ties the variance of the score to the average curvature of the log-probability.

Both steps differentiate a sum or an integral with respect to $\theta$. That is legitimate only under conditions, and they are stated here because the last section of the article shows what happens without them.

  • R1 (common support). The set $S = \{x : f(x; \theta) > 0\}$ of possible observations is the same for every $\theta \in \Theta$.
  • R2 (smoothness). For every $x \in S$, $f(x; \theta)$ is twice differentiable in $\theta$ on $\Theta$.
  • R3 (differentiation under the sign). The sum $\sum_{x \in S} f(x;\theta)$, or the integral $\int_S f(x;\theta)\,dx$, may be differentiated twice with respect to $\theta$ by differentiating each term, or the integrand.

R3 holds automatically when $S$ is finite, because the derivative of a finite sum is the sum of the derivatives. For an infinite sum or an integral, a sufficient condition is that near each $\theta$ the first and second derivatives of $f$ are bounded in absolute value by a function of $x$ alone whose sum or integral is finite. That sufficiency rests on the dominated convergence theorem, which this collection does not prove; Random Variables and Distribution Functions says what the collection leaves to measure theory. For the families below, R3 can be checked directly.

Theorem (the score identities). Under R1 to R3, for every $\theta \in \Theta$,

$$E_\theta[u(X;\theta)] = 0 \qquad \text{and} \qquad E_\theta\big[u(X;\theta)^2\big] = -E_\theta\!\left[\frac{\partial u(X;\theta)}{\partial\theta}\right].$$

Proof. Write the proof for a density; for a mass function, read every integral over $S$ as a sum over $S$. For every $\theta$,

$$\int_S f(x;\theta)\,dx = 1.$$

Differentiate both sides with respect to $\theta$. The right side gives 0. By R3 the left side is the integral of the derivative, and by the definition of the score $\partial f/\partial\theta = u f$ on $S$. So

$$0 = \int_S \frac{\partial f(x;\theta)}{\partial\theta}\,dx = \int_S u(x;\theta)\, f(x;\theta)\,dx = E_\theta[u(X;\theta)]. \qquad (1)$$

That is the first identity. R1 was used when the limits of integration were treated as fixed: the set $S$ did not move with $\theta$. Now differentiate (1) again, by R3, using the product rule and $\partial f/\partial\theta = uf$ once more:

$$0 = \int_S \left[\frac{\partial u}{\partial\theta}\, f + u\,\frac{\partial f}{\partial\theta}\right] dx = \int_S \frac{\partial u}{\partial\theta}\, f\,dx + \int_S u^2 f\,dx = E_\theta\!\left[\frac{\partial u}{\partial\theta}\right] + E_\theta[u^2].$$

That is the second identity. $\square$

For the sample of 40 both identities can be checked without calculus, because the number of cases has a binomial distribution with 41 possible values. The article's script sums over them in exact fractions with $p = 389/1{,}696$: the mean of $U(p)$ is exactly 0, and the mean of $U(p)^2$ and the mean of $-U'(p)$ are both exactly $40/(p(1-p))$.

Fisher information

Since the score has mean zero at the true parameter, its mean square is its variance. The Fisher information in one observation is

$$I(\theta) = E_\theta\big[u(X;\theta)^2\big] = \operatorname{Var}_\theta\big(u(X;\theta)\big),$$

and by the second identity it has a second form,

$$I(\theta) = -E_\theta\!\left[\frac{\partial^2}{\partial\theta^2} \ln f(X;\theta)\right].$$

The two forms say different things. The first measures how much the score varies from sample to sample: if a small change of $\theta$ changes the probabilities of the possible observations a great deal, in relative terms, the observations discriminate well between nearby values of $\theta$. The second measures the average curvature of the log-likelihood at the true value: a sharply curved log-likelihood has a narrow peak. Under R1 to R3 the two are the same number.

For the hypertension indicator the first form gives $I(p) = \operatorname{Var}(X)/[p(1-p)]^2 = 1/(p(1-p))$, since $\operatorname{Var}(X) = p(1-p)$. The second form agrees: $-\partial u/\partial p = x/p^2 + (1-x)/(1-p)^2$ has mean $1/p + 1/(1-p) = 1/(p(1-p))$. At $p = 0.2294$ this is 5.658 per observation.

Example (information is not about the amount of data alone). One might read "information" as a count of observations. It is a property of the model at a parameter value. An indicator observed where $p = 0.5$ carries $I = 4$, the minimum of $1/(p(1-p))$, and one observed where $p = 0.01$ carries $I = 1/(0.01 \times 0.99)$, about 101. The rarer the event, the more precisely, in absolute terms, each observation fixes $p$. In relative terms the order reverses: the bound on the standard deviation of an estimate, divided by $p$, is $\sqrt{(1-p)/(np)}$, which grows as $p$ falls.

Information adds over independent observations

Theorem (additivity). Under R1 to R3, the score $U(\theta)$ of $n$ independent observations with the same distribution has mean 0 and variance

$$I_n(\theta) = \operatorname{Var}_\theta\big(U(\theta)\big) = n\,I(\theta), \qquad \text{and} \qquad I_n(\theta) = -E_\theta[\ell''(\theta)].$$

Proof. $U(\theta) = \sum_i u(X_i;\theta)$ is a sum of independent variables, each with mean 0 and variance $I(\theta)$. The mean of a sum is the sum of the means, and the variance of a sum of independent variables is the sum of the variances (Expectation, Variance and Covariance). For the second form, $-\ell''(\theta) = \sum_i -\partial u(X_i;\theta)/\partial\theta$, and each term has mean $I(\theta)$. $\square$

The proof did not use that the observations have the same distribution, only that they are independent. If observation $i$ has information $I^{(i)}(\theta)$, the sample has information $\sum_i I^{(i)}(\theta)$. This is why the information is defined through a variance and not a standard deviation: variances of independent pieces add.

For the 40 adults $I_{40}(p) = 40 \times 5.658 = 226.3$, so the score at the true value has standard deviation $\sqrt{226.3} = 15.04$, the figure used above.

The quantity $J(\theta) = -\ell''(\theta)$, computed from the sample in hand, is called the observed information. Its expected value is $I_n(\theta)$. For the 40 adults at the estimate, $J(0.25) = 10/0.25^2 + 30/0.75^2 = 213.3$. A sample of 400 with 100 cases has the same estimate and ten times the curvature, 2,133.3.

Left: the log-likelihood of p, measured from its maximum, for 10 cases in 40 adults and for 100 cases in 400. Notice that both peak at 0.25 and that the larger sample's curve is ten times as sharply curved there: the curvature at the peak is the observed information, 213.3 and 2,133.3. The dotted parabolas have exactly those curvatures. Right: the likelihood of the upper limit of a uniform distribution, for five observations whose largest is 0.9. Notice that it is zero up to the largest observation and falls from there: the maximum is at a cliff, not at a rounded peak, and there is no curvature to measure.
Left: the log-likelihood of p, measured from its maximum, for 10 cases in 40 adults and for 100 cases in 400. Notice that both peak at 0.25 and that the larger sample's curve is ten times as sharply curved there: the curvature at the peak is the observed information, 213.3 and 2,133.3. The dotted parabolas have exactly those curvatures. Right: the likelihood of the upper limit of a uniform distribution, for five observations whose largest is 0.9. Notice that it is zero up to the largest observation and falls from there: the maximum is at a cliff, not at a rounded peak, and there is no curvature to measure.

The standard families

The same two-line calculation gives the information of the families used across the site. In each row the score is written for one observation, and the information follows from its variance or from the mean of minus its derivative.

Model, parameter Score $u(x;\theta)$ $I(\theta)$
Bernoulli, $p$ $(x - p)/(p(1-p))$ $1/(p(1-p))$
Poisson, mean $\lambda$ $(x - \lambda)/\lambda$ $1/\lambda$
Normal, mean $\mu$ ($\sigma$ known) $(x - \mu)/\sigma^2$ $1/\sigma^2$
Normal, standard deviation $\sigma$ ($\mu$ known) $-1/\sigma + (x-\mu)^2/\sigma^3$ $2/\sigma^2$
Normal, variance $v = \sigma^2$ ($\mu$ known) $-1/(2v) + (x-\mu)^2/(2v^2)$ $1/(2v^2)$
Exponential, mean $\theta$ $(x - \theta)/\theta^2$ $1/\theta^2$
Cauchy, location $\theta$ $2(x-\theta)/(1 + (x-\theta)^2)$ $1/2$

The derivations are short.

  • Poisson. $\ln f = -\lambda + x\ln\lambda - \ln x!$, so $u = x/\lambda - 1$. Its variance is $\operatorname{Var}(X)/\lambda^2 = \lambda/\lambda^2$, by the equality of mean and variance proved in The Poisson Distribution.
  • Normal mean. $\ln f = -(x-\mu)^2/(2\sigma^2)$ plus terms without $\mu$, so $u = (x-\mu)/\sigma^2$, with variance $\sigma^2/\sigma^4$.
  • Normal standard deviation. $\ln f = -\ln\sigma - (x-\mu)^2/(2\sigma^2)$ plus a constant. Then $u = -1/\sigma + (x-\mu)^2/\sigma^3$ and $\partial u/\partial\sigma = 1/\sigma^2 - 3(x-\mu)^2/\sigma^4$. Since $E[(X-\mu)^2] = \sigma^2$, minus its mean is $-1/\sigma^2 + 3/\sigma^2 = 2/\sigma^2$.
  • Normal variance. With $v = \sigma^2$, $\ln f = -\tfrac12\ln v - (x-\mu)^2/(2v)$ plus a constant. Then $\partial u/\partial v = 1/(2v^2) - (x-\mu)^2/v^3$, and minus its mean is $-1/(2v^2) + 1/v^2 = 1/(2v^2)$.
  • Exponential. $\ln f = -\ln\theta - x/\theta$, so $u = -1/\theta + x/\theta^2$. The exponential distribution with mean $\theta$ has variance $\theta^2$, so the variance of $u$ is $\theta^2/\theta^4$.
  • Cauchy location. The density is $1/[\pi(1 + (x-\theta)^2)]$, so $u = 2(x-\theta)/(1+(x-\theta)^2)$. With $y = x - \theta$,

$$I = \int_{-\infty}^{\infty} \frac{4y^2}{(1+y^2)^2}\cdot\frac{dy}{\pi(1+y^2)} = \frac{4}{\pi}\int_{-\pi/2}^{\pi/2} \sin^2 t\,\cos^2 t\,dt = \frac{4}{\pi}\cdot\frac{\pi}{8} = \frac12 .$$

The substitution is $y = \tan t$, for which $1 + y^2 = 1/\cos^2 t$ and $dy = dt/\cos^2 t$, so the integrand $y^2/(1+y^2)^3\,dy$ becomes $\sin^2 t\cos^2 t\,dt$. Since $\sin^2 t\cos^2 t = \tfrac14\sin^2 2t = \tfrac18(1 - \cos 4t)$, its integral over an interval of length $\pi$ is $\pi/8$.

The script checks every row by numerical summation or integration of the squared score and of minus its derivative. Two rows deserve a remark. The information about a normal mean, $1/\sigma^2$, does not depend on the mean: every location is equally easy to estimate. The Cauchy distribution has no mean and no variance, and its sample mean is useless as an estimate of its center, yet its information is finite and positive. The information is a property of the shape of the density, not of its moments.

Information depends on the parameter chosen

The table has two rows for the spread of a normal distribution, with different answers. That is not an inconsistency: the information is measured per unit of the parameter, so it changes when the parameter is rescaled.

Theorem (change of parameter). Let $\theta = g(\eta)$ with $g$ differentiable, and let $I^*(\eta)$ be the information about $\eta$ in the same observation. Then $I^*(\eta) = I(g(\eta))\,[g'(\eta)]^2$.

Proof. As a function of $\eta$ the log-probability is $\ln f(x; g(\eta))$. By the chain rule its derivative is $u(x; g(\eta))\,g'(\eta)$. The factor $g'(\eta)$ does not depend on $x$, so the mean square of the new score is $[g'(\eta)]^2$ times the mean square of the old one. $\square$

The two normal rows agree with the theorem: $v = \sigma^2$ gives $dv/d\sigma = 2\sigma$, and $\frac{1}{2\sigma^4}(2\sigma)^2 = 2/\sigma^2$. Two changes of parameter for a proportion are used elsewhere on the site.

  • Log odds. With $\eta = \ln\frac{p}{1-p}$, $p = 1/(1+e^{-\eta})$ and $dp/d\eta = p(1-p)$. So $I^*(\eta) = \frac{1}{p(1-p)}[p(1-p)]^2 = p(1-p)$, which is 0.1768 at $p = 0.2294$. This is the scale of logistic regression and of the log odds ratio in Relative Risk and the Odds Ratio.
  • Arcsine scale. With $\psi = \arcsin\sqrt{p}$, $p = \sin^2\psi$ and $dp/d\psi = 2\sin\psi\cos\psi = 2\sqrt{p(1-p)}$. So $I^*(\psi) = \frac{1}{p(1-p)}\cdot 4p(1-p) = 4$, the same at every $p$. On this scale every proportion is equally hard to estimate, which is why the transformation is called variance-stabilizing.

The rule is a rule for the expected information. The observed information transforms differently. If $g$ is twice differentiable and $\ell^*(\eta) = \ell(g(\eta))$, then by the chain and product rules $J^*(\eta) = -\ell^{*\prime\prime}(\eta) = J(g(\eta))\,[g'(\eta)]^2 - U(g(\eta))\,g''(\eta)$. The extra term vanishes at a root of the likelihood equation, where $U = 0$, and where $g''(\eta) = 0$, as for a linear $g$, and nowhere else. This matters for the score test of Likelihood Ratio, Wald and Score Tests: the statistic $U(\theta_0)^2/I_n(\theta_0)$, built with the expected information, is unchanged by a change of parameter, since numerator and denominator are both multiplied by $[g'(\eta_0)]^2$. A version built with the observed information $J(\theta_0)$ in its place is not.

Information and the Kullback–Leibler divergence

The information is also the local size of the divergence between neighboring members of the family. The Kullback–Leibler divergence from the distribution at $\theta_0$ to the one at $\theta$ is

$$D(\theta_0 \,\|\, \theta) = E_{\theta_0}\big[\ln f(X;\theta_0) - \ln f(X;\theta)\big],$$

defined and shown to be nonnegative in Cross-Entropy and Kullback–Leibler Divergence. Hold $\theta_0$ fixed and treat it as a function of $\theta$. At $\theta = \theta_0$ it is 0. Suppose the expectation may be differentiated twice under the sign, as in R3. Its first derivative is $-E_{\theta_0}[u(X;\theta)]$, which at $\theta = \theta_0$ is 0 by the first score identity. Its second derivative is $-E_{\theta_0}[\partial u(X;\theta)/\partial\theta]$, which at $\theta = \theta_0$ is $I(\theta_0)$ by the second. So the second-order Taylor approximation is

$$D(\theta_0 \,\|\, \theta) \approx \tfrac12\, I(\theta_0)\,(\theta - \theta_0)^2 \qquad \text{for } \theta \text{ near } \theta_0 .$$

For one parameter on a finite set of outcomes, Cross-Entropy and Kullback–Leibler Divergence proves the limit that this approximation suggests, $D/(\theta - \theta_0)^2 \to \tfrac12 I(\theta_0)$, in its section "Small differences: chi-square and Fisher information".

For the hypertension indicator with $p_0 = 0.2294$, the divergence to $p = 0.25$ is 0.00116 nats and the approximation gives 0.00120. At $p = 0.30$ the two are 0.01251 and 0.01411, and at $p = 0.40$ they are 0.06532 and 0.08236. The approximation is local, as a Taylor approximation is. This is the sense in which natural evolution strategies measure the size of a step by the information matrix: a step of fixed length in that measure changes the distribution by a fixed divergence, whatever the parametrization.

Information and data reduction

A statistic $T = T(\mathbf{X})$ computed from the sample has its own distribution, with its own information about $\theta$. Reducing the data cannot create information, and it loses none exactly when the statistic is sufficient in the sense of Sufficient Statistics.

Theorem (information in a statistic). Let the sample $\mathbf{X}$ be discrete with joint mass function $f(\mathbf{x};\theta)$ satisfying R1 to R3, and let $T = T(\mathbf{X})$ have mass function $g(t;\theta)$ and information $I_T(\theta)$. Suppose also that each sum $g(t;\theta) = \sum_{\mathbf{x}: T(\mathbf{x}) = t} f(\mathbf{x};\theta)$ may be differentiated term by term, as it may when the sample space is finite, or under the domination condition stated after R3. Then $I_T(\theta) \le I_n(\theta)$, with equality at a given $\theta$ exactly when the sample score $U(\theta)$ is a function of $T$ with probability 1. If $T$ is sufficient, equality holds for every $\theta$.

Proof. The idea is that the score of $T$ is the average of the sample score over the samples that give the same value of $T$. By R1 the values $t$ with $g(t;\theta) > 0$ are the same for every $\theta$; take such a $t$. Since $g(t;\theta) = \sum_{\mathbf{x}: T(\mathbf{x}) = t} f(\mathbf{x};\theta)$, differentiating term by term, as assumed, and using $\partial f/\partial\theta = Uf$,

$$\frac{\partial g(t;\theta)/\partial\theta}{g(t;\theta)} = \sum_{\mathbf{x}: T(\mathbf{x}) = t} U(\mathbf{x};\theta)\,\frac{f(\mathbf{x};\theta)}{g(t;\theta)} = E_\theta\big[U(\theta) \mid T = t\big],$$

because $f(\mathbf{x};\theta)/g(t;\theta)$ is the conditional probability of the sample $\mathbf{x}$ given $T = t$. So the score of $T$ is the conditional expectation of the sample score. By the law of total variance (Conditional Expectation),

$$I_n(\theta) = \operatorname{Var}(U) = \operatorname{Var}\big(E[U \mid T]\big) + E\big[\operatorname{Var}(U \mid T)\big] = I_T(\theta) + E\big[\operatorname{Var}(U \mid T)\big] \ge I_T(\theta).$$

Equality holds exactly when $\operatorname{Var}(U \mid T) = 0$ with probability 1, that is, when $U$ is constant on each set $\{T = t\}$ of positive probability. If $T$ is sufficient, the factorization theorem gives $f(\mathbf{x};\theta) = \tilde{g}(T(\mathbf{x});\theta)\,h(\mathbf{x})$ for some functions $\tilde{g}$ and $h$, so $U = \partial \ln \tilde{g}(T;\theta)/\partial\theta$, a function of $T$. $\square$

Example (recording only whether an event occurred). A Poisson count $X$ with mean $\lambda$ carries information $1/\lambda$. Suppose only the indicator $Z$ of the event $X = 0$ is kept. $Z$ is a Bernoulli variable with success probability $\pi_0 = e^{-\lambda}$, and by the change-of-parameter theorem its information about $\lambda$ is

$$\frac{1}{\pi_0(1-\pi_0)}\left(\frac{d\pi_0}{d\lambda}\right)^2 = \frac{e^{-2\lambda}}{e^{-\lambda}(1 - e^{-\lambda})} = \frac{1}{e^{\lambda} - 1}.$$

The share of the information kept is $\lambda/(e^\lambda - 1)$, always below 1. Problem 2 below meets it again with $\lambda = 0.61$, where the reciprocal of the share is 1.378.

The Cramér–Rao bound

Back to the question of the opening: how small can the variance of an unbiased estimator be?

Here is the idea in plain words. An estimator $W$ that is unbiased for $\theta$ at every $\theta$ must respond when $\theta$ moves: its mean moves one for one. The only way the mean of a fixed function of the data can move is through the changing probabilities of the data, and the score records those changes. So $W$ must be correlated with the score, in fact with covariance exactly 1. A variable can have a given covariance with another only if its own variance is large enough, by the Cauchy–Schwarz inequality. The variance of the score is the information, and the bound follows.

Theorem (Cramér–Rao bound). Let $X_1, \ldots, X_n$ be independent with mass function or density $f(x;\theta)$ satisfying R1 to R3, with $0 < I(\theta) < \infty$. Let $W = W(\mathbf{X})$ be an estimator with finite variance and mean $E_\theta[W] = \tau(\theta)$, and suppose the sum or integral defining $E_\theta[W]$ may be differentiated under the sign. Then

$$\operatorname{Var}_\theta(W) \ge \frac{[\tau'(\theta)]^2}{n\,I(\theta)}.$$

In particular, if $W$ is unbiased for $\theta$, so that $\tau(\theta) = \theta$, then $\operatorname{Var}_\theta(W) \ge 1/(nI(\theta))$.

Proof. Write $f(\mathbf{x};\theta) = \prod_i f(x_i;\theta)$ for the joint density of the sample, so that $\partial f(\mathbf{x};\theta)/\partial\theta = U(\mathbf{x};\theta) f(\mathbf{x};\theta)$, and write integrals over the sample space (sums, for discrete data). Differentiate the definition of $\tau$ under the sign:

$$\tau'(\theta) = \frac{d}{d\theta}\int W(\mathbf{x})\,f(\mathbf{x};\theta)\,d\mathbf{x} = \int W(\mathbf{x})\,U(\mathbf{x};\theta)\,f(\mathbf{x};\theta)\,d\mathbf{x} = E_\theta[W\,U(\theta)].$$

Since $E_\theta[U(\theta)] = 0$ by the additivity theorem, $E_\theta[WU] = \operatorname{Cov}_\theta(W, U)$. So

$$\operatorname{Cov}_\theta\big(W, U(\theta)\big) = \tau'(\theta). \qquad (2)$$

The Cauchy–Schwarz inequality, applied to the centered variables $W - \tau(\theta)$ and $U(\theta)$, says that the square of a covariance is at most the product of the variances (Probability Inequalities, section "The Cauchy–Schwarz inequality"):

$$[\tau'(\theta)]^2 = \operatorname{Cov}_\theta(W, U)^2 \le \operatorname{Var}_\theta(W)\,\operatorname{Var}_\theta(U) = \operatorname{Var}_\theta(W)\; n\,I(\theta).$$

Dividing by $nI(\theta) > 0$ gives the bound. $\square$

Three remarks fix what the theorem says.

  • The bound is a statement about each parameter value separately, and it involves the unknown $\theta$. It is a floor under a variance, not an interval for $\theta$.
  • An estimator with bias $b(\theta) = E_\theta[W] - \theta$ has $\tau'(\theta) = 1 + b'(\theta)$, so its variance is at least $[1 + b'(\theta)]^2/(nI(\theta))$. This can be smaller than $1/(nI(\theta))$.
  • The assumption that $E_\theta[W]$ may be differentiated under the sign concerns the estimator as well as the model. For data with a finite sample space, such as the 40 indicators, it always holds.

For the 40 adults, $nI(p) = 226.3$ and the bound for unbiased estimators of $p$ is $1/226.3 = 0.00442$, a standard deviation of 0.0665. The script confirms equation (2) in exact fractions: the covariance of $\hat{p}$ with the score is exactly 1.

Example (what the bound does not forbid). A common reading is "no estimator can be more accurate than $1/(nI)$". The bound limits the variance of unbiased estimators. The estimator $(X + 1)/(n + 2)$, where $X$ is the number of cases, is biased toward one half, and at $p = 0.2294$ and $n = 40$ its mean squared error is 0.00417, below 0.00442. (Its bias is $(1-2p)/(n+2)$ and its variance $np(1-p)/(n+2)^2$, so its mean squared error is $[np(1-p) + (1-2p)^2]/(n+2)^2$.) It gains more from its smaller variance than it loses to its bias at this $p$. The constant estimator that always answers 0.23 has variance 0, and the theorem agrees: its mean does not move with $p$, so $\tau' = 0$ and the bound is 0. Unbiasedness is a restriction, and the bound is the price list within it.

When the bound is attained

The Cauchy–Schwarz inequality is an equality exactly when one centered variable is a constant multiple of the other, with probability 1. Since $\operatorname{Var}_\theta(U) = nI(\theta) > 0$, equality in the bound at a given $\theta$ therefore means

$$W - \tau(\theta) = c(\theta)\,U(\theta) \qquad \text{with probability 1},$$

for some number $c(\theta)$. Taking the covariance of both sides with $U$ and using (2) gives $\tau'(\theta) = c(\theta)\,nI(\theta)$, so $c(\theta) = \tau'(\theta)/(nI(\theta))$.

Corollary (attainment). An estimator $W$ with mean $\tau(\theta)$ attains the Cramér–Rao bound at every $\theta$ exactly when, for every $\theta$,

$$U(\theta) = \frac{n\,I(\theta)}{\tau'(\theta)}\,\big[W - \tau(\theta)\big] \qquad \text{with probability 1}$$

(at values of $\theta$ where $\tau'(\theta) \ne 0$). The score of the sample must be a linear function of the estimator.

For the three families whose estimators the course uses, the score has exactly this form.

  • Proportion. $U(p) = \sum_i (x_i - p)/(p(1-p)) = \frac{n}{p(1-p)}(\hat{p} - p)$. So $\hat{p}$ attains the bound, and no unbiased estimator of $p$ has smaller variance at any $p$. This is the claim of the opening.
  • Poisson mean. $U(\lambda) = \frac{n}{\lambda}(\bar{x} - \lambda)$. The sample mean attains the bound $\lambda/n$.
  • Normal mean, $\sigma$ known. $U(\mu) = \frac{n}{\sigma^2}(\bar{x} - \mu)$. The sample mean attains the bound $\sigma^2/n$.

Attainment is the exception, for two reasons. First, it restricts the model. The corollary requires $\partial\ln f(\mathbf{x};\theta)/\partial\theta = a(\theta)[W(\mathbf{x}) - \tau(\theta)]$ with $a(\theta) = nI(\theta)/\tau'(\theta)$. The corollary gives this relation with probability 1 at each $\theta$, and the exceptional samples could depend on $\theta$. Suppose it holds for every sample $\mathbf{x}$ in the support and every $\theta$, as it does for discrete data, where every possible sample has positive probability, and for continuous data when both sides are continuous in $\mathbf{x}$. If $a$ and $a\tau$ are continuous, integrating in $\theta$ from a fixed point then gives $\ln f(\mathbf{x};\theta) = A(\theta)W(\mathbf{x}) + B(\theta) + C(\mathbf{x})$, where $A$ is an antiderivative of $a$, $B$ an antiderivative of $-a\tau$, and $C$ does not involve $\theta$. That is the form of a one-parameter exponential family (Exponential Families). Second, within such a model it restricts the function estimated. Suppose $W$ attains the bound for $\tau$ and another estimator $W_2$ attains it for $\tau_2$. Both are then linear functions of the same score: at a fixed $\theta$, $W_2 - \tau_2(\theta) = c_2(\theta)U(\theta) = \frac{c_2(\theta)}{c(\theta)}[W - \tau(\theta)]$. So $W_2 = \alpha W + \beta$ for two constants $\alpha$ and $\beta$, and taking expectations at every $\theta$ gives $\tau_2 = \alpha\tau + \beta$. Only $\tau$ and its linear functions can be estimated with equality. Problems 2 and 4 give two unbiased estimators that lie above the bound at every sample size. For the estimator $W$ of Problem 2, Sufficient Statistics proves that no unbiased estimator has a smaller variance, so the bound there is out of reach, not merely missed.

Proposition (no estimator attains the bound for a Cauchy location). For the Cauchy location model no unbiased estimator of $\theta$ attains the bound $2/n$ at every $\theta$.

Proof. If one did, the scores at two parameter values $\theta_1 \ne \theta_2$ would both be linear functions of it, and so linear functions of each other, with probability 1 and therefore, both scores being continuous in the observations, for every sample. Take samples whose $n$ observations all equal $x$: the two scores are $n$ times $2(x-\theta_1)/(1+(x-\theta_1)^2)$ and $n$ times $2(x-\theta_2)/(1+(x-\theta_2)^2)$. Both tend to 0 as $x \to \infty$, so the linear relation has no constant term, and one score is a constant multiple of the other. But the first is zero only at $x = \theta_1$ and the second only at $x = \theta_2$, a contradiction. $\square$

Experiment (how far above the bound a simple estimator sits). The sample mean of Cauchy observations has no variance at all. The sample median of 41 observations is unbiased by symmetry and has a finite variance: it exceeds $t$ only if at least 21 observations do, so its tail probabilities are at most a constant times the 21st power of a single observation's. Its variance can be measured. We drew 20,000 samples of 41 observations with $\theta = 0$, for each of 30 seeds, and computed $n$ times the mean squared distance of the median from 0, which estimates $n$ times its variance. The median over the seeds is 2.66 (interquartile range 2.63 to 2.67), against the bound's 2. Every one of the 30 values is above 2. The quantity has an exact value to compare with. A standard Cauchy observation is $\tan(\pi(V - \tfrac12))$ for $V$ uniform on $(0, 1)$, an increasing function of $V$, so the median of 41 of them is the same function of the median of 41 uniform variables. That median has the density $\frac{41!}{20!\,20!}v^{20}(1-v)^{20}$, because 20 of the other observations must fall below it and 20 above. Integrating numerically, $41\,E[\tan^2(\pi(B - \tfrac12))] = 2.658$ for $B$ with that density. The simulation agrees with it. Asymptotics of the Maximum Likelihood Estimator repeats the experiment for the root of the likelihood equation reached from the median, which comes much closer to the bound as $n$ grows, and proves that a consistent root has the bound as its limiting variance.

Several parameters

Most models have more than one parameter. Let $\boldsymbol\theta = (\theta_1, \ldots, \theta_k)$, and let $u_j(x;\boldsymbol\theta) = \partial\ln f(x;\boldsymbol\theta)/\partial\theta_j$ be the $j$th component of the score. Assume R1 to R3 for every partial derivative. The Fisher information matrix of one observation is the $k \times k$ matrix with entries

$$I(\boldsymbol\theta)_{jl} = E_{\boldsymbol\theta}[u_j u_l] = -E_{\boldsymbol\theta}\!\left[\frac{\partial^2 \ln f(X;\boldsymbol\theta)}{\partial\theta_j\,\partial\theta_l}\right].$$

The equality of the two forms is proved as before. Differentiating $\int_S f\,dx = 1$ with respect to $\theta_l$ gives $\int_S u_l f\,dx = 0$, so each component of the score has mean 0. Differentiating that with respect to $\theta_j$ gives $\int_S (\partial u_l/\partial\theta_j) f\,dx + \int_S u_l u_j f\,dx = 0$. The matrix is the covariance matrix of the score vector, so it is symmetric and positive semidefinite, and for $n$ independent observations it is multiplied by $n$, entry by entry, by the argument of the additivity theorem.

Example (the normal distribution with both parameters unknown). For a normal distribution with mean $\mu$ and standard deviation $\sigma$, the diagonal entries are those of the table, $1/\sigma^2$ and $2/\sigma^2$. The off-diagonal entry is minus the mean of $\partial u_\mu/\partial\sigma = -2(x-\mu)/\sigma^3$, which is 0. So

$$I(\mu, \sigma) = \begin{pmatrix} 1/\sigma^2 & 0 \\ 0 & 2/\sigma^2 \end{pmatrix}.$$

At $\sigma = 0.5$ the entries are 4 and 8; the script confirms all three by numerical integration. This is the matrix $F$ that Natural Evolution Strategies inverts for a one-dimensional Gaussian search distribution. Its example at $\sigma = 0.5$ divides the gradient in $\mu$ by 4 and the gradient in $\sigma$ by 8.

Theorem (Cramér–Rao bound, several parameters). Under the conditions above, let $I(\boldsymbol\theta)$ be invertible, and let $W$ be an estimator with finite variance whose mean $\tau(\boldsymbol\theta)$ may be differentiated under the sign, with gradient $\nabla\tau$. Then

$$\operatorname{Var}_{\boldsymbol\theta}(W) \ge \frac{1}{n}\,\nabla\tau^{\mathsf T}\, I(\boldsymbol\theta)^{-1}\,\nabla\tau .$$

Proof. Let $\mathbf{U}$ be the score vector of the sample, with covariance matrix $nI(\boldsymbol\theta)$. Differentiating $\tau$ with respect to each $\theta_j$, exactly as in the proof of (2), gives $\operatorname{Cov}(W, U_j) = \partial\tau/\partial\theta_j$. Take any nonzero vector $\mathbf{a}$ of $k$ numbers. The single variable $\mathbf{a}^{\mathsf T}\mathbf{U}$ has covariance $\mathbf{a}^{\mathsf T}\nabla\tau$ with $W$ and variance $n\,\mathbf{a}^{\mathsf T}I\mathbf{a} > 0$. By the Cauchy–Schwarz inequality,

$$\operatorname{Var}(W) \ge \frac{(\mathbf{a}^{\mathsf T}\nabla\tau)^2}{n\,\mathbf{a}^{\mathsf T} I\,\mathbf{a}} \qquad \text{for every } \mathbf{a} \ne \mathbf{0}. \qquad (3)$$

If $\nabla\tau = \mathbf{0}$ the claimed bound is 0 and there is nothing to prove. Otherwise choose $\mathbf{a} = I^{-1}\nabla\tau$. Then $\mathbf{a}^{\mathsf T}\nabla\tau = \nabla\tau^{\mathsf T}I^{-1}\nabla\tau$ and $\mathbf{a}^{\mathsf T}I\mathbf{a} = \nabla\tau^{\mathsf T}I^{-1}\nabla\tau$, because $I^{-1}$ is symmetric. The right side of (3) becomes $\nabla\tau^{\mathsf T}I^{-1}\nabla\tau/n$. $\square$

Inequality (3) holds for every $\mathbf{a}$, and the theorem uses the choice that makes it strongest. Another choice shows what unknown parameters cost. For an unbiased estimator of $\theta_1$, $\nabla\tau = (1, 0, \ldots, 0)$. Taking $\mathbf{a} = (1, 0, \ldots, 0)$ in (3) gives $1/(nI_{11})$, the bound that would apply if the other parameters were known. The theorem's bound, $[I^{-1}]_{11}/n$, is the largest value the right side of (3) can take, so $[I^{-1}]_{11} \ge 1/I_{11}$. Not knowing the other parameters can only raise the floor. The two agree when the matrix is diagonal, as for the normal distribution: the bound for the mean is $\sigma^2/n$ whether or not $\sigma$ is known, and the bound for an unbiased estimator of $\sigma$ is $\sigma^2/(2n)$.

The matrix bound stated here is for one function $\tau$ of the parameters. A statement about the whole covariance matrix of a vector of estimators follows by applying it to every linear combination of them. The note on influence functions and the article on CMA-ES work with curvature and covariance matrices of this kind for models with many parameters.

When the conditions fail: the uniform distribution

The conditions R1 to R3 were used at identifiable steps, so a model that violates one of them shows what they protect. Let $X_1, \ldots, X_n$ be independent and uniform on $(0, \theta)$, with density $f(x;\theta) = 1/\theta$ for $0 < x < \theta$. The set of possible observations, $(0, \theta)$, depends on $\theta$, so R1 fails.

Compute the formal quantities anyway. On the support $\ln f = -\ln\theta$, so $u = -1/\theta$ for every $x$. The score is a nonzero constant: its mean is $-1/\theta$, not 0. Its mean square is $1/\theta^2$, while minus the mean of its derivative is $-1/\theta^2$. The two forms of the information disagree, and one of them is negative. The step of the proof that fails is the first. Differentiating $\int_0^\theta \frac{1}{\theta}\,dx = 1$ with respect to $\theta$ gives 0, but the integral of the derivative of the integrand is $\int_0^\theta (-1/\theta^2)\,dx = -1/\theta$. The difference, $1/\theta$, is the contribution of the moving upper limit, which is the value of the integrand at $x = \theta$. Equation (1) silently assumed there was no such term.

If we nevertheless insert the mean square $1/\theta^2$ into the formula, the "bound" for unbiased estimators of $\theta$ is $\theta^2/n$. It is false.

Theorem (an unbiased estimator below the formal bound). Let $M = \max(X_1, \ldots, X_n)$. Then $W = \frac{n+1}{n}M$ is unbiased for $\theta$, with

$$\operatorname{Var}_\theta(W) = \frac{\theta^2}{n(n+2)} < \frac{\theta^2}{n} \qquad \text{for every } n \ge 1.$$

Proof. The maximum is at most $m$ exactly when every observation is, so $P(M \le m) = (m/\theta)^n$ for $0 \le m \le \theta$, and $M$ has density $n m^{n-1}/\theta^n$ there (Random Variables and Distribution Functions treats the maximum of independent variables). Then

$$E[M] = \int_0^\theta m\,\frac{n m^{n-1}}{\theta^n}\,dm = \frac{n\theta}{n+1}, \qquad E[M^2] = \int_0^\theta m^2\,\frac{n m^{n-1}}{\theta^n}\,dm = \frac{n\theta^2}{n+2}.$$

So $E[W] = \theta$, and

$$\operatorname{Var}(W) = \frac{(n+1)^2}{n^2}\left[\frac{n}{n+2} - \frac{n^2}{(n+1)^2}\right]\theta^2 = \left[\frac{(n+1)^2}{n(n+2)} - 1\right]\theta^2 = \frac{\theta^2}{n(n+2)},$$

since $(n+1)^2 - n(n+2) = 1$. $\square$

For $n = 5$ the variance is $\theta^2/35 = 0.0286\,\theta^2$, a seventh of the formal bound $\theta^2/5$; a simulation of 400,000 samples in the script gives 0.029. For $n = 40$ it is a forty-second of the formal bound. The variance falls like $1/n^2$, not $1/n$: the largest observation crowds up against the edge of the support much faster than an average settles. The right panel of the figure shows why no information-based argument applies. The likelihood is $\theta^{-n}$ for $\theta \ge M$ and 0 below, so its maximum sits at a discontinuity. Asymptotics of the Maximum Likelihood Estimator finds the limiting distribution of the maximum in this model, which is not normal.

What is proved, what is cited, what the experiment shows

Proved on this page: the two score identities under R1 to R3; that no unbiased estimator attains the bound for a Cauchy location; additivity over independent observations; the information of seven standard families; the change-of-parameter rule; the information in a statistic, for discrete data; the Cramér–Rao bound for one parameter with its equality condition, and for a function of several parameters; the failure of the identities and of the bound for the uniform distribution.

Used from other pages: the Cauchy–Schwarz inequality with its equality case (Probability Inequalities), the variance of a sum and the law of total variance (Expectation, Variance and Covariance; Conditional Expectation), and the nonnegativity of the Kullback–Leibler divergence (Cross-Entropy and Kullback–Leibler Divergence).

Not proved: that R3 follows from a domination condition (it rests on the dominated convergence theorem); the quadratic approximation of the divergence beyond the computation of its first two derivatives (the finite-outcome case is proved in the divergence article). Random variables are discrete or have densities throughout, and the measure-theoretic statement is not attempted.

Computed: every number is produced by tools/experiments/statistics/fisher-information-and-the-cramer-rao-bound.py, with the binomial identities in exact fractions. The Cauchy experiment is an observation at one sample size, 41, with 30 seeds.

Sources. The information is named for R. A. Fisher. The Likelihood Function gives the history of his paper of 1922, which introduced the method of maximum likelihood under that name (Fisher, 1922, p. 323), from the accounts of Aldrich and Stigler. The inequality carries the names of Harald Cramér and C. R. Rao. Rao (1945) states the aim of his paper as "to derive certain inequality relations connecting the elements of the Information Matrix as defined by Fisher (1921) and the variances and covariances of the estimating functions" (p. 236 of the reprint). His section on a single parameter (p. 237) proves the inequality as this page does: differentiating the condition of unbiasedness under the integral sign shows that the covariance of $t$ and $\frac{1}{\phi}\frac{d\phi}{d\theta}$ is unity, and since "the square of the covariance of two variates is not greater than the product of the variances of the variates", $V(t)$ is not less than $1/I$, where $I$ is "the intrinsic accuracy defined by Fisher (1921)"; for an unbiased estimate of a function $f(\theta)$ the bound is $\{f'(\theta)\}^2/I$. (Rao's reference list dates Fisher's paper 1921, the year it was read; this site cites it as Fisher, 1922.) The next section (pp. 239–240) gives the form for several parameters: the smallest variance attainable for an unbiased estimate of $\theta_i$ is the $i$th diagonal element of the inverse of the information matrix. Cramér's book appeared in 1946 (Cramér, 1946). For a textbook treatment see the chapter on point estimation in Casella and Berger (2002). We did not have Cramér's book or the textbook open: the first is cited for its place in the history, and the second by the title of its chapter, as further reading.

Worked problems

These problems are posed for this article on the unit's data and on small exact cases; the unit has no course assignments. Every number is recomputed by the article's script.

Problem 1 (the bound for a Poisson mean). Iowa recorded 38,395 cancer deaths in the 313 weeks of 2014 to 2019, a mean of 122.67 a week (Overdispersion in Count Data shows that these counts fit the Poisson model). Treat the weekly counts as independent Poisson counts with mean $\lambda$. Find the information in the sample, the Cramér–Rao bound for unbiased estimators of $\lambda$, and an estimator that attains it.

Solution. The bound is $\lambda/313$, and the sample mean attains it. One count has $\ln f = -\lambda + x\ln\lambda - \ln x!$ and score $u = x/\lambda - 1$, with information $I(\lambda) = 1/\lambda$. The 313 weeks have $I_{313}(\lambda) = 313/\lambda$, so every unbiased estimator has variance at least $\lambda/313$. The score of the sample is $U(\lambda) = \sum_i (x_i/\lambda - 1) = \frac{313}{\lambda}(\bar{x} - \lambda)$, a linear function of $\bar{x}$, so by the attainment corollary $\bar{x}$ attains the bound. Directly, $\operatorname{Var}(\bar{X}) = \lambda/313$. The bound depends on the unknown $\lambda$. At the estimate, the information per week is $1/122.67 = 0.00815$, the bound is $122.67/313 = 0.3919$, and its square root, the standard error of the mean, is 0.626 deaths a week.

Problem 2 (a function of the mean, where the bound is not attained). Bortkiewicz's ten comparable Prussian corps had 122 deaths from horse kicks in 200 corps-years, with no death in 109 of them (The Poisson Distribution). Treat the 200 counts as independent Poisson counts with mean $\lambda$, and let $S$ be their total.

  • (a) Find the Cramér–Rao bound for unbiased estimators of $\tau(\lambda) = e^{-\lambda}$, the probability of a year with no death.
  • (b) Show that the share of years with no death is unbiased, and compare its variance with the bound.
  • (c) Show that $W = (1 - 1/n)^S$ is unbiased, find its variance, and show that it is above the bound for every $n$ and $\lambda$.

Solution. (a) $\tau'(\lambda) = -e^{-\lambda}$ and $nI(\lambda) = n/\lambda$, so the bound is

$$\frac{[\tau'(\lambda)]^2}{nI(\lambda)} = \frac{\lambda e^{-2\lambda}}{n}.$$

At the estimate $\hat\lambda = 122/200 = 0.61$ and $n = 200$ this is $9.005 \times 10^{-4}$, a standard deviation of 0.0300.

(b) The share $Z$ of years with no death is a sample proportion with success probability $e^{-\lambda}$, so it is unbiased with variance $e^{-\lambda}(1 - e^{-\lambda})/n$. Its ratio to the bound is

$$\frac{e^{-\lambda}(1 - e^{-\lambda})}{\lambda e^{-2\lambda}} = \frac{e^{\lambda} - 1}{\lambda},$$

which exceeds 1 for every $\lambda > 0$, because $e^\lambda > 1 + \lambda$. At $\lambda = 0.61$ it is 1.378. The observed share is $109/200 = 0.5450$, with standard error 0.0352 at the estimate. The share discards how many deaths occurred in the years that had any; the example on data reduction above found that the indicator keeps the fraction $\lambda/(e^\lambda - 1) = 1/1.378$ of the information.

(c) A sum of independent Poisson counts is a Poisson count, so $S$ has the Poisson distribution with mean $n\lambda$ (The Poisson Distribution proves that Poisson counts add). For any number $a$,

$$E[a^S] = \sum_{s=0}^\infty a^s\,\frac{e^{-n\lambda}(n\lambda)^s}{s!} = e^{-n\lambda}e^{an\lambda} = e^{n\lambda(a-1)}.$$

With $a = 1 - 1/n$ the exponent is $-\lambda$, so $E[W] = e^{-\lambda}$: $W$ is unbiased. With $a^2 = (1-1/n)^2$ the exponent is $n\lambda(-2/n + 1/n^2) = -2\lambda + \lambda/n$, so $E[W^2] = e^{-2\lambda}e^{\lambda/n}$ and

$$\operatorname{Var}(W) = e^{-2\lambda}\big(e^{\lambda/n} - 1\big).$$

Its ratio to the bound is $(e^{\lambda/n} - 1)/(\lambda/n)$, which exceeds 1 for every $n$ and $\lambda$, again because $e^t > 1 + t$ for $t > 0$. So $W$ never attains the bound. Here the ratio is 1.0015, and for a single corps over 20 years with the same mean it would be 1.0154. For the data, $W = (199/200)^{122} = 0.5425$, close to $e^{-0.61} = 0.5434$, the maximum likelihood estimate of $e^{-\lambda}$, which is biased.

This is the situation the attainment corollary describes. The score is linear in $\bar{x}$, so only linear functions of $\lambda$ can be estimated with equality, and $e^{-\lambda}$ is not one. Sufficient Statistics reaches the same $W$ by the Rao–Blackwell theorem and proves, from the completeness of the Poisson total, that it has the smallest variance of all unbiased estimators of $e^{-\lambda}$. So for this function the bound is not attainable by any unbiased estimator.

Problem 3 (the same model in two parametrizations). Lifetimes are independent and exponential.

  • (a) Find the information and the Cramér–Rao bound for the mean lifetime $\theta$.
  • (b) Find them for the rate $\rho = 1/\theta$.
  • (c) Is the bound attained in each case?

Solution. (a) From the table, $I(\theta) = 1/\theta^2$, so unbiased estimators of $\theta$ have variance at least $\theta^2/n$. The score of the sample is $U(\theta) = \sum_i (x_i - \theta)/\theta^2 = \frac{n}{\theta^2}(\bar{x} - \theta)$, linear in $\bar{x}$, so $\bar{x}$ attains the bound; indeed $\operatorname{Var}(\bar{X}) = \theta^2/n$.

(b) By the change-of-parameter theorem with $\theta = g(\rho) = 1/\rho$ and $g'(\rho) = -1/\rho^2$,

$$I^*(\rho) = \frac{1}{\theta^2}\cdot\frac{1}{\rho^4} = \rho^2\cdot\frac{1}{\rho^4} = \frac{1}{\rho^2},$$

so unbiased estimators of $\rho$ have variance at least $\rho^2/n$. As a check, directly: $\ln f = \ln\rho - \rho x$, $u = 1/\rho - x$, and $\operatorname{Var}(u) = \operatorname{Var}(X) = 1/\rho^2$.

(c) For $\theta$, yes. For $\rho$, no. The score in $\rho$ is $U = n(1/\rho - \bar{x})$, linear in $\bar{x}$, so by the attainment corollary the only functions of $\rho$ that can be estimated with equality are linear functions of $E[\bar{X}] = 1/\rho$. The rate $\rho$ itself is not a linear function of $1/\rho$. The same model has an estimator that attains the bound in one parametrization and none in the other: attainment belongs to the function being estimated, not to the model.

Problem 4 (the sample variance of a normal sample). For a sample of $n$ from a normal distribution with unknown mean $\mu$ and unknown variance $v = \sigma^2$, compare the variance of the sample variance $S^2$ with the Cramér–Rao bound for unbiased estimators of $v$.

Solution. $\operatorname{Var}(S^2) = 2\sigma^4/(n-1)$, against a bound of $2\sigma^4/n$: the ratio is $n/(n-1)$. For the bound, take the parameters $(\mu, v)$. The entry of the information matrix for $v$ is $1/(2v^2)$, from the table. The off-diagonal entry is minus the mean of $\partial u_\mu/\partial v = -(x-\mu)/v^2$, which is 0. The matrix is diagonal, so by the several-parameter theorem the bound for an unbiased estimator of $v$ is $\frac{1}{n}\cdot 2v^2 = 2\sigma^4/n$, the same as if $\mu$ were known. For the variance of $S^2$: by The Chi-Squared and t Distributions, $(n-1)S^2/\sigma^2$ has the chi-squared distribution with $n - 1$ degrees of freedom, whose variance is $2(n-1)$. So $\operatorname{Var}(S^2) = \frac{\sigma^4}{(n-1)^2}\cdot 2(n-1) = \frac{2\sigma^4}{n-1}$. The ratio $n/(n-1)$ is 1.111 for $n = 10$ and 1.042 for $n = 25$. $S^2$ is unbiased and does not attain the bound at any $n$. If $\mu$ were known, the estimator $\frac1n\sum_i (x_i - \mu)^2$ would attain it: the score for $v$ is $\sum_i [-(1/(2v)) + (x_i-\mu)^2/(2v^2)] = \frac{n}{2v^2}\big[\frac1n\sum_i(x_i-\mu)^2 - v\big]$, linear in that estimator. Unlike the nuisance parameter in the bound, the unknown mean does cost the estimator something: one degree of freedom.

Summary

  • The score $u(x;\theta) = \partial\ln f(x;\theta)/\partial\theta$ has mean 0 at the true parameter, and its variance equals minus the mean of its derivative, when the support does not depend on $\theta$ and differentiation under the sum or integral is allowed (R1 to R3).
  • The Fisher information is that variance: $I(\theta) = E_\theta[u^2] = -E_\theta[\partial u/\partial\theta]$. A sample of $n$ independent observations has $I_n = nI$.
  • $I = 1/(p(1-p))$ for a proportion, $1/\lambda$ for a Poisson mean, $1/\sigma^2$ for a normal mean, $2/\sigma^2$ for a normal standard deviation, $1/2$ for a Cauchy location.
  • Under $\theta = g(\eta)$ the information is multiplied by $[g'(\eta)]^2$. A statistic carries at most the information of the sample, and all of it when it is sufficient. Locally, $D(\theta_0\|\theta) \approx \frac12 I(\theta_0)(\theta-\theta_0)^2$.
  • Cramér–Rao: $\operatorname{Var}_\theta(W) \ge [\tau'(\theta)]^2/(nI(\theta))$ for an estimator with mean $\tau(\theta)$. Equality holds exactly when the score is a linear function of $W$, which restricts it to linear functions of one statistic in an exponential family.
  • With several parameters the bound is $\nabla\tau^{\mathsf T} I^{-1}\nabla\tau/n$, never below the bound with the other parameters known.
  • For the uniform distribution on $(0,\theta)$ the score does not have mean 0, and $\frac{n+1}{n}\max X_i$ is unbiased with variance $\theta^2/(n(n+2))$, below the formal $\theta^2/n$.

What we have learned

The information in an observation is the variance of its score, and equally the average curvature of its log-probability. It adds over independent observations, rescales with the parameter, and can only shrink when the data are reduced. Because an unbiased estimator must have covariance exactly 1 with the score, the Cauchy–Schwarz inequality puts a floor of $1/(nI(\theta))$ under its variance. The sample proportion, the Poisson mean and the normal mean sit on that floor; most estimators, including the best unbiased estimator of $e^{-\lambda}$ and the sample variance of a normal sample, sit above it. Every step leaned on the regularity conditions, and the uniform distribution, which breaks the first of them, breaks the conclusion too.

The bound is a statement about unbiased estimators at a fixed sample size, and maximum likelihood estimates are usually biased. What the information does say about them is a large-sample statement: under conditions of the same kind, a consistent root of the likelihood equation, which in the standard models is the maximum likelihood estimate, is approximately normal with variance $1/(nI(\theta))$. Asymptotics of the Maximum Likelihood Estimator proves it for one parameter.

References

  • Casella, G., and Berger, R. L. (2002). Statistical Inference (2nd ed.). Duxbury. Chapter 7, "Point Estimation". record · 11 pages
  • Cramér, H. (1946). Mathematical Methods of Statistics. Princeton University Press. record
  • Fisher, R. A. (1922). On the mathematical foundations of theoretical statistics. Philosophical Transactions of the Royal Society of London, Series A, 222, 309–368. https://doi.org/10.1098/rsta.1922.0009 record · 4 pages
  • Rao, C. R. (1945). Information and the accuracy attainable in the estimation of statistical parameters. Bulletin of the Calcutta Mathematical Society, 37, 81–91. Reprinted in S. Kotz and N. L. Johnson (eds.), Breakthroughs in Statistics, Volume I: Foundations and Basic Theory (1992), 235–247. Springer. https://doi.org/10.1007/978-1-4612-0919-5_16 record · 2 pages