Sufficient Statistics

When a sample can be replaced by a summary with nothing lost about the parameter. Sufficiency defined through the conditional distribution of the data; the factorization theorem proved for discrete data; minimal sufficiency and the likelihood-ratio criterion, proved; the Rao–Blackwell theorem proved and measured on Bortkiewicz's horse-kick deaths; completeness proved for the binomial and Poisson families, and the Lehmann–Scheffé theorem that makes an unbiased function of a complete sufficient statistic the best unbiased estimator. Worked problems on weekly deaths, the uniform model, sampling until the tenth case and an unbiased estimate of a squared rate.

The 40 young adults of the unit's hypertension sample were drawn one at a time, and the outcomes, in order, were

0000010100 1110110000 0001001000 0001000000

with 1 for hypertension. Maximum Likelihood Estimation used only one number from this record: the count, 10. Did the reduction from 40 outcomes to one number lose anything about the proportion $p$ of young adults with hypertension? It lost nothing. Given that there are 10 ones, every arrangement of them among the 40 positions has the same probability, $1/847{,}660{,}528$, whatever $p$ is. The order is noise whose distribution is known without knowing $p$, so it cannot help to estimate $p$.

A statistic is any function $T = T(\mathbf{X})$ of the sample $\mathbf{X} = (X_1, \ldots, X_n)$: a count, a mean, the largest observation, a pair of sums. A statistic is sufficient for the parameter $\theta$ if the conditional distribution of the sample, given the value of the statistic, does not depend on $\theta$. This article is the home of sufficiency on this site. It explains why every closed-form estimate of the previous article depended on the data through one or two totals, and why the text models of the information retrieval course may discard word order.

First we prove the claim about the 40 adults and state the definition with care. Then we prove the factorization theorem, which finds sufficient statistics by inspection of the likelihood, and connect sufficiency to the likelihood function. Next we ask for the greatest reduction possible, the minimal sufficient statistic. Then come the two results that make sufficiency useful for estimation: the Rao–Blackwell theorem, which improves any estimator by averaging it over what the sufficient statistic leaves out, and the theorem of Lehmann and Scheffé, which says when the improvement cannot be improved further.

The article assumes The Likelihood Function and Maximum Likelihood Estimation, conditional probability, and the laws of total expectation and total variance, which Random Variables, Expectation and Variance proves for variables with finitely many values and Conditional Expectation for discrete conditioning variables, for pairs with a joint density, and for two mixed settings, one of which covers Problem 2 below. Proofs are written for discrete data, where conditional probabilities are ratios of probabilities. For continuous data the same statements hold with densities, and the general proofs need measure theory; they are cited. Notation follows the unit: $f(\mathbf{x}; \theta)$ is the joint probability mass function or density of the sample, $P_\theta$ and $E_\theta$ are probability and expectation when the parameter is $\theta$, and "i.i.d." means independent and identically distributed.

The order carries no information

Let $X_1, \ldots, X_n$ be independent Bernoulli trials with success probability $p$, and let $T = X_1 + \cdots + X_n$ be the number of successes. Fix a sequence $\mathbf{x}$ of zeros and ones with $t$ ones. Its probability is $p^t(1 - p)^{n - t}$, and the probability that $T = t$ is the binomial $\binom{n}{t}p^t(1 - p)^{n - t}$. The event $\{\mathbf{X} = \mathbf{x}\}$ is contained in the event $\{T = t\}$, so for $0 < p < 1$

$$P_p(\mathbf{X} = \mathbf{x} \mid T = t) = \frac{P_p(\mathbf{X} = \mathbf{x})}{P_p(T = t)} = \frac{p^t(1 - p)^{n - t}}{\binom{n}{t}p^t(1 - p)^{n - t}} = \frac{1}{\binom{n}{t}}.$$

The parameter cancels. Given the count, the sequence is equally likely to be any of the $\binom{n}{t}$ arrangements.

For the 40 adults, the observed sequence has probability $4.24 \times 10^{-12}$ under $p = 0.10$, $1.70 \times 10^{-10}$ under $p = 0.25$ and $2.32 \times 10^{-11}$ under $p = 0.40$. The count 10 has probability 0.0036, 0.1444 and 0.0196 under the same three values. Each ratio is $1/847{,}660{,}528 = 1.18 \times 10^{-9}$.

Here is a way to see what that means. Someone who is told only that $T = 10$ can produce a sequence with exactly the distribution of the original sample: place the ten ones in 10 positions chosen at random from the 40. No knowledge of $p$ is needed to do it. Whatever could be computed from the original sequence could be computed, with the same distribution, from this reconstruction. The sample is the count plus a random arrangement that anyone can generate.

Example (what the first adult adds). Given $T = 10$, the probability that the first adult drawn has hypertension is $10/40 = 0.25$, for every $p$: of the $\binom{40}{10}$ equally likely arrangements, $\binom{39}{9}$ begin with a one, and $\binom{39}{9}/\binom{40}{10} = 10/40$. Before the count is known, that probability is $p$. After it is known, the first outcome is a draw from a known distribution and tells nothing more about $p$.

The definition, and two contrasts

Sufficiency. A statistic $T = T(\mathbf{X})$ is sufficient for $\theta$ if, for every value $t$ of $T$ and every sample point $\mathbf{x}$, the conditional probability $P_\theta(\mathbf{X} = \mathbf{x} \mid T = t)$ is the same for all $\theta$ under which $P_\theta(T = t) > 0$.

The calculation above proves that the number of successes is sufficient for $p$ in Bernoulli trials. Two contrasts show that the definition has content.

Example (a statistic that is not sufficient). Let $T'$ be the number with hypertension among the first 20 adults only, here 7. Given $T' = 7$, the first 20 outcomes are equally likely to be any of the $\binom{20}{7}$ arrangements, but the last 20 are still Bernoulli trials. The conditional probability of the observed sequence, with 3 ones among its last 20, is $\frac{1}{\binom{20}{7}}\,p^{3}(1 - p)^{17}$, which depends on $p$. The last 20 outcomes still carry information, and $T'$ is not sufficient.

Example (a statistic that is sufficient but wasteful). Let $T'' = (T_1, T_2)$ be the pair of counts in the first and the last 20 adults, here $(7, 3)$. Given the pair, each half is a random arrangement, so the conditional probability of the sequence is $1\big/\big[\binom{20}{7}\binom{20}{3}\big] = 1/(77{,}520 \times 1{,}140) = 1/88{,}372{,}800 = 1.13 \times 10^{-8}$, free of $p$. The pair is sufficient. So is the whole sample, trivially: given the sample, the sample is known. Sufficient statistics are not unique, and the useful ones are those that reduce the data most. The section on minimal sufficiency makes that precise.

Poisson counts and their total

The second running example is Bortkiewicz's table of deaths from horse kicks: 200 corps-years with 122 deaths in all (The Poisson Distribution). Model the 200 counts as i.i.d. Poisson with mean $\lambda$. Is the total sufficient?

Let $T = X_1 + \cdots + X_n$. A sum of independent Poisson counts is Poisson, as that article proves, so $T$ is Poisson with mean $n\lambda$. For counts $x_1, \ldots, x_n$ with total $t$,

$$P_\lambda(\mathbf{X} = \mathbf{x} \mid T = t) = \frac{\prod_i e^{-\lambda}\lambda^{x_i}/x_i!}{e^{-n\lambda}(n\lambda)^t/t!} = \frac{t!}{x_1!\cdots x_n!}\Big(\frac{1}{n}\Big)^{t},$$

because the numerator is $e^{-n\lambda}\lambda^{t}/\prod x_i!$. The rate cancels, so the total is sufficient for $\lambda$. The conditional distribution has a name: it is the multinomial distribution of $t$ objects thrown independently into $n$ equally likely cells. Given 122 deaths, the model says each death fell in one of the 200 corps-years chosen at random.

A small check with three counts: the outcome $(2, 0, 1)$ has probability 0.01821 when $\lambda = 0.61$ and 0.00992 when $\lambda = 2$, and the total 3 has probability 0.16385 and 0.08924. Both ratios are $\frac{3!}{2!\,0!\,1!}\big(\frac{1}{3}\big)^3 = 0.1111$.

The course has already used this fact without the name. To test the Poisson model on the horse kicks, The Poisson Distribution simulated tables "given the total number of deaths, with each death equally likely to fall in any corps-year". That simulation needs no value of $\lambda$, and the reason is that the total is sufficient. The index of dispersion computed there, 199.31 on 199 degrees of freedom with a p-value of 0.48, examines the part of the data that sufficiency sets aside.

The factorization theorem

Checking the definition requires the distribution of $T$, which is not always easy to find. The factorization theorem replaces the check by an inspection of the joint mass function.

Theorem (Fisher–Neyman factorization, discrete case). Let $\mathbf{X}$ be discrete with joint mass function $f(\mathbf{x}; \theta)$. A statistic $T$ is sufficient for $\theta$ if and only if there are functions $g$ and $h$ such that

$$f(\mathbf{x}; \theta) = g(T(\mathbf{x}); \theta)\,h(\mathbf{x}) \qquad \text{for all } \mathbf{x} \text{ and } \theta,$$

where $h$ does not depend on $\theta$, and $g$ depends on the data only through $T(\mathbf{x})$.

Here is the idea in one sentence. The factor $g$ is, up to a constant, the probability of the observed value of $T$, and $h$ is what remains, the conditional probability of the sample given $T$; the theorem says sufficiency is exactly the statement that the remainder is free of $\theta$.

Proof. Factorization implies sufficiency. Suppose $f(\mathbf{x}; \theta) = g(T(\mathbf{x}); \theta)h(\mathbf{x})$. The probability of $T = t$ is the sum of $f$ over the sample points that $T$ sends to $t$:

$$P_\theta(T = t) = \sum_{\mathbf{y}:\,T(\mathbf{y}) = t} g(T(\mathbf{y}); \theta)\,h(\mathbf{y}) = g(t; \theta)\sum_{\mathbf{y}:\,T(\mathbf{y}) = t} h(\mathbf{y}) = g(t; \theta)\,H(t),$$

where $H(t)$ is the sum of $h$ over those points. Take any $\theta$ with $P_\theta(T = t) > 0$, and any $\mathbf{x}$. If $T(\mathbf{x}) \ne t$ the conditional probability $P_\theta(\mathbf{X} = \mathbf{x} \mid T = t)$ is 0. If $T(\mathbf{x}) = t$,

$$P_\theta(\mathbf{X} = \mathbf{x} \mid T = t) = \frac{f(\mathbf{x}; \theta)}{P_\theta(T = t)} = \frac{g(t; \theta)\,h(\mathbf{x})}{g(t; \theta)\,H(t)} = \frac{h(\mathbf{x})}{H(t)}.$$

In both cases the value does not depend on $\theta$, so $T$ is sufficient.

Sufficiency implies factorization. Suppose $T$ is sufficient. Consider a sample point $\mathbf{x}$ that has positive probability under at least one $\theta$, and write $t = T(\mathbf{x})$. By sufficiency the conditional probability $P_\theta(\mathbf{X} = \mathbf{x} \mid T = t)$ has one common value for all $\theta$ with $P_\theta(T = t) > 0$; call it $h(\mathbf{x})$. For a point with probability zero under every $\theta$, set $h(\mathbf{x}) = 0$. Set $g(t; \theta) = P_\theta(T = t)$. Since $\{\mathbf{X} = \mathbf{x}\}$ is contained in $\{T = t\}$,

$$f(\mathbf{x}; \theta) = P_\theta(\mathbf{X} = \mathbf{x},\ T = t) = P_\theta(T = t)\,P_\theta(\mathbf{X} = \mathbf{x} \mid T = t) = g(T(\mathbf{x}); \theta)\,h(\mathbf{x})$$

whenever $P_\theta(T = t) > 0$. When $P_\theta(T = t) = 0$, both sides are zero, because $f(\mathbf{x}; \theta) \le P_\theta(T = t)$. $\square$

For continuous data the theorem holds with $f$ a joint density. Its proof cannot divide by $P_\theta(T = t)$, which is zero, and needs the measure-theoretic definition of conditional probability. Halmos and Savage (1949) give the result in that generality, as "a direct generalization to the abstract case of the well known Fisher-Neyman result", for families of distributions that have densities with respect to one fixed measure, a condition that covers both discrete and continuous models. This article uses the density form without proof.

Sufficient statistics by inspection

  • Bernoulli trials. $f(\mathbf{x}; p) = p^{\sum x_i}(1 - p)^{n - \sum x_i}$. Take $g(t; p) = p^t(1 - p)^{n - t}$ and $h = 1$: the count is sufficient.
  • Poisson counts. $f(\mathbf{x}; \lambda) = e^{-n\lambda}\lambda^{\sum x_i}\big/\prod x_i!$. Take $g(t; \lambda) = e^{-n\lambda}\lambda^{t}$ and $h(\mathbf{x}) = 1/\prod x_i!$: the total is sufficient.
  • Normal observations, both parameters unknown. By the split of the sum of squares in The Likelihood Function, the joint density is $(2\pi\sigma^2)^{-n/2}\exp\!\big(-[\sum(x_i - \bar{x})^2 + n(\bar{x} - \mu)^2]/(2\sigma^2)\big)$. It depends on the data only through the pair $\big(\bar{x},\ \sum(x_i - \bar{x})^2\big)$, so with $h = 1$ that pair is sufficient for $(\mu, \sigma^2)$. Equivalently $(\bar{x}, s^2)$, or $(\sum x_i, \sum x_i^2)$: each pair determines the others.
  • Uniform on $[0, \theta]$. The joint density is $\theta^{-n}$ if $0 \le x_i \le \theta$ for all $i$ and 0 otherwise. Write $\mathbf{1}\{A\}$ for the indicator of a condition $A$, equal to 1 when it holds and 0 when not. Then $f(\mathbf{x}; \theta) = \theta^{-n}\,\mathbf{1}\{x_{(n)} \le \theta\}\cdot\mathbf{1}\{x_{(1)} \ge 0\}$, with $x_{(n)}$ the largest and $x_{(1)}$ the smallest observation. The first two factors depend on the data only through $x_{(n)}$: the largest observation is sufficient.
  • Tokens from a vocabulary. Under the model of Naive Bayes and Language Models for Retrieval, the tokens of a text are independent draws with probability $\theta_t$ for term $t$, and a sequence in which term $t$ occurs $T_t$ times has probability $\prod_t\theta_t^{T_t}$. The vector of term counts is sufficient. This is the precise sense in which a bag of words loses nothing: under the model, word order is a random arrangement. The eight training tokens of the class China in that article, with counts 5, 1, 1, 1, can be ordered in $8!/(5!\,1!\,1!\,1!) = 336$ ways, and given the counts each order has probability $1/336 = 0.0030$ whatever the term probabilities are.
  • A bigram model. Under the bigram language model, a text $x_1 \ldots x_n$ has probability $\pi(x_1)\prod_{a, b}P(b \mid a)^{N_{ab}}$, where $\pi$ is the distribution of the first token and $N_{ab}$ counts the positions where $a$ is followed by $b$. The first token and the table of bigram counts are sufficient. The strings aabab and abaab begin with the same character and have the same bigram counts (aa once, ab twice, ba once), so every bigram model gives them the same probability; with $\pi(a) = 0.5$, $P(a \mid a) = 0.3$ and $P(a \mid b) = 0.6$ it is $0.5 \times 0.3 \times 0.7 \times 0.6 \times 0.7 = 0.0441$ for both. Training such a model reduces to counting pairs.

The first three are instances of one pattern. A family of distributions whose mass function or density can be written as

$$f(x; \theta) = h(x)\exp\!\big(\eta(\theta)\,T(x) - A(\theta)\big)$$

is called a one-parameter exponential family. For an i.i.d. sample the joint function is $\big[\prod h(x_i)\big]\exp\!\big(\eta(\theta)\sum T(x_i) - nA(\theta)\big)$, so by the theorem $\sum T(x_i)$ is sufficient, a single number however large the sample. The Bernoulli mass function is $\exp\!\big(x\ln\frac{p}{1 - p} + \ln(1 - p)\big)$, with $T(x) = x$; the Poisson is $\frac{1}{x!}\exp(x\ln\lambda - \lambda)$, again with $T(x) = x$. With several parameters $\eta(\theta)T(x)$ becomes a sum of such products, as for the normal with $T(x) = (x, x^2)$. Generalized Linear Models builds regression on this structure. Which families allow a reduction to a statistic of fixed size, whatever the sample size, is a question this article does not pursue.

Sufficiency and the likelihood

The factorization says something directly about the likelihood function $L(\theta) = f(\mathbf{x}; \theta)$.

Theorem (the likelihood depends on the data through a sufficient statistic). Let $T$ be sufficient. If two samples $\mathbf{x}$ and $\mathbf{y}$, each with positive probability under some $\theta$, have $T(\mathbf{x}) = T(\mathbf{y})$, then their likelihood functions are proportional: $L(\theta; \mathbf{x}) = c\,L(\theta; \mathbf{y})$ for all $\theta$, with $c > 0$ free of $\theta$. Consequently they have the same likelihood ratios, the same likelihood intervals and the same maximum likelihood estimates.

Proof. By the factorization theorem, $L(\theta; \mathbf{x}) = g(T(\mathbf{x}); \theta)h(\mathbf{x})$ and $L(\theta; \mathbf{y}) = g(T(\mathbf{y}); \theta)h(\mathbf{y}) = g(T(\mathbf{x}); \theta)h(\mathbf{y})$. A sample point with positive probability under some $\theta$ has $h > 0$. So $L(\theta; \mathbf{x}) = [h(\mathbf{x})/h(\mathbf{y})]\,L(\theta; \mathbf{y})$. Proportional likelihoods have equal ratios by the first theorem of The Likelihood Function, and they are maximized at the same values of $\theta$. $\square$

So the maximum likelihood estimate, when it is unique, is a function of every sufficient statistic. That is why the estimates of the previous article were functions of a count, a total, or a mean and a sum of squares.

Example (two different samples, one likelihood). Under the normal model, the samples $(1, 2, 3)$ and $(6/7,\ 17/7,\ 19/7) = (0.8571, 2.4286, 2.7143)$ both have sum 6 and sum of squares 14. They share $\bar{x} = 2$ and $\sum(x_i - \bar{x})^2 = 2$, and their log-likelihoods agree everywhere: $-3.7568$ at $(\mu, \sigma^2) = (2, 1)$, $-6.5863$ at $(0, 4)$, and $-10.4671$ at $(3.5, 0.5)$, for both. No analysis based on the normal likelihood can tell the two samples apart.

Minimal sufficiency

The count of the 40 adults is sufficient, and so is the pair of half-counts $(7, 3)$, and so is the full sequence. Which summary is the smallest that still loses nothing? A reduction of the data is a function of it, so "smaller" means "a function of".

A sufficient statistic $T$ is minimal sufficient if it is a function of every other sufficient statistic: for every sufficient $T'$ there is a function $r$ with $T = r(T')$. The count is a function of the pair of half-counts (their sum), and of the sequence. The pair is not a function of the count. The following criterion identifies a minimal sufficient statistic from the likelihood alone. Throughout, the sample space means the set of sample points that have positive probability under at least one $\theta$.

Theorem (the likelihood-ratio criterion, discrete case). Suppose a statistic $T$ has the following property for every pair of sample points $\mathbf{x}$, $\mathbf{y}$: $T(\mathbf{x}) = T(\mathbf{y})$ if and only if there is a constant $c > 0$, not depending on $\theta$, with $f(\mathbf{x}; \theta) = c\,f(\mathbf{y}; \theta)$ for all $\theta$. Then $T$ is minimal sufficient.

Here is the idea in one sentence. Two samples should be merged by a summary exactly when they have proportional likelihoods; merging fewer keeps noise, and merging more loses information.

Proof. $T$ is sufficient. For each value $t$ of $T$ choose one sample point $\mathbf{x}_t$ with $T(\mathbf{x}_t) = t$. Any sample point $\mathbf{x}$ has $T(\mathbf{x}) = T(\mathbf{x}_{T(\mathbf{x})})$, so by the property there is a constant $c(\mathbf{x}) > 0$ with $f(\mathbf{x}; \theta) = c(\mathbf{x})\,f(\mathbf{x}_{T(\mathbf{x})}; \theta)$ for all $\theta$. Put $h(\mathbf{x}) = c(\mathbf{x})$ and $g(t; \theta) = f(\mathbf{x}_t; \theta)$. Then $f(\mathbf{x}; \theta) = g(T(\mathbf{x}); \theta)h(\mathbf{x})$, and $T$ is sufficient by the factorization theorem.

$T$ is a function of every sufficient statistic. Let $T'$ be sufficient, with factorization $f(\mathbf{x}; \theta) = g'(T'(\mathbf{x}); \theta)\,h'(\mathbf{x})$. Every point of the sample space has $h' > 0$, since otherwise its probability would be zero under every $\theta$. Take two sample points with $T'(\mathbf{x}) = T'(\mathbf{y})$. Then

$$f(\mathbf{x}; \theta) = g'(T'(\mathbf{x}); \theta)\,h'(\mathbf{x}) = g'(T'(\mathbf{y}); \theta)\,h'(\mathbf{x}) = \frac{h'(\mathbf{x})}{h'(\mathbf{y})}\,f(\mathbf{y}; \theta)$$

for all $\theta$, with a positive constant free of $\theta$. By the property, $T(\mathbf{x}) = T(\mathbf{y})$. So the value of $T$ is determined by the value of $T'$: $T$ is a function of $T'$. $\square$

The criterion is the construction that Lehmann and Scheffé (1950, Section 6) give for a minimal sufficient statistic. Their operation puts two sample points $\mathbf{x}$ and $\mathbf{x}^0$ in the same set when $f(\mathbf{x}; \theta) = k\,f(\mathbf{x}^0; \theta)$ for all $\theta$, with a nonzero $k$ that does not depend on $\theta$: "Roughly speaking, we may say that $D(x^0)$ consists of all $x$ for which the ratio $f_\theta(x)/f_\theta(x^0)$ is independent of $\theta$" (p. 256 of the reprint, whose page numbers are cited throughout). They prove that the resulting partition belongs to a minimal sufficient statistic when the family of distributions is countable (Theorem 6.1), and, for a family with densities that is not countable, that a minimal sufficient statistic exists and is obtained by applying the operation to a suitable countable subfamily of the densities, which can always be found when the sample space is Euclidean (Theorem 6.3, p. 264). Their examples include the count for Bernoulli trials, the pair $(\sum x_i, \sum x_i^2)$ for normal observations and the sorted sample for the Cauchy location model (Examples 6.1 to 6.3), the three cases worked below. Taraldsen (2026) lists four textbooks that state the criterion. For continuous data the criterion is used with densities, by an argument that again needs measure theory and is not given here. It also needs a care that the discrete case does not, and Lehmann and Scheffé point it out: a density can be changed on a set of probability zero without changing the model, and "the result of this operation does depend on which determination of the family $\mathfrak{p}$ is used" (p. 256), which is why their general theorem applies it to a countable subfamily. Taraldsen's Example 1 changes a normal density in this way so that the criterion, applied to the changed version, is satisfied by a statistic that is not minimal. The examples below use the usual continuous versions of the densities. The same paper relates minimality to completeness: a sufficient statistic whose family of distributions is complete, or only boundedly complete (the condition imposed on bounded functions alone), is equivalent to a minimal sufficient statistic whenever one exists (Theorem 3.1, p. 244).

Examples.

  • Bernoulli trials. For sequences with $s$ and $t$ ones, $f(\mathbf{x}; p)/f(\mathbf{y}; p) = [p/(1 - p)]^{s - t}$, constant in $p$ exactly when $s = t$. The count is minimal sufficient.
  • Poisson counts. The ratio is $\lambda^{\sum x_i - \sum y_i}\prod y_i!/\prod x_i!$, constant in $\lambda$ exactly when the totals agree. The total is minimal sufficient.
  • Normal observations. The ratio of two joint densities is $\exp$ of $-\frac{1}{2\sigma^2}\big[\sum x_i^2 - \sum y_i^2\big] + \frac{\mu}{\sigma^2}\big[\sum x_i - \sum y_i\big]$. That is constant in $(\mu, \sigma^2)$ exactly when both brackets are zero, so $(\sum x_i, \sum x_i^2)$ is minimal sufficient (by the density form of the criterion).
  • A model with no reduction. For the Cauchy density $1/[\pi(1 + (x - \theta)^2)]$ of the previous article, the ratio of joint densities is $\prod_i(1 + (y_i - \theta)^2)\big/\prod_i(1 + (x_i - \theta)^2)$. Numerator and denominator are polynomials in $\theta$ of degree $2n$ with leading coefficient 1. If their ratio is a constant the constant is 1 and the polynomials are equal, so they have the same roots. The roots of $1 + (x_i - \theta)^2$ are $\theta = x_i \pm \sqrt{-1}$, so equal polynomials mean that the $x_i$ and the $y_i$ are the same numbers in some order. The minimal sufficient statistic is the sorted sample. Only the order of observation can be discarded, and the sample cannot be reduced to a fixed number of summaries.

The last example marks the limit of the idea. In every i.i.d. model the sorted sample is sufficient, because the joint density $\prod f(x_i; \theta)$ is unchanged by reordering. Reduction beyond that is a special property of particular families.

The Rao–Blackwell theorem

Sufficiency was defined as a property of information. Its use in estimation is a theorem: any estimator that is not a function of a sufficient statistic can be improved, and the improvement is explicit.

An estimator of a quantity $\tau(\theta)$ is a statistic $W$ used to estimate it. It is unbiased if $E_\theta[W] = \tau(\theta)$ for all $\theta$, and its mean squared error is $E_\theta[(W - \tau(\theta))^2] = \operatorname{Var}_\theta(W) + (E_\theta[W] - \tau(\theta))^2$, the variance plus the squared bias.

Theorem (Rao–Blackwell). Let $T$ be sufficient for $\theta$, let $W$ be an estimator of $\tau(\theta)$ with finite variance, and define $\varphi(t) = E[W \mid T = t]$. Then:

  1. $\varphi(T)$ is a statistic: it can be computed from the data without knowing $\theta$.
  2. $E_\theta[\varphi(T)] = E_\theta[W]$ for all $\theta$. In particular, if $W$ is unbiased, so is $\varphi(T)$.
  3. $\operatorname{Var}_\theta(\varphi(T)) \le \operatorname{Var}_\theta(W)$ for all $\theta$, and so the mean squared error of $\varphi(T)$ is no larger than that of $W$. The variances are equal only if $W$ is already a function of $T$ (with probability 1).

Here is the idea in one sentence. Whatever variation $W$ has among samples with the same value of $T$ is pure noise, since those samples differ only by a random arrangement that does not involve $\theta$, and averaging over it removes that noise without changing the mean.

Proof. (1) The conditional expectation is $\varphi(t) = \sum_{\mathbf{x}} W(\mathbf{x})\,P(\mathbf{X} = \mathbf{x} \mid T = t)$. Because $T$ is sufficient, the conditional probabilities do not depend on $\theta$, so neither does $\varphi$. This is the only place sufficiency is used, and without it $\varphi$ would not be an estimator at all.

(2) By the law of total expectation, $E_\theta[W] = E_\theta\big[E[W \mid T]\big] = E_\theta[\varphi(T)]$.

(3) By the law of total variance,

$$\operatorname{Var}_\theta(W) = E_\theta\big[\operatorname{Var}(W \mid T)\big] + \operatorname{Var}_\theta\big(E[W \mid T]\big) = E_\theta\big[\operatorname{Var}(W \mid T)\big] + \operatorname{Var}_\theta(\varphi(T)).$$

The first term is the expected value of a non-negative quantity, so it is at least 0, and $\operatorname{Var}_\theta(\varphi(T)) \le \operatorname{Var}_\theta(W)$. The two estimators have the same mean by (2), so the same bias, and the mean squared error of $\varphi(T)$ is no larger. Equality requires $E_\theta[\operatorname{Var}(W \mid T)] = 0$, that is, $\operatorname{Var}(W \mid T = t) = 0$ for every $t$ of positive probability: given $T$, $W$ is constant, so $W$ is a function of $T$. $\square$

Blackwell (1947, p. 105) stated the result in this form: whenever there is a sufficient statistic $u$ and an unbiased estimate $t$ that is not a function of $u$ only, $E(t \mid u)$ is an unbiased estimate with a smaller variance. Rao (1945) is the other source to which the theorem's name refers. In the section of his paper on a single parameter he assumes a sufficient statistic $T$, so that the density of the sample factors as the density $\Phi(T, \theta)$ of $T$ times a function free of $\theta$. He writes the mean of an unbiased estimate $t$ as an integral against $\Phi(T, \theta)$, which gives a function $f(T)$, free of $\theta$, that is also unbiased, and he shows that $V[f(T)]$ is not greater than $V(t)$. His conclusion: "if a sufficient statistic and an unbiassed estimate exist for $\theta$, then the best unbiassed estimate of $\theta$ is an explicit function of the sufficient statistic" (pp. 237–238 of the reprint). The two names were joined early: Lehmann and Scheffé (1950) call the result "a theorem of Rao (1945) and Blackwell (1947) on unbiased estimation" (p. 234 of the reprint).

The probability of a year without a death

Back to the horse kicks. What is the probability $\tau(\lambda) = e^{-\lambda}$ that a corps passes a year with no death from a horse kick? Start from the crudest unbiased estimator imaginable: look at the first corps-year only, and report 1 if it had no death and 0 if it had any. Call it $W = \mathbf{1}\{X_1 = 0\}$. Its mean is $P_\lambda(X_1 = 0) = e^{-\lambda}$, so it is unbiased, and it ignores 199 of the 200 observations.

Now apply the theorem with the sufficient statistic $T = \sum X_i$. Given $T = t$, the $t$ deaths fall independently into the $n$ corps-years with equal probabilities, by the multinomial result above. The first corps-year has no death exactly when all $t$ deaths fall elsewhere:

$$\varphi(t) = E[W \mid T = t] = P(X_1 = 0 \mid T = t) = \Big(1 - \frac{1}{n}\Big)^{t}.$$

For the data, $\varphi(122) = (199/200)^{122} = 0.5425$. An estimator that used one observation has become one that uses the total of all 200, with no ingenuity beyond a conditional expectation.

Is it really unbiased, and how much better is it? Both follow from one identity. If $T$ is Poisson with mean $m$, then for any number $a$

$$E[a^T] = \sum_{t=0}^{\infty}a^t\,\frac{e^{-m}m^t}{t!} = e^{-m}\sum_{t=0}^{\infty}\frac{(am)^t}{t!} = e^{-m}e^{am} = e^{m(a - 1)}.$$

With $m = n\lambda$ and $a = 1 - 1/n$: $E_\lambda[\varphi(T)] = e^{n\lambda(-1/n)} = e^{-\lambda}$, unbiased as the theorem promised. With $a^2 = 1 - 2/n + 1/n^2$ in place of $a$: $E_\lambda[\varphi(T)^2] = e^{-2\lambda + \lambda/n}$, so

$$\operatorname{Var}_\lambda(\varphi(T)) = e^{-2\lambda}\big(e^{\lambda/n} - 1\big).$$

Two other estimators are natural. The share of corps-years with no death, $109/200 = 0.545$, is the average of the indicators $\mathbf{1}\{X_i = 0\}$; it is unbiased with variance $e^{-\lambda}(1 - e^{-\lambda})/n$. The maximum likelihood estimate is $e^{-\bar{x}} = e^{-0.61} = 0.5434$, by the invariance theorem. At $\lambda = 0.61$ and $n = 200$:

Estimator of $e^{-\lambda}$ Value on the data Bias Variance Simulated variance (30 seeds)
First corps-year only, $\mathbf{1}\{X_1 = 0\}$ 0 or 1 0 0.2481
Share of zeros 0.545 0 0.001241 0.001239 (0.001233 to 0.001253)
Rao–Blackwell, $(1 - 1/n)^T$ 0.5425 0 0.000902 0.000902 (0.000895 to 0.000909)
Maximum likelihood, $e^{-T/n}$ 0.5434 0.000828 0.000900 0.000901 (0.000894 to 0.000907)

The exact variances come from the identity above (for the maximum likelihood estimate, with $a = e^{-1/n}$ and $a = e^{-2/n}$). The last column is a check: 20,000 simulated tables of 200 Poisson counts for each of 30 seeds, with the median and interquartile range of the variance across seeds. Conditioning cut the variance of the crude estimator by a factor of 275.

The comparison with the share of zeros is the instructive one. Applying the theorem to the share of zeros gives the same $\varphi$, since each indicator has the same conditional expectation. So $(1 - 1/n)^T$ improves on it too: its variance is smaller by the factor 0.727, and the share of zeros would need 1.376 times as many corps-years for the same precision. The share of zeros looks only at whether each count is zero. The improved estimator uses the Poisson model to learn about zeros from the ones, twos and threes as well. That gain is bought with the model: if the counts are not Poisson, the share of zeros still estimates the probability of a zero, and $(1 - 1/n)^T$ does not.

Variance of the Rao–Blackwell estimator of the probability of a zero count, as a fraction of the variance of the share of zeros, against the Poisson mean, for samples of 200. The dot marks the horse-kick mean 0.61, where the fraction is 0.727. Notice that the two estimators are nearly equivalent when the mean is small, when almost all the information is in whether a count is zero, and that the share of zeros becomes nearly worthless as the mean grows and zeros become rare.
Variance of the Rao–Blackwell estimator of the probability of a zero count, as a fraction of the variance of the share of zeros, against the Poisson mean, for samples of 200. The dot marks the horse-kick mean 0.61, where the fraction is 0.727. Notice that the two estimators are nearly equivalent when the mean is small, when almost all the information is in whether a count is zero, and that the share of zeros becomes nearly worthless as the mean grows and zeros become rare.

For large $n$, $e^{\lambda/n} - 1$ is close to $\lambda/n$, and the ratio of the two variances is close to $\lambda/(e^{\lambda} - 1)$: 0.951 at $\lambda = 0.1$, 0.726 at 0.61, 0.582 at 1, 0.313 at 2 and 0.034 at 5.

Example (a common misstatement). "The Rao–Blackwell estimator is the best estimator." The theorem says it is at least as good as the estimator it started from. The table shows a competitor that is not unbiased and does slightly better: the maximum likelihood estimate has mean squared error $0.000900 + 0.000828^2 = 0.000901$, against 0.000902. Whether the improved estimator is best among unbiased estimators is a separate question, answered by completeness below.

The variance of one trial

A second case, on the 40 adults. The variance of a single Bernoulli trial is $\tau(p) = p(1 - p)$. A crude unbiased estimator is $W = X_1(1 - X_2)$, which is 1 when the first adult has hypertension and the second does not: by independence $E_p[W] = p(1 - p)$. Given $T = t$, all arrangements are equally likely, and the number with a one in the first position and a zero in the second is $\binom{n - 2}{t - 1}$. So

$$\varphi(t) = \frac{\binom{n - 2}{t - 1}}{\binom{n}{t}} = \frac{t(n - t)}{n(n - 1)}.$$

For the sample, $W = 0$ (the first adult drawn did not have hypertension) and $\varphi(10) = \frac{10 \times 30}{40 \times 39} = 0.1923$. The maximum likelihood estimate is $\hat{p}(1 - \hat{p}) = 0.1875$. The two differ by the factor $n/(n - 1)$: $\varphi(T) = \frac{n}{n - 1}\hat{p}(1 - \hat{p})$, the Bernoulli counterpart of dividing a sum of squares by $n - 1$ instead of $n$. At the population value $p_0 = 0.2294$, where $p_0(1 - p_0) = 0.1768$, the crude estimator has variance 0.1455 and the improved one 0.00133, smaller by a factor of 109. The maximum likelihood estimate has mean $\frac{n - 1}{n}p_0(1 - p_0) = 0.1723$; the script confirms both means by exact summation over the binomial distribution.

Completeness and the best unbiased estimator

Rao–Blackwell improves an estimator. Does the result depend on which estimator was improved? For the Poisson zero probability two starting points led to the same $\varphi$. In general, different unbiased estimators could lead to different functions of $T$. A property of the family of distributions of $T$ rules that out.

A statistic $T$ is complete (for a given model) if the only function $g$ with $E_\theta[g(T)] = 0$ for every $\theta$ is the function that is zero at every value $T$ can take. In words: no non-trivial function of $T$ is an unbiased estimator of zero.

Theorem (the binomial count is complete). Let $T$ be binomial with $n$ trials and success probability $p$, $0 < p < 1$. If $E_p[g(T)] = 0$ for all $p$, then $g(0) = g(1) = \cdots = g(n) = 0$.

Proof. Write $r = p/(1 - p)$, which takes every positive value as $p$ runs over $(0, 1)$. Then

$$E_p[g(T)] = \sum_{t=0}^{n}g(t)\binom{n}{t}p^t(1 - p)^{n - t} = (1 - p)^n\sum_{t=0}^{n}g(t)\binom{n}{t}r^t.$$

The factor $(1 - p)^n$ is positive, so the polynomial $\sum_t g(t)\binom{n}{t}r^t$ is zero for every $r > 0$. A polynomial of degree at most $n$ with more than $n$ roots has all coefficients zero. So $g(t)\binom{n}{t} = 0$, and $g(t) = 0$, for every $t$. $\square$

Theorem (the Poisson total is complete). Let $T$ be Poisson with mean $n\lambda$, $\lambda > 0$. If $E_\lambda[g(T)]$ exists and equals 0 for all $\lambda$, then $g(t) = 0$ for $t = 0, 1, 2, \ldots$

Proof. $E_\lambda[g(T)] = e^{-n\lambda}\sum_{t=0}^{\infty}g(t)\frac{n^t}{t!}\lambda^t$. The factor $e^{-n\lambda}$ is positive, so the power series $\sum_t c_t\lambda^t$ with $c_t = g(t)n^t/t!$ converges and equals zero for every $\lambda > 0$. Such a series has all coefficients zero. It converges for every $\lambda > 0$, so its radius of convergence is infinite, and it may be differentiated term by term on the whole line. Its $t$th derivative is continuous, and is zero on $(0, \infty)$ because the sum is zero there; hence it is zero at 0 as well. That derivative at 0 is $t!\,c_t$. So $c_t = 0$ and $g(t) = 0$ for every $t$. $\square$

Theorem (Lehmann–Scheffé). Let $T$ be a complete sufficient statistic, and let $\varphi(T)$ be an unbiased estimator of $\tau(\theta)$ with finite variance. Then $\varphi(T)$ has the smallest variance among all unbiased estimators of $\tau(\theta)$, for every $\theta$, and it is the only unbiased estimator with that property (up to events of probability zero).

Proof. Let $W$ be any unbiased estimator of $\tau(\theta)$ with finite variance, and let $\psi(T) = E[W \mid T]$. By the Rao–Blackwell theorem $\psi(T)$ is unbiased and $\operatorname{Var}_\theta(\psi(T)) \le \operatorname{Var}_\theta(W)$. The function $g = \varphi - \psi$ satisfies $E_\theta[g(T)] = \tau(\theta) - \tau(\theta) = 0$ for all $\theta$. By completeness $g$ is zero at every value of $T$, so $\psi(T) = \varphi(T)$. Hence

$$\operatorname{Var}_\theta(\varphi(T)) = \operatorname{Var}_\theta(\psi(T)) \le \operatorname{Var}_\theta(W).$$

For uniqueness, suppose $W$ attains the same variance as $\varphi(T)$. Then $\operatorname{Var}_\theta(W) = \operatorname{Var}_\theta(\psi(T))$, and by the equality case of the Rao–Blackwell theorem $W$ is a function of $T$, so $W = E[W \mid T] = \psi(T) = \varphi(T)$. $\square$

Such an estimator is called the uniformly minimum variance unbiased estimator. The theorem and the notion of completeness are in Lehmann and Scheffé (1950). Their introduction says "we introduce the notion of completeness of a family of measures" and announces the result (p. 234 of the reprint). Section 3 defines completeness by the condition used above and bounded completeness by the same condition imposed on bounded functions only (pp. 239–240), and proves completeness for families that include the Poisson and binomial (Examples 3.8 and 3.9, p. 243). Section 5 calls "an unbiased estimate of $g(\theta)$ which is of uniformly minimum variance" a minimum variance estimate and states, "by applying the notion of completeness to a result of Rao and Blackwell", the theorem as Theorem 5.1: if a sufficient statistic $T$ has a complete family of distributions, then every estimable function has a minimum variance estimate, and an estimate with finite variance is the minimum variance estimate of its own expected value if and only if it is a function of $T$ (p. 249). Their paper is written for general families of measures and is the source for completeness beyond the two families proved here. The consequences for this unit's examples:

  • The sample proportion $T/n$ is unbiased for $p$ and a function of the complete sufficient count. No unbiased estimator of a proportion has smaller variance.
  • $T(n - T)/[n(n - 1)]$ is the best unbiased estimator of $p(1 - p)$, and the maximum likelihood estimate $\hat{p}(1 - \hat{p})$, a function of $T$ with a different mean, is not unbiased.
  • $(1 - 1/n)^T$ is the best unbiased estimator of the Poisson probability of zero. Its variance, 0.000902 for the horse kicks, cannot be beaten by any unbiased estimator. Fisher Information and the Cramér–Rao Bound derives a general lower bound for unbiased estimators, which here is $\lambda e^{-2\lambda}/n = 0.000900$. The best unbiased estimator does not reach it: the bound is not always attainable.

Example (when no unbiased estimator exists). There is no unbiased estimator of the odds $p/(1 - p)$ from a binomial count. For any function $g$, $E_p[g(T)] = \sum_t g(t)\binom{n}{t}p^t(1 - p)^{n - t}$ is a polynomial in $p$, and a polynomial is bounded on the interval $0 \le p \le 1$. The odds grow without bound as $p \to 1$. So no $g$ has $E_p[g(T)] = p/(1 - p)$ for all $p$, and since any estimator can be replaced by a function of $T$ with the same mean, no estimator does. The maximum likelihood estimate of the odds, $0.25/0.75 = 0.3333$ for the 40 adults, exists all the same. Unbiasedness is a restrictive requirement, and the best unbiased estimator is best only within the class the requirement leaves.

What sufficiency does not say

Sufficiency is relative to a model. The count of the 40 adults is sufficient if the draws are independent with a common $p$. The arrangement that sufficiency discards is noise for estimating $p$, and it is the only evidence there is about whether those assumptions hold. A sequence with all ten ones at the start would have the same count and a very different message about the sampling. The course's check of the Poisson model on the horse kicks worked exactly this way: the total was set aside as the sufficient statistic, and the pattern of the counts given the total was compared with what the model predicts. A data reduction justified by sufficiency should be made after the model has been checked, or the raw data kept.

Where the idea came from

Stigler (2007) recounts that Fisher discovered sufficiency in 1918, in the context of estimating the standard deviation of a normal sample, and published it in 1920. The concept is one of those Fisher built around maximum likelihood in the paper of 1922 (Aldrich, 1997), whose list of definitions gives it in words: "A statistic satisfies the criterion of sufficiency when no other statistic which can be calculated from the same sample provides any additional information as to the value of the parameter to be estimated" (Fisher, 1922, p. 310). Stigler also records what he calls a productive mistake. The abstract of that paper, printed in Nature in November 1921, stated: "Statistics obtained by the method of maximum likelihood are always sufficient statistics." They are not. In the Cauchy example above the estimate is a single number, and no reduction below the sorted sample is sufficient. Stigler conjectures that this very problem, the location of a Cauchy distribution, is where Fisher learned that a sufficient statistic smaller than the sample need not exist.

What is proved, what is cited, what the experiment shows

Proved on this page, for discrete data:

  • sufficiency of the count in Bernoulli trials and of the total of Poisson counts, from the definition;
  • the factorization theorem;
  • proportional likelihoods for samples with equal sufficient statistics;
  • the likelihood-ratio criterion for minimal sufficiency;
  • the Rao–Blackwell theorem with its case of equality;
  • completeness of the binomial count and the Poisson total;
  • the Lehmann–Scheffé theorem;
  • the non-existence of an unbiased estimator of the odds.

The laws of total expectation and total variance are taken from the pages named at the top.

Cited: the factorization theorem and the minimal sufficiency criterion for continuous data (Halmos and Savage, 1949, for the first; Lehmann and Scheffé, 1950, Section 6, for the second, and Taraldsen, 2026, for an example of the caution it needs); completeness beyond the two families treated (Lehmann and Scheffé, 1950); and the history. The normal, uniform and Cauchy examples use the density forms of the two theorems. Problem 2 conditions on a continuous maximum. It is covered twice: by a limit over thin bands with a direct verification of the conclusion, on this page, and by Setting M of Conditional Expectation. The Lehmann–Scheffé proof is written for a discrete $T$, where "zero at every value" has its plain meaning.

The seeded simulation shows that the exact variances of three estimators of $e^{-\lambda}$ are reproduced by sampling: 0.001239, 0.000902 and 0.000901 against the exact 0.001241, 0.000902 and 0.000900.

Worked problems

These problems are posed on the data of this unit and on small exact cases. None comes from a course assignment or a textbook. The script named under "In Python" checks every number.

Problem 1 (a week given the six-year total). Iowa recorded 38,395 deaths from cancer in the 313 weeks of 2014 to 2019, and 125 in the first of those weeks. Under the Poisson model with a constant mean, find the conditional distribution of the first week's count given the total, with its mean and standard deviation. What does it say about the first week, and about the model?

Solution. The total is sufficient, and given the total $t = 38{,}395$ the counts are multinomial with $n = 313$ equally likely cells. Each death falls in the first week with probability $1/313$, independently, so the first week's count is binomial with $t$ trials and success probability $1/313$. Its mean is $38{,}395/313 = 122.67$, and its variance is $t\cdot\frac{1}{313}\cdot\frac{312}{313} = 122.28$, a standard deviation of 11.06. The observed 125 is $(125 - 122.67)/11.06 = 0.21$ standard deviations above the conditional mean, an unremarkable week.

None of these numbers involves the unknown $\lambda$. That is what makes the conditional distribution usable for checking the model: it says what the weekly counts should look like, given their total, if the mean is constant. The index of dispersion of Overdispersion in Count Data, 298.8 on 312 degrees of freedom with a p-value of 0.69, sums the squared deviations of all 313 weeks from that common mean, divided by the mean, and finds them as large as the model predicts.

Problem 2 (improving twice the mean). Three observations, 1.2, 2.4 and 1.3, are modeled as uniform on $[0, \theta]$. The estimator $2\bar{X}$ is unbiased for $\theta$. Improve it by the Rao–Blackwell theorem and compare the variances.

Solution. Each observation has mean $\theta/2$ and variance $\theta^2/12$, so $2\bar{X}$ has mean $\theta$ and variance $4\cdot\frac{\theta^2}{12n} = \frac{\theta^2}{3n}$, which is $\theta^2/9 = 0.1111\,\theta^2$ for $n = 3$. On the data it is $2 \times 1.6333 = 3.27$.

The largest observation $X_{(n)}$ is sufficient, by the factorization above. The theorem asks for $E[X_1 \mid X_{(n)} = m]$. The maximum is a continuous variable, so the event $\{X_{(n)} = m\}$ has probability zero, and the pair $(X_1, X_{(n)})$ has no joint density, because $X_1$ equals the maximum with positive probability. The ratio definition of this article does not apply directly. We therefore condition on a thin band, $m < X_{(n)} \le m + h$ with $0 < m < m + h \le \theta$, which has positive probability, let $h \to 0$ to find the candidate estimator, and then verify the candidate's mean and variance directly, so that nothing rests on a general theory of conditioning for such pairs.

The band. All $n$ observations are at most $t$ with probability $(t/\theta)^n$, so

$$P(m < X_{(n)} \le m + h) = \frac{(m + h)^n - m^n}{\theta^n}.$$

The part below $m$. Fix $x$ with $0 \le x \le m$. The event $\{X_1 \le x,\ X_{(n)} \le t\}$, for $t \ge x$, requires $X_1 \le x$ and the other $n - 1$ observations at most $t$, so it has probability $\frac{x}{\theta}\big(\frac{t}{\theta}\big)^{n - 1}$. Subtracting its values at $t = m + h$ and $t = m$ and dividing by the probability of the band,

$$P(X_1 \le x \mid m < X_{(n)} \le m + h) = \frac{x\,\big[(m + h)^{n - 1} - m^{n - 1}\big]}{(m + h)^n - m^n} \longrightarrow \frac{x\,(n - 1)m^{n - 2}}{n\,m^{n - 1}} = \frac{(n - 1)\,x}{n\,m} \quad (h \to 0),$$

because each bracket divided by $h$ tends to a derivative.

The part at the top. The event that $X_1$ itself lies in the band, with all others at most $m + h$, has probability $\frac{h}{\theta}\big(\frac{m + h}{\theta}\big)^{n - 1}$, and

$$P(m < X_1 \le m + h \mid m < X_{(n)} \le m + h) = \frac{h\,(m + h)^{n - 1}}{(m + h)^n - m^n} \longrightarrow \frac{m^{n - 1}}{n\,m^{n - 1}} = \frac{1}{n}.$$

The two limits add to $\frac{n - 1}{n} + \frac{1}{n} = 1$ at $x = m$, so they describe a complete distribution: given a maximum at $m$, the observation $X_1$ is itself the maximum with probability $1/n$, and with probability $(n - 1)/n$ it is uniform on $[0, m]$. Neither limit involves $\theta$, as sufficiency requires. The mean of that distribution is

$$\frac{1}{n}\,m + \frac{n - 1}{n}\cdot\frac{m}{2} = \frac{(n + 1)\,m}{2n}.$$

For $n = 3$ and $m = 2.4$, the script evaluates the two band probabilities at $h = 10^{-6}$ and finds them within $10^{-6}$ of the limits $\frac{2x}{3m}$ and $\frac{1}{3}$.

By symmetry every $X_i$ has the same conditional distribution, so the candidate for $E[2\bar{X} \mid X_{(n)} = m]$ is $2\cdot\frac{(n + 1)m}{2n} = \frac{n + 1}{n}m$, and the candidate estimator is $\frac{n + 1}{n}X_{(n)}$: $\frac{4}{3}\times 2.4 = 3.2$ on the data.

Direct verification. Maximum Likelihood Estimation computed, from the density of the maximum alone, $E[X_{(n)}] = \frac{n}{n + 1}\theta$ and $\operatorname{Var}(X_{(n)}) = \frac{n\theta^2}{(n + 1)^2(n + 2)}$. So $\frac{n + 1}{n}X_{(n)}$ has mean $\theta$, the mean of $2\bar{X}$, and variance

$$\frac{(n + 1)^2}{n^2}\cdot\frac{n\,\theta^2}{(n + 1)^2(n + 2)} = \frac{\theta^2}{n(n + 2)},$$

which is $\theta^2/15 = 0.0667\,\theta^2$ for $n = 3$. Since $n(n + 2) \ge 3n$ for $n \ge 1$, this never exceeds the variance $\theta^2/(3n)$ of $2\bar{X}$, and is smaller for $n \ge 2$. The two conclusions of the Rao–Blackwell theorem hold, by calculation. Conditional Expectation treats this pair in its Setting M and finds the same conditional mean, so the candidate is $E[2\bar{X} \mid X_{(n)}]$ and the improvement is the one the theorem promises. For $n = 3$ the variance of $2\bar{X}$ is 1.67 times as large, and the ratio $(n + 2)/3$ grows with $n$: the mean wastes most of the information in a uniform sample, which lies at its edge.

Problem 3 (sampling until the tenth case). Suppose young adults had been drawn until the tenth with hypertension appeared, and that this took $N = 40$ draws. Find a sufficient statistic, the maximum likelihood estimate of $p$, and an unbiased estimator of $p$ obtained by the Rao–Blackwell theorem.

Solution. With $r = 10$ fixed in advance, the data are a sequence ending in a success with $r$ successes in all, of random length $N$. A sequence of length $m$ has probability $p^r(1 - p)^{m - r}$, which depends on the sequence only through its length. By the factorization theorem $N$ is sufficient. The likelihood is proportional to $p^{10}(1 - p)^{30}$, the same as for a fixed sample of 40 with 10 successes, so the maximum likelihood estimate is again $r/N = 10/40 = 0.25$.

For an unbiased estimator, start with $W = X_1$, the outcome of the first draw, whose mean is $p$. Given $N = m$, the sequence is equally likely to be any arrangement with a success in the last position and $r - 1$ successes among the first $m - 1$; there are $\binom{m - 1}{r - 1}$ of them. Those that begin with a success have $r - 2$ successes among positions 2 to $m - 1$, and there are $\binom{m - 2}{r - 2}$. So

$$E[X_1 \mid N = m] = \frac{\binom{m - 2}{r - 2}}{\binom{m - 1}{r - 1}} = \frac{r - 1}{m - 1}.$$

The improved unbiased estimator is $(r - 1)/(N - 1) = 9/39 = 0.2308$. Under this sampling plan the maximum likelihood estimate $r/N$ is not unbiased: at the population value $p_0 = 0.2294$ its mean is 0.2480, above $p_0$, which the script computes by summing over the distribution of $N$ (the same sum confirms that $(r - 1)/(N - 1)$ has mean exactly $p_0$).

The two sampling plans give proportional likelihoods and the same maximum likelihood estimate, as The Likelihood Function showed. Their unbiased estimators differ, $x/n$ for the fixed sample and $(r - 1)/(N - 1)$ for sampling to the tenth case, because unbiasedness is a property of the sampling distribution, and the plans have different ones.

Problem 4 (an unbiased estimate of the squared rate). For the 200 horse-kick counts, find the best unbiased estimator of $\lambda^2$ and compare it with the maximum likelihood estimate.

Solution. A crude unbiased estimator is $W = X_1X_2$: the two counts are independent with mean $\lambda$, so $E_\lambda[W] = \lambda^2$. Condition on the total $T = t$. Given the total, each of the $t$ deaths falls independently in one of the $n$ corps-years. Write $X_1$ as the number of deaths that fall in the first and $X_2$ as the number in the second. Their product counts the ordered pairs of distinct deaths $(d, e)$ with $d$ in the first corps-year and $e$ in the second; a single death cannot be in both. There are $t(t - 1)$ ordered pairs of distinct deaths, and each has probability $\frac{1}{n}\cdot\frac{1}{n}$ of falling that way, so by linearity

$$E[X_1X_2 \mid T = t] = \frac{t(t - 1)}{n^2}.$$

The improved estimator is $T(T - 1)/n^2$. It is unbiased by the Rao–Blackwell theorem, and since the Poisson total is complete, the Lehmann–Scheffé theorem makes it the unique best unbiased estimator of $\lambda^2$. For the data, $122 \times 121/200^2 = 0.3691$. The crude estimator, by contrast, is zero whenever either of its two corps-years had no death.

The maximum likelihood estimate is $\hat\lambda^2 = 0.61^2 = 0.3721$, by invariance. The two differ by $T/n^2 = 0.00305$: since $E_\lambda[(T/n)^2] = \operatorname{Var}(T/n) + \lambda^2 = \lambda/n + \lambda^2$, the maximum likelihood estimate is biased upward by $\lambda/n$, and subtracting the estimate $T/n^2$ of that bias gives the unbiased estimator exactly.

Summary

Result Statement
Sufficient statistic the distribution of the sample given $T$ does not depend on $\theta$
Factorization theorem $T$ is sufficient if and only if $f(\mathbf{x}; \theta) = g(T(\mathbf{x}); \theta)\,h(\mathbf{x})$
Bernoulli, Poisson the count and the total; given $T$, a random arrangement or a multinomial with equal cells
Normal $(\bar{x}, \sum(x_i - \bar{x})^2)$; uniform on $[0, \theta]$: the largest observation
Exponential family $\sum T(x_i)$ is sufficient for every $n$
Likelihood samples with equal $T$ have proportional likelihoods; the MLE is a function of $T$
Minimal sufficiency $T(\mathbf{x}) = T(\mathbf{y})$ exactly when $f(\mathbf{x}; \theta)/f(\mathbf{y}; \theta)$ is free of $\theta$
Rao–Blackwell $E[W \mid T]$ has the mean of $W$ and no larger variance
Completeness $E_\theta[g(T)] = 0$ for all $\theta$ only for $g = 0$; holds for the binomial count and Poisson total
Lehmann–Scheffé an unbiased function of a complete sufficient statistic is the unique best unbiased estimator

What we have learned

A sufficient statistic holds everything the sample says about the parameter, under the model: given the count of 10, the order of the 40 adults is one of 847,660,528 equally likely arrangements whatever $p$ is. The factorization theorem reads sufficient statistics off the likelihood, and the likelihood-ratio criterion finds the smallest one. Conditioning any estimator on a sufficient statistic keeps its mean and lowers its variance: for the probability of a year without a horse-kick death, from 0.2481 for one corps-year to 0.000902. When the statistic is complete, as the binomial count and the Poisson total are, the result is the best unbiased estimator there is.

For the horse kicks that best variance, 0.000902, sits just above a number, 0.000900, that this article quoted without deriving. Fisher Information and the Cramér–Rao Bound derives it: a lower bound on the variance of every unbiased estimator, computed from the curvature of the log-likelihood.

In Python

import numpy as np

rng = np.random.default_rng(0)
x = rng.poisson(0.61, size=(20_000, 200))      # 20,000 tables of 200 corps-years
t = x.sum(axis=1)
share = (x == 0).mean(axis=1)                  # share of zeros in each table
rb = (1 - 1 / 200) ** t                        # Rao-Blackwell estimator
print(share.var(), rb.var())                   # about 0.00124 and 0.00090

Every number in this article is computed by tools/experiments/statistics/sufficient-statistics.py in the site's repository, with the data loaded by ledata.py beside it.

References