Minimal Sufficient Statistics in Standard Models

Why Minimal Sufficiency Matters

A sufficient statistic retains all the information in a sample about an unknown parameter. But there may be several sufficient statistics, and some may retain more sample detail than is needed. Minimal sufficiency identifies the coarsest sufficient summary: it groups together sample outcomes that carry the same likelihood information about the parameter.

The factorization theorem is useful for finding a sufficient statistic. The likelihood-ratio criterion goes further: it tests whether the statistic distinguishes precisely the sample outcomes that must be distinguished. This tutorial applies that criterion to independent Bernoulli, Poisson, and normal samples. All three models below have a parameter-independent sample space, an important condition for the criterion.

Definition: A statistic \(T(X)\) is minimal sufficient for a parameter \(\theta\) if it is sufficient and, up to events of probability zero, it is a function of every other sufficient statistic. Equivalently, it is a sufficient summary that retains no distinctions among sample outcomes that are unnecessary for inference about \(\theta\).

The Likelihood-Ratio Criterion

Suppose a sample has a probability mass function or density \(L(\theta;x)\), with the same support for every \(\theta\). For two possible sample points \(x\) and \(y\), compare their likelihoods as the parameter varies.

Theorem: For a dominated model with parameter-independent support, a statistic \(T\) is minimal sufficient if it is sufficient and, for any sample points \(x\) and \(y\), \[ \frac{L(\theta;x)}{L(\theta;y)} \text{ is independent of }\theta \quad\Longleftrightarrow\quad T(x)=T(y), \] where the equivalence is understood up to null sets. The likelihood ratio is considered where the likelihoods are positive.

The criterion concerns whether the ratio changes with the unknown parameter, not whether the numerical ratio equals one. A ratio can be a constant other than one and still be parameter-independent. Conversely, if the ratio contains \(\theta\), the two sample points cannot be combined into the same value of a minimal sufficient statistic.

Here is the reason the criterion identifies the minimal summary. In a discrete model, if a sufficient statistic assigns the same value to \(x\) and \(y\), then the conditional probability of observing \(x\) rather than \(y\), given that statistic, cannot depend on \(\theta\). The ratio of those conditional probabilities is also the likelihood ratio \(L(\theta;x)/L(\theta;y)\), so that ratio must be parameter-independent. Thus a sufficient statistic cannot group points whose likelihood ratio varies with \(\theta\). Conversely, if the ratio is parameter-independent exactly for points with the same \(T\), then \(T\)'s groups are precisely the groups that retain no parameter-dependent likelihood distinction. If \(T\) is also sufficient, no other sufficient statistic can be coarser. For continuous models, the same argument is expressed using conditional densities and holds almost everywhere.

Write the likelihood.
Keep all sample-dependent terms long enough to compare two possible samples.

Form a ratio.
Cancel terms common to the two samples, then check whether the unknown parameter remains.

Identify the equivalence classes.
Samples belong to the same class exactly when their likelihood ratio is parameter-independent.

Verify sufficiency.
Use the factorization theorem or the parameter-free conditional distribution to confirm that the proposed statistic is sufficient.

Bernoulli Samples

Let \(X_1,\ldots,X_n\) be independent Bernoulli random variables with \(P(X_i=1)=p\), where \(0<p<1\). The sample space is \(\{0,1\}^n\), the same for every allowed \(p\). Write \(S=\sum_{i=1}^n X_i\), the number of observed successes. The likelihood is

L(p;x)=p^{\sum x_i}(1-p)^{n-\sum x_i}.

For two sample points \(x\) and \(y\), let their success counts be \(s_x\) and \(s_y\). Their likelihood ratio is

\frac{L(p;x)}{L(p;y)} = \left(\frac{p}{1-p}\right)^{s_x-s_y}.

If \(s_x=s_y\), this ratio is 1 for every allowed \(p\). If \(s_x\ne s_y\), the ratio varies with \(p\). Therefore the equivalence classes are determined exactly by the success count, so \(S=\sum X_i\) is minimal sufficient.

The factorization theorem agrees: the likelihood depends on the sample only through \(S\), with the remaining parameter-free factor equal to 1. The sample order carries no further information about \(p\).

Worked Example: Comparing Bernoulli Samples

Suppose six independent trials have success probability \(p\), with \(0<p<1\). Compare \(x=(1,0,1,1,0,0)\) with \(y=(0,1,0,1,1,0)\), and compare \(x\) with \(z=(1,0,0,1,0,0)\). Identify the minimal sufficient statistic and calculate the two likelihood ratios.

Method. For a Bernoulli sample, use the success count \(S=\sum X_i\), then form each likelihood ratio.

For \(x\), the count is \(1+0+1+1+0+0=3\). For \(y\), it is \(0+1+0+1+1+0=3\). Thus

\frac{L(p;x)}{L(p;y)} = \left(\frac{p}{1-p}\right)^{3-3}=1.

For \(z\), the count is \(1+0+0+1+0+0=2\), so

\frac{L(p;x)}{L(p;z)} = \left(\frac{p}{1-p}\right)^{3-2} = \frac{p}{1-p}.

The first ratio is parameter-independent, while the second depends on \(p\). Hence \(x\) and \(y\) share the same minimal sufficient statistic value, \(S=3\), but \(z\) has \(S=2\). The factorization above confirms that \(S\) is sufficient. Common mistake avoided: declaring two samples equivalent merely because they have the same length; their success counts, not their lengths, determine the likelihood ratio.

Poisson Samples

Let \(X_1,\ldots,X_n\) be independent Poisson random variables with common rate \(\lambda>0\). Their sample space consists of nonnegative integer vectors and does not depend on \(\lambda\). With \(S=\sum X_i\), the likelihood is

L(\lambda;x) = \frac{e^{-n\lambda}\lambda^{\sum x_i}}{\prod_{i=1}^n x_i!}.

For two sample points, the common \(e^{-n\lambda}\) terms cancel:

\frac{L(\lambda;x)}{L(\lambda;y)} = \lambda^{s_x-s_y} \frac{\prod_{i=1}^n y_i!}{\prod_{i=1}^n x_i!}.

The factorial ratio is constant in \(\lambda\). The remaining factor is parameter-independent exactly when \(s_x=s_y\). Consequently, \(S=\sum X_i\) is minimal sufficient. The factorization theorem gives the same result: all the \(\lambda\)-dependent terms involve the data only through \(S\).

Worked Example: Comparing Poisson Samples

Four independent counts have a common Poisson rate \(\lambda>0\). Let \(x=(0,2,1,3)\), \(y=(1,1,2,2)\), and \(z=(0,1,1,2)\). Find the minimal sufficient statistic for these observations and compare \(x\)'s likelihood with those of \(y\) and \(z\).

Method. Use the total count \(S\) and the Poisson likelihood-ratio formula.

The totals are \(s_x=0+2+1+3=6\), \(s_y=1+1+2+2=6\), and \(s_z=0+1+1+2=4\). For \(x\), the product of factorials is \(0!2!1!3!=1\cdot2\cdot1\cdot6=12\). For \(y\), it is \(1!1!2!2!=1\cdot1\cdot2\cdot2=4\). Therefore

\frac{L(\lambda;x)}{L(\lambda;y)} = \lambda^{6-6}\frac{4}{12} = \frac{1}{3}.

This constant is not 1, but it is independent of \(\lambda\). For \(z\), the factorial product is \(0!1!1!2!=2\), giving

\frac{L(\lambda;x)}{L(\lambda;z)} = \lambda^{6-4}\frac{2}{12} = \frac{\lambda^2}{6}.

The second ratio depends on \(\lambda\), so \(x\) and \(z\) have different minimal sufficient statistic values. The statistic is \(S=6\) for \(x\) and \(y\), and \(S=4\) for \(z\). Common mistake avoided: requiring a parameter-independent likelihood ratio to equal one; a fixed non-unit ratio, such as \(1/3\), also meets the criterion.

Normal Samples

The minimal sufficient statistic for a normal sample depends on which parameters are known. First suppose \(X_1,\ldots,X_n\) are independent with \(X_i\sim N(\mu,\sigma^2)\), where \(\sigma^2>0\) is known and \(\mu\) is unknown. Define \(S=\sum X_i\) and \(Q=\sum X_i^2\). The likelihood ratio for sample points \(x\) and \(y\) has logarithm

\log\frac{L(\mu;x)}{L(\mu;y)} = \frac{\mu(s_x-s_y)}{\sigma^2} - \frac{q_x-q_y}{2\sigma^2}.

Since \(\sigma^2\) is fixed, this expression is independent of \(\mu\) exactly when \(s_x=s_y\). Thus \(S=\sum X_i\), or equivalently the sample mean \(\bar X=S/n\), is minimal sufficient for \(\mu\). The factorization theorem confirms this because the likelihood can be written as a parameter-only factor times a factor involving the data through \(S\).

Now suppose both \(\mu\) and \(\sigma^2\) are unknown, with \(\mu\in\mathbb R\) and \(\sigma^2>0\). The likelihood is

L(\mu,\sigma^2;x) = (2\pi\sigma^2)^{-n/2} \exp\left\{-\frac{q_x-2\mu s_x+n\mu^2}{2\sigma^2}\right\}.

In the ratio between two samples of the same size, the normalizing terms cancel:

\log\frac{L(\mu,\sigma^2;x)}{L(\mu,\sigma^2;y)} = -\frac{(q_x-q_y)-2\mu(s_x-s_y)}{2\sigma^2}.

For this to be independent of \(\mu\), we must have \(s_x=s_y\). Once those sums agree, the ratio is independent of \(\sigma^2\) exactly when \(q_x=q_y\). Therefore the pair \((S,Q)=(\sum X_i,\sum X_i^2)\) is minimal sufficient when both parameters are unknown. An equivalent pair is \((\bar X,\sum (X_i-\bar X)^2)\), because \(S=n\bar X\) and \(Q=\sum(X_i-\bar X)^2+n\bar X^2\). The factorization theorem yields the same pair from the terms in the likelihood involving the data.

Worked Example: Unknown Mean and Variance

Three independent observations come from a normal distribution with both \(\mu\) and \(\sigma^2\) unknown. Compare \(x=(1,2,5)\) and \(y=(0,3,5)\). Find their minimal sufficient statistic values and calculate the likelihood ratio.

Method. For the two-parameter normal model, calculate the sample sum and sum of squares, then substitute their differences into the likelihood ratio.

For \(x\), \(s_x=1+2+5=8\) and \(q_x=1^2+2^2+5^2=1+4+25=30\). For \(y\), \(s_y=0+3+5=8\) and \(q_y=0^2+3^2+5^2=0+9+25=34\). Their statistic values are therefore \((8,30)\) and \((8,34)\). The log likelihood ratio is

-\frac{(30-34)-2\mu(8-8)}{2\sigma^2} = -\frac{-4}{2\sigma^2} = \frac{2}{\sigma^2}.

Exponentiating gives \(L(\mu,\sigma^2;x)/L(\mu,\sigma^2;y)=e^{2/\sigma^2}\), which depends on the unknown variance. Equal sample sums alone do not make the samples equivalent when both normal parameters are unknown. The minimal sufficient statistic is \((\sum X_i,\sum X_i^2)\). If the variances were known, the ratio would instead be constant in the unknown mean because the sums agree. Common mistake avoided: using the known-variance statistic when variance is also unknown.

Common Traps and a Practical Check

Minimal sufficiency asks how much of the sample must be retained to preserve every parameter-dependent likelihood comparison. It does not say that the statistic estimates the parameter well, nor does it establish completeness or any optimality property of an estimator.

Stopping at sufficiency. Factorization can show that a statistic is sufficient, but the likelihood-ratio criterion is what checks minimality.
Dropping a needed normal summary. With known variance, the sample sum suffices for unknown \(\mu\). With both parameters unknown, the sum of squares is also needed.
Ignoring support assumptions. The criterion as stated requires a common, parameter-independent support. If the support changes with the parameter, the likelihood ratio must be handled with care; this tutorial's three models avoid that issue.
Misreading “constant.” The ratio must be constant as a function of the parameter. It need not equal one.

Key takeaway: For independent Bernoulli and Poisson samples, the total count is minimal sufficient. For normal samples, the sample sum is minimal sufficient for an unknown mean with known variance; when both mean and variance are unknown, retain both the sum and the sum of squares.

Check Your Understanding

For each problem, identify the relevant likelihood-ratio comparison and use the result to determine the minimal sufficient statistic.

Eight independent Bernoulli trials have \(0<p<1\). A sample has four successes and another has three. What is the likelihood ratio of the first sample to the second, and what statistic is minimal sufficient? (Answer: The ratio is \(p/(1-p)\); \(T=\sum X_i\).)
Five independent Poisson observations with common \(\lambda>0\) are \((2,0,1,3,0)\). Find the minimal sufficient statistic value. (Answer: \(T=\sum X_i=6\).)
Independent observations have distribution \(N(\mu,9)\), with known variance \(9\) and unknown mean. State a minimal sufficient statistic. (Answer: \(T=\sum X_i\), equivalently \(\bar X\).)
For a normal sample with both parameters unknown, compare \(x=(1,1,4)\) and \(y=(0,2,4)\). Do they have the same minimal sufficient statistic value? (Answer: No; both sums are 6, but the sums of squares are 18 and 20, so their statistic values differ.)
For a normal sample with both \(\mu\) and \(\sigma^2\) unknown, two sample points have sums 10 and 10, and sums of squares 38 and 38. What is their likelihood ratio, and what does the criterion imply? (Answer: The ratio is 1; the points have the same \((\sum X_i,\sum X_i^2)\) value and belong to the same minimal-sufficiency class.)

What You'll Learn