Minimal Sufficiency From Likelihood Ratios

Why Likelihood Ratios Reveal Minimal Sufficiency

A sufficient statistic retains all the sample information about the model parameter, but different sufficient statistics can retain different amounts of information about the observed data. Minimal sufficiency identifies the coarsest such summary: it groups together data sets that no sufficient statistic needs to distinguish. The likelihood-ratio criterion makes those groupings concrete by asking whether two data sets have likelihoods whose ratio changes with the parameter.

The comparison is between likelihoods at two possible data sets, not between likelihoods at two parameter values. If changing the data set from \(x\) to \(y\) only multiplies the likelihood by a constant independent of the parameter, that change does not affect the relative evidence about the parameter. The Bernoulli, Poisson, and normal examples below show exactly how this idea determines a minimal sufficient statistic.

Definition: A statistic \(T(X)\) is minimal sufficient for a parameter \(\theta\) if it is sufficient and is a function of every other sufficient statistic, up to probability-zero sets. Thus, any sufficient statistic can retain more information, but cannot retain less information than \(T\).

For a model with densities or mass functions \(f_\theta(x)\), the likelihood ratio for two possible data sets \(x\) and \(y\) is \(f_\theta(x)/f_\theta(y)\). We will work with families that have a common support and strictly positive densities or mass functions on that support. These conditions ensure that the ratio is defined for every pair of sample points under comparison.

The Likelihood-Ratio Criterion

Theorem (likelihood-ratio criterion for minimal sufficiency): Suppose a family of distributions has strictly positive densities or mass functions on a common support. Let \(T\) be a sufficient statistic. If, for every pair \(x,y\) in the support,

\(T(x)=T(y)\) if and only if \(f_\theta(x)/f_\theta(y)\) is constant in \(\theta\),

then \(T\) is minimal sufficient. For ordinary Euclidean sample spaces and measurable statistics, the “function of every sufficient statistic” statement is understood up to probability-zero sets.

Here is the main argument behind the criterion. The factorization criterion says that sufficiency of a statistic \(S\) allows the likelihood to be written as \(f_\theta(x)=g_\theta(S(x))h(x)\), with the parameter dependence carried by \(S\). In particular, if \(S(x)=S(y)\), the likelihood ratio for \(x\) and \(y\) is independent of \(\theta\).

Now suppose \(T\) meets the theorem’s condition. For any sufficient statistic \(S\), whenever \(S(x)=S(y)\), the factorization argument shows that the ratio is parameter-independent. The condition on \(T\) then gives \(T(x)=T(y)\). Thus \(T\) is constant on every set of data points that \(S\) groups together. Under the stated measurability conditions, \(T\) is a function of \(S\) (up to null sets). Since \(T\) is itself sufficient, it is minimal sufficient. \(\square\)

The argument also gives a useful check: equal values of any sufficient statistic must imply a parameter-independent likelihood ratio. To establish minimal sufficiency by the criterion, however, we check both directions: equal statistic values give constant ratios, and constant ratios occur only when the statistic values are equal.

Bernoulli Samples

Worked Example: Minimal Sufficiency for Bernoulli Data

Let \(X_1,\ldots,X_n\) be independent Bernoulli random variables with success probability \(p\), where \(0<p<1\). For a sample \(x=(x_1,\ldots,x_n)\), each \(x_i\) is either 0 or 1. Write \(s(x)=\sum_{i=1}^n x_i\). The likelihood is

L_x(p)=p^{s(x)}(1-p)^{n-s(x)}.

For two samples \(x,y\), the ratio is

\frac{L_x(p)}{L_y(p)} = \left(\frac{p}{1-p}\right)^{s(x)-s(y)}.

If \(s(x)=s(y)\), the ratio is 1 and does not depend on \(p\). If the sums differ, the ratio varies with \(p\): for example, its value changes as \(p/(1-p)\) changes over positive values. Thus the ratio is parameter-independent exactly when the two samples have the same number of successes. The factorization of the likelihood shows that \(\sum_i X_i\) is sufficient, and the criterion shows that it is minimal sufficient.

For a numerical check, take \(x=(1,0,1,0)\) and \(y=(0,1,1,0)\). Both sums equal 2, so their likelihood ratio is 1. If instead \(y=(1,0,0,0)\), its sum is 1, and the ratio against \(x\) is \(p/(1-p)\), which depends on \(p\).

Poisson Samples

Worked Example: Minimal Sufficiency for Poisson Data

Let \(X_1,\ldots,X_n\) be independent Poisson random variables with mean \(\lambda\), where \(\lambda>0\). For nonnegative integer observations \(x_i\), set \(s(x)=\sum_i x_i\). The joint mass function and the ratio for samples \(x\) and \(y\) are

L_x(\lambda)= \frac{e^{-n\lambda}\lambda^{s(x)}}{\prod_{i=1}^n x_i!}, \qquad \frac{L_x(\lambda)}{L_y(\lambda)} = \frac{\prod_{i=1}^n y_i!}{\prod_{i=1}^n x_i!} \lambda^{s(x)-s(y)}.

When the sums agree, the power of \(\lambda\) is zero, so the ratio is constant in \(\lambda\). When the sums differ, it contains a nonzero power of \(\lambda\), so it varies with \(\lambda\). The likelihood factors into a function of \(\lambda\) and a data-dependent factor through the sum, so \(\sum_i X_i\) is sufficient; the ratio calculation establishes that it is minimal sufficient.

For instance, with \(n=3\), take \(x=(0,2,1)\) and \(y=(1,1,1)\). Each sum is 3, and

\frac{L_x(\lambda)}{L_y(\lambda)} = \frac{1!\,1!\,1!}{0!\,2!\,1!} =\frac{1}{2}.

By contrast, for \(x=(2,0,0)\) and \(y=(1,0,0)\), the sums are 2 and 1. The ratio is \(\lambda/2\), which depends on \(\lambda\). These data sets must therefore receive different values of a minimal sufficient statistic.

Normal Samples

Worked Example: Unknown Mean and Variance

Let \(X_1,\ldots,X_n\) be independent \(N(\mu,\sigma^2)\) observations, with both \(\mu\in\mathbb{R}\) and \(\sigma^2>0\) unknown. Define \(S(x)=\sum_i x_i\) and \(Q(x)=\sum_i x_i^2\). Expanding the squared deviations gives

L_x(\mu,\sigma^2) =(2\pi\sigma^2)^{-n/2} \exp\left\{-\frac{Q(x)-2\mu S(x)+n\mu^2}{2\sigma^2}\right\}.

For two samples, all terms that depend only on the parameters and \(n\) cancel in the ratio:

\frac{L_x(\mu,\sigma^2)}{L_y(\mu,\sigma^2)} = \exp\left\{ -\frac{Q(x)-Q(y)}{2\sigma^2} +\frac{\mu\{S(x)-S(y)\}}{\sigma^2} \right\}.

If both \(S(x)=S(y)\) and \(Q(x)=Q(y)\), the ratio is 1. Conversely, if the ratio is constant in both parameters, its dependence on \(\mu\) forces \(S(x)-S(y)=0\). With this difference zero, dependence on \(\sigma^2\) forces \(Q(x)-Q(y)=0\). Hence the ratio is parameter-independent exactly when both sums agree. The likelihood factorization and criterion together show that \((\sum_i X_i,\sum_i X_i^2)\) is minimal sufficient.

For a concrete pair, let \(x=(1,-1,0)\) and \(y=(1/\sqrt{3},1/\sqrt{3},-2/\sqrt{3})\). Each sample has sum 0 and sum of squares 2, so their likelihood ratio is 1 for every \(\mu\) and \(\sigma^2\), even though their observations are not the same.

The parameter space matters. If \(\sigma^2\) is known and only \(\mu\) varies, the ratio above is parameter-independent exactly when \(S(x)=S(y)\); the sum alone then gives the likelihood-ratio grouping. Minimal sufficiency is a property of a specified model, including which parameters are allowed to vary.

Common Mistakes and Practical Checks

A frequent error is to treat a ratio that is constant for one selected parameter value as if it were constant in the parameter. The criterion requires constancy over the entire parameter space. For example, the Bernoulli ratio \(p/(1-p)\) happens to equal 1 at \(p=1/2\), but it is not constant as \(p\) varies, so samples with different success counts cannot be grouped.

Another error is to compare likelihoods while ignoring a parameter-dependent term. In the normal model, the term involving \(\mu\{S(x)-S(y)\}/\sigma^2\) cannot be discarded when both mean and variance are unknown. Omitting it could incorrectly suggest that the sum of squares alone determines the grouping.

Write the joint likelihood.
Keep every factor that depends on the parameter, and record the common support and parameter space.

Form the ratio for two samples.
Cancel terms shared by both samples, but do not discard any remaining parameter-dependent factor.

Check both directions.
Show that equal statistic values give a parameter-independent ratio, and that a parameter-independent ratio forces the proposed statistic values to agree.

These calculations explain why “sufficient” and “minimal sufficient” are different claims. A statistic is sufficient if it preserves parameter information; it is minimal sufficient only if it groups exactly the data sets that have parameter-independent likelihood ratios. Changing the parameter space can change those groupings, as the normal model with known variance illustrates.

Check Your Understanding

Use the likelihood-ratio criterion to answer the following questions.

For Bernoulli samples with \(0<p<1\), what feature of two samples determines whether their likelihood ratio is independent of \(p\)?
For Poisson samples, explain why unequal sample sums make the likelihood ratio depend on \(\lambda\), even though the factorial terms may differ.
In the normal model with both parameters unknown, which two sample summaries must agree for the likelihood ratio to be parameter-independent?
For a normal model with known variance and unknown mean, why does the minimal sufficient statistic differ from the one used when both parameters are unknown?
Why is checking that a ratio is constant at one parameter value not enough to establish minimal sufficiency?