Minimal Sufficiency Through Likelihood Ratios

Why Likelihood Ratios Reveal the Information in a Sample

A sufficient statistic retains all the sample information relevant to the unknown parameter. Minimal sufficiency asks a sharper question: can any of that retained information be removed while remaining sufficient? For a dominated model, compare the likelihoods of two possible sample points. If their ratio does not depend on the parameter, the data distinguish those points in a way that carries no information about the parameter. This comparison leads to a practical criterion for finding minimal sufficient statistics.

The criterion is especially useful for independent samples from exponential-family models. Their likelihoods often depend on the sample through a few sums, and the ratio test shows whether those sums are not merely sufficient but minimal.

Likelihood-Ratio Equivalence

Let $X$ take values in a sample space $\mathcal X$, and suppose its distribution has density or mass function $f_\theta(x)$ with respect to a common dominating measure $\mu$, for $\theta\in\Theta$. For the pointwise version of the criterion below, assume there is a common support $S$ on which $f_\theta(x)>0$ for every $\theta$, with density zero outside $S$. In continuous models, statements about sample points and fibers are understood up to $\mu$-null sets; use density versions for which the stated factorizations hold on a common full-measure subset. Also assume the equivalence classes below form a measurable quotient, so they can be represented by a statistic.

Definition: Two sample points $x,y\in S$ are likelihood-ratio equivalent if $$ \frac{f_\theta(x)}{f_\theta(y)} $$ is constant as a function of $\theta\in\Theta$. Equivalently, the likelihoods at $x$ and $y$ differ only by a factor that does not depend on the parameter. A statistic $T$ represents these equivalence classes if $T(x)=T(y)$ exactly when $x$ and $y$ are likelihood-ratio equivalent.

A statistic $T$ is sufficient when the likelihood factors as $f_\theta(x)=g_\theta(T(x))h(x)$, where $g_\theta$ and $h$ are nonnegative measurable functions and $h$ does not depend on $\theta$. This is the factorization criterion. A sufficient statistic $T$ is minimal sufficient if it is a function, up to null sets, of every other sufficient statistic. Thus it retains the coarsest data summary that is still sufficient.

Theorem: Under the common-support and measurability conditions above, a statistic representing likelihood-ratio equivalence classes is minimal sufficient. In particular, its equal-value fibers satisfy the likelihood-ratio criterion: $T(x)=T(y)$ if and only if $f_\theta(x)/f_\theta(y)$ is independent of $\theta$.

Why the Criterion Works

First suppose $T$ is sufficient, so $f_\theta(x)=g_\theta(T(x))h(x)$. If $T(x)=T(y)$, then $$ \frac{f_\theta(x)}{f_\theta(y)} = \frac{g_\theta(T(x))h(x)}{g_\theta(T(y))h(y)} = \frac{h(x)}{h(y)}, $$ which does not depend on $\theta$, wherever the factors are positive. Hence a sufficient statistic cannot separate sample points whose likelihood ratio is parameter-independent.

For the converse, fix a reference value $\theta_0\in\Theta$. On the common support, define $h(x)=f_{\theta_0}(x)$. If $T(x)=T(y)$, likelihood-ratio equivalence gives $$ \frac{f_\theta(x)}{f_{\theta_0}(x)} = \frac{f_\theta(y)}{f_{\theta_0}(y)} $$ for every $\theta$. The ratio $f_\theta(x)/f_{\theta_0}(x)$ is therefore constant on each equivalence class; by the measurability assumption it defines a function $g_\theta(T(x))$. Consequently, $$ f_\theta(x)=g_\theta(T(x))h(x), $$ so $T$ is sufficient by the factorization criterion.

It remains to show minimality. Let $U$ be any sufficient statistic, with $f_\theta(x)=a_\theta(U(x))b(x)$. If $U(x)=U(y)$, then, wherever the terms are positive, $$ \frac{f_\theta(x)}{f_\theta(y)} = \frac{a_\theta(U(x))b(x)}{a_\theta(U(y))b(y)} = \frac{b(x)}{b(y)}, $$ which is independent of $\theta$. Thus $x$ and $y$ belong to the same likelihood-ratio class, so $T(x)=T(y)$. The equivalence classes of $U$ can only be finer than those of $T$; under the stated quotient condition, $T$ is a function of $U$, up to null sets. Since $T$ is itself sufficient, it is minimal sufficient. $\square$

Worked Examples in Independent Samples

Worked Example: Bernoulli Trials

Let $X_1,\ldots,X_n$ be independent Bernoulli random variables with success probability $p\in(0,1)$. For a sample point $x=(x_1,\ldots,x_n)\in\{0,1\}^n$, the likelihood is $$ f_p(x)=p^{\sum_i x_i}(1-p)^{n-\sum_i x_i}. $$ Take two sample points $x,y$, and write $s_x=\sum_i x_i$ and $s_y=\sum_i y_i$. Their likelihood ratio is $$ \frac{f_p(x)}{f_p(y)} = \left(\frac{p}{1-p}\right)^{s_x-s_y}. $$ If $s_x=s_y$, the ratio is $1$, independent of $p$. If $s_x\ne s_y$, the ratio changes with $p$, since $p/(1-p)$ ranges over all positive values as $p$ ranges over $(0,1)$. Therefore two samples are equivalent exactly when they have the same number of successes. The minimal sufficient statistic is $$ T(X)=\sum_{i=1}^n X_i. $$

Worked Example: Normal Samples With Unknown Mean and Variance

Let $X_1,\ldots,X_n$ be independent $N(\mu,\sigma^2)$ observations, with $\mu\in\mathbb R$ and $\sigma^2>0$ both unknown. The joint density is $$ f_{\mu,\sigma^2}(x) = (2\pi\sigma^2)^{-n/2} \exp\left\{-\frac{1}{2\sigma^2}\sum_{i=1}^n(x_i-\mu)^2\right\}. $$ Let $S_1(x)=\sum_i x_i$ and $S_2(x)=\sum_i x_i^2$, and define $\Delta_1=S_1(x)-S_1(y)$, $\Delta_2=S_2(x)-S_2(y)$. Expanding the squares gives $$ \log\frac{f_{\mu,\sigma^2}(x)}{f_{\mu,\sigma^2}(y)} = \frac{\mu\Delta_1}{\sigma^2}-\frac{\Delta_2}{2\sigma^2}. $$ If both differences are zero, the log ratio is zero for every parameter value. Conversely, if the ratio is independent of the parameters, its logarithm must be independent of $\mu$ for each fixed $\sigma^2$. The coefficient of $\mu$ must therefore vanish, giving $\Delta_1=0$. The remaining expression is $-\Delta_2/(2\sigma^2)$; since it must be constant as $\sigma^2$ varies over positive values, $\Delta_2=0$. Thus the likelihood-ratio classes are indexed by $(S_1,S_2)$, and $$ T(X)=\left(\sum_{i=1}^n X_i,\ \sum_{i=1}^n X_i^2\right) $$ is minimal sufficient.

Worked Example: Exponential Samples With Unknown Rate

Let $X_1,\ldots,X_n$ be independent exponential observations with rate $\lambda>0$, so the density of each observation is $\lambda e^{-\lambda x}$ for $x>0$. The joint density on the common support $(0,\infty)^n$ is $$ f_\lambda(x)=\lambda^n\exp\left(-\lambda\sum_{i=1}^n x_i\right). $$ For two sample points $x,y$, their likelihood ratio is $$ \frac{f_\lambda(x)}{f_\lambda(y)} = \exp\left\{-\lambda\left(\sum_i x_i-\sum_i y_i\right)\right\}. $$ If the two sums agree, this ratio is $1$. If they differ, the ratio varies with $\lambda>0$, so the samples are not equivalent. Therefore the minimal sufficient statistic is $$ T(X)=\sum_{i=1}^n X_i. $$ The statistic is continuous-valued, but the criterion works in the same way as for the discrete Bernoulli sample, with the continuous-model conclusions interpreted up to null sets.

How to Apply the Test Reliably

For an independent sample, begin with the joint likelihood, compare it at two arbitrary sample points, and ask exactly when the resulting ratio is constant over the entire parameter space. The differences between the sample summaries appearing in the log ratio determine which summaries must agree. It is important to check necessity as well as sufficiency: showing that equal summaries make the ratio constant only proves that the proposed statistic is sufficient, not that it is minimal.

Write the joint density or mass.
Keep its support visible, especially if it might depend on the parameter.

Form a ratio for arbitrary samples.
Cancel factors that are common to numerator and denominator, but retain any parameter-dependent terms.

Find when the ratio is parameter-free.
Use variation over the full parameter space to establish both directions of the equivalence.

Read off the equivalence classes.
Collect the sample summaries that must agree; these summaries form a minimal sufficient statistic when the classes are measurable.

Common Pitfall: Stopping at Sufficiency

A common mistake is to factor a likelihood through a statistic and immediately call that statistic minimal sufficient. Factorization establishes sufficiency only. For instance, in the normal model with unknown mean and variance, $(\sum_i X_i,\sum_i X_i^2)$ is sufficient, but minimality requires checking that equality of these two summaries is also necessary for a parameter-independent likelihood ratio. The ratio calculation above supplies that missing direction.

Another caution is parameter-dependent support. If densities vanish on different regions for different parameter values, a ratio can be zero, undefined, or apparently parameter-free in ways that obscure information carried by support membership. The common-support positivity assumption avoids this issue in the theorem as stated. In models with changing support, the support must be included in the likelihood comparison and the criterion handled with appropriate care. For continuous distributions, individual points have probability zero; the density-based argument identifies statistics only up to the relevant null sets, not by assigning inferential meaning to a particular density value at an isolated point.

Key takeaway: Under a common-support dominated model, minimal sufficient statistics can be found by grouping sample points exactly when their likelihood ratio is independent of the parameter. Prove both directions of that equivalence, and distinguish the resulting minimality claim from sufficiency alone.

Check Your Understanding

For each question, use the likelihood-ratio criterion rather than relying only on factorization.

For a Bernoulli sample, why does a nonzero difference in success counts make the likelihood ratio depend on $p$?
In the normal model with unknown mean and variance, which two sample summaries must agree for two samples to be likelihood-ratio equivalent?
For an exponential sample with unknown rate, write the likelihood ratio and state when it is independent of the rate.
Why does proving that a statistic is sufficient not by itself prove that it is minimal sufficient?
What additional caution is needed when the support of a density depends on the parameter?