Sufficient Statistics from Factorization

Why Likelihood Factorization Identifies Sufficiency

A statistic is useful for inference when it retains the sample information relevant to the unknown parameter. In an exponential family, the likelihood makes that information visible: the observations enter the parameter-dependent part through a small number of sums. The factorization criterion turns this observation into a precise test for sufficiency.

The key point for independent observations is that likelihoods multiply. When each observation contributes a term involving its own sufficient statistic, multiplying the likelihoods adds those contributions in the exponent. This is why sums of individual statistics commonly summarize an entire sample.

Definition: Let \(X\) have density or mass function \(f_\theta\), with respect to a dominating measure \(\mu\), where \(\theta\) is the parameter. A statistic \(T(X)\) is sufficient for \(\theta\) if the conditional distribution of \(X\) given \(T(X)\) does not depend on \(\theta\). Equivalently, once \(T(X)\) is known, the remaining details of the observed sample carry no further information about \(\theta\).

A statistic is a measurable function of the data; it need not be a scalar. If \(T(X)\) is vector-valued, sufficiency refers to the whole vector. The conditional-distribution definition is conceptually direct, while the factorization criterion is often easier to use with a likelihood.

Factorization Criterion: For a dominated family with a common support, \(T(X)\) is sufficient for \(\theta\) if and only if the density can be written, for every \(\theta\), as \(f_\theta(x)=g_\theta(T(x))h(x)\) for \(\mu\)-almost every \(x\). Here \(g_\theta\) may depend on the parameter and on the data only through \(T(x)\), while \(h\) does not depend on \(\theta\). The common-support condition holds for the exponential-family densities considered below, whose support is determined by their parameter-free base measure.

Why the Factorization Criterion Works

The factorization has two directions. If it holds, the parameter-dependent multiplier is constant among data values with the same statistic, so conditioning on that statistic removes it. Conversely, under the common-support condition, sufficiency makes likelihood ratios depend on the data only through the statistic.

For the first direction, regard \(h(x)\mu(dx)\) as a parameter-free measure. Under this measure, take a regular conditional distribution of \(X\) given \(T(X)=t\). Multiplying the measure by \(g_\theta(t)\) changes the distribution of \(T(X)\), but within each fixed value \(t\) it multiplies every possible \(x\) by the same quantity. Thus the conditional distribution of \(X\) given \(T=t\) is unchanged by \(\theta\). This proves sufficiency.

For the converse, fix a reference value \(\theta_0\). Common support permits the likelihood ratio \(R_\theta(x)=f_\theta(x)/f_{\theta_0}(x)\) on that support. Sufficiency means the conditional distribution of \(X\) within each value of \(T\) is the same under \(\theta\) and \(\theta_0\). Disintegrating each distribution into its distribution of \(T\) and its conditional distribution given \(T\), the ratio of the joint densities must therefore be the ratio of the marginal densities of \(T\). That ratio is a function \(q_\theta(T(x))\) alone, so \(f_\theta(x)=q_\theta(T(x))f_{\theta_0}(x)\). This is the required factorization with \(h(x)=f_{\theta_0}(x)\). Both directions rely on the stated domination and common-support conditions.

Independent Samples: The Sum Appears

Suppose \(X_1,\ldots,X_n\) are independent and identically distributed, with density or mass function in canonical exponential-family form \(f_\theta(x)=h(x)\exp\{\eta(\theta)^\mathsf{T}T(x)-A(\theta)\}\). Here \(T(x)\) is a vector of statistics, \(\eta(\theta)\) is the corresponding vector of natural parameters, and \(A(\theta)\) is the log-partition function. The base measure term \(h(x)\) does not depend on \(\theta\).

Independence makes the joint density the product of the individual densities. Collecting the parameter-dependent terms shows exactly which statistic summarizes the sample.

Theorem: Under the preceding iid exponential-family model, \(S(X_1,\ldots,X_n)=\sum_{i=1}^n T(X_i)\) is sufficient for \(\theta\). The assertion does not require that this statistic be minimal or that the parameterization be one-to-one.

\begin{aligned} f_\theta(x_1,\ldots,x_n) &=\prod_{i=1}^n h(x_i) \exp\{\eta(\theta)^\mathsf{T}T(x_i)-A(\theta)\}\\ &=\left[\prod_{i=1}^n h(x_i)\right] \exp\left\{\eta(\theta)^\mathsf{T}\sum_{i=1}^n T(x_i)-nA(\theta)\right\}. \end{aligned}

The bracketed term is independent of \(\theta\), and the other factor depends on the sample only through \(S=\sum_i T(X_i)\). The factorization criterion therefore proves sufficiency. In particular, if the single-observation statistic has several components, one sums each component across observations; the sufficient statistic is then a vector of sums.

Worked Example: Bernoulli Observations

Let \(X_1,\ldots,X_5\) be independent Bernoulli observations with success probability \(p\), where \(0<p<1\). For one observation, \(f_p(x)=p^x(1-p)^{1-x}\), for \(x\in\{0,1\}\). This has \(T(x)=x\), \(\eta(p)=\log(p/(1-p))\), \(A(p)=\log(1-p)^{-1}\), and \(h(x)=1\) on the support. Therefore \(K=\sum_{i=1}^5X_i\) is sufficient. Directly, the joint mass function is \(p^K(1-p)^{5-K}\), which depends on the observed vector only through \(K\).

For the observed data \((1,0,1,1,0)\), \(K=3\), and the likelihood is \(p^3(1-p)^2\). Any other ordered vector of five zeros and ones with three ones has exactly the same likelihood for every \(p\). More directly, conditional on \(K=3\), each of the \(\binom{5}{3}=10\) possible vectors has probability \(p^3(1-p)^2/[p^3(1-p)^2]=1/10\). The conditional probabilities do not involve \(p\), confirming sufficiency from its definition.

Worked Example: Independent Poisson Observations

Let \(X_1,\ldots,X_4\) be independent with \(X_i\sim\operatorname{Poisson}(\lambda)\), where \(\lambda>0\). The individual mass function is \(e^{-\lambda}\lambda^x/x!\), so the individual statistic is \(T(x)=x\). For the observed values \((1,2,0,3)\), the sum is \(S=6\). The joint mass function at these data is

\prod_{i=1}^4\frac{e^{-\lambda}\lambda^{x_i}}{x_i!} =\frac{e^{-4\lambda}\lambda^6}{1!\,2!\,0!\,3!} =\frac{e^{-4\lambda}\lambda^6}{12}.

For any four nonnegative integer observations with sum six, the likelihood has the same parameter-dependent part \(e^{-4\lambda}\lambda^6\). The factorization criterion gives sufficiency of \(S=\sum_iX_i\).

As a check using conditional probabilities, four independent Poisson variables with common mean, given their sum is six, have a multinomial distribution with six trials and four equal cell probabilities \(1/4\). Thus the conditional probability of the particular vector \((1,2,0,3)\) is \(\frac{6!}{1!\,2!\,0!\,3!}(1/4)^6 =\frac{60}{4096}=\frac{15}{1024}\). This value is independent of \(\lambda\), as sufficiency requires.

Worked Example: Normal Observations with Both Parameters Unknown

Let \(X_1,X_2,X_3\) be independent with \(X_i\sim N(\mu,\sigma^2)\), where \(\mu\) is real and \(\sigma^2>0\) are both unknown. The individual density can be written \((2\pi\sigma^2)^{-1/2}\exp\{-x^2/(2\sigma^2)+\mu x/\sigma^2-\mu^2/(2\sigma^2)\}\). Its statistic is the two-component vector \(T(x)=(x,x^2)\). Consequently, the sample statistic is \((S_1,S_2)=(\sum_iX_i,\sum_iX_i^2)\), and both components are needed in this factorization when both parameters are unknown.

For the sample \((1,2,4)\), \(S_1=7\) and \(S_2=21\). The sum of squared deviations appearing in the likelihood is \(\sum_i(X_i-\mu)^2=S_2-2\mu S_1+3\mu^2=21-14\mu+3\mu^2\). Thus the likelihood is

(2\pi\sigma^2)^{-3/2} \exp\left\{-\frac{21-14\mu+3\mu^2}{2\sigma^2}\right\}.

The data enter this expression only through \(S_1\) and \(S_2\), proving their joint sufficiency. For example, at \(\mu=2\) and \(\sigma^2=1\), the exponent is \(-(21-28+12)/2=-5/2\), agreeing with the direct calculation \(-[(1-2)^2+(2-2)^2+(4-2)^2]/2=-5/2\).

Using the Criterion Carefully

When applying the criterion, separate the joint likelihood into a part involving the parameter and a part that does not. Then check whether every appearance of the data in the parameter-dependent part can be expressed through the proposed statistic. A factor that still depends on the data is allowed in \(h(x)\), provided it is parameter-free.

The same multiplication argument extends to independent, not necessarily identically distributed observations when their log likelihoods share a common coefficient for individual statistics. For example, if \(f_{i,\theta}(x_i)=h_i(x_i)\exp\{\eta(\theta)^\mathsf{T}T_i(x_i)-A_i(\theta)\}\), then their product depends on the data through \(\sum_iT_i(X_i)\). If the coefficients of \(T_i\) depend on \(i\) in a parameter-dependent way, the unweighted sum may not suffice; one must factor the actual joint likelihood and retain the required weighted sums or individual components.

Sufficiency alone does not say that a statistic is the smallest possible summary. Different statistics can be sufficient for the same model, and some may contain redundant information. Whether a sufficient statistic can be reduced further is a question of minimal sufficiency, a separate issue from applying the factorization criterion.

Common Mistakes and Why Sufficiency Matters

Keeping only one component of a vector statistic. In the normal model with unknown mean and variance, \(\sum_iX_i\) alone does not express the likelihood: \(\sum_iX_i^2\) also appears. The factorization requires both.
Putting a parameter-dependent term into \(h(x)\). The factor \(h\) must be free of \(\theta\). If a proposed remainder still involves the unknown parameter, the factorization has not established sufficiency.
Assuming every individual sufficient statistic can be summed without checking the model. The sum follows when the joint likelihood combines the individual contributions through that sum, as in the iid exponential-family form. Independence by itself does not guarantee that a particular unweighted sum is sufficient.
Calling a sufficient statistic minimal. Factorization establishes sufficiency, not minimality. A separate argument is needed to show that no coarser statistic is sufficient.

The criterion matters because it converts a conditional-distribution question into an algebraic inspection of the likelihood. In exponential families, this makes the sample sums that preserve parameter information immediate, while also revealing when a proposed summary has omitted a necessary component.

Check Your Understanding

Use the factorization criterion and the definitions above to answer these questions.

For iid Bernoulli observations, write the likelihood and identify the statistic through which it depends on the data.
For iid Poisson observations, explain why the product of the individual likelihoods involves the observations through their sum.
In the normal model with both \(\mu\) and \(\sigma^2\) unknown, name both components of the sufficient statistic and state where each appears in the likelihood.
What role does the parameter-free factor \(h(x)\) play in the factorization criterion?
Why does showing that a statistic is sufficient not, by itself, show that it is minimal?