Why Factorization Identifies a Sufficient Statistic
A statistic is useful for inference when it retains the information in the sample that concerns the parameter. In a likelihood, that information appears in the part of the expression that changes with the parameter. The factorization theorem makes this idea precise: if all parameter dependence can be expressed through a statistic, the remaining details of the sample are unnecessary for inference about that parameter.
This criterion is especially convenient for exponential families. Their likelihoods have a structured form, and the statistic multiplying the natural parameter is often a sum across observations. That sum is the natural statistic of the sample, and factorization shows directly why it is sufficient.
The definition is about the conditional distribution of the whole sample, not just about whether a particular estimate can be calculated from \(T\). We will use the factorization theorem under explicit regularity conditions. Assume the sample space is a standard Borel space, the distributions have densities \(f_\theta\) with respect to a common measure \(\mu\), and the model has common support: there is a set \(S\) such that each density is positive \(\mu\)-almost everywhere on \(S\) and zero outside \(S\). These conditions let us compare every model distribution with a fixed reference distribution.
Proof of the Factorization Theorem
First suppose the factorization holds. Fix a reference parameter value \(\theta_0\). On the common support, $$ \frac{f_\theta(x)}{f_{\theta_0}(x)} = \frac{g_\theta(T(x))}{g_{\theta_0}(T(x))} $$ where the denominator is positive, up to null sets. Thus the likelihood ratio \(dP_\theta/dP_{\theta_0}\) is a function of \(T\). Weighting the reference distribution by a nonnegative function of \(T\) changes the distribution of \(T\), but does not change the conditional distribution of \(X\) given \(T\). More explicitly, for any measurable set \(A\), conditional expectation under \(P_{\theta_0}\) gives $$ P_\theta(X\in A\mid T) = \frac{E_{\theta_0}\!\left[1_{\{X\in A\}}\,r_\theta(T)\mid T\right]} {E_{\theta_0}\!\left[r_\theta(T)\mid T\right]} = P_{\theta_0}(X\in A\mid T), $$ where \(r_\theta(T)=g_\theta(T)/g_{\theta_0}(T)\). Hence the conditional law is parameter-independent, so \(T\) is sufficient.
Conversely, suppose \(T\) is sufficient. Let \(Q_\theta\) be the distribution of \(T\) under \(P_\theta\). Since \(P_\theta\) and \(P_{\theta_0}\) are mutually absolutely continuous, so are \(Q_\theta\) and \(Q_{\theta_0}\). Write \(q_\theta=dQ_\theta/dQ_{\theta_0}\), a function of the statistic value. Sufficiency gives a common conditional distribution \(K(A\mid T)\) of \(X\) given \(T\). For every measurable sample set \(A\), $$ P_\theta(X\in A) = \int K(A\mid t)\,Q_\theta(dt) = E_{\theta_0}\!\left[q_\theta(T)1_{\{X\in A\}}\right]. $$ The last equality follows by conditioning under \(P_{\theta_0}\) on \(T\). It follows that \(dP_\theta/dP_{\theta_0}=q_\theta(T)\), \(P_{\theta_0}\)-almost surely. In terms of densities, $$ f_\theta(x)=q_\theta(T(x))f_{\theta_0}(x) $$ almost everywhere. Taking \(g_\theta=q_\theta\) and \(h=f_{\theta_0}\) proves the factorization. \(\square\)
Natural Statistics in Exponential-Family Samples
Consider independent observations \(X_1,\ldots,X_n\) with density, relative to a common measure, $$ f_\theta(x)=h(x)\exp\!\left\{\eta(\theta)^{\mathsf T}t(x)-A(\theta)\right\}. $$ Here \(t(x)\) is a \(k\)-dimensional statistic, \(\eta(\theta)\) is a \(k\)-dimensional natural-parameter function, \(A(\theta)\) is the log normalizing term, and the support is assumed not to depend on \(\theta\). The joint density is $$ f_\theta(x_1,\ldots,x_n) = \left(\prod_{i=1}^n h(x_i)\right) \exp\!\left\{ \eta(\theta)^{\mathsf T}\sum_{i=1}^n t(x_i)-nA(\theta) \right\}. $$ All dependence on the parameter through the observations is carried by $$ T(X_1,\ldots,X_n)=\sum_{i=1}^n t(X_i). $$ The product of base-density terms does not depend on \(\theta\). The factorization theorem therefore gives the following result.
This conclusion follows from the factorization; it does not require that the components of \(T\) be distinct or nonredundant. Whether a statistic can be reduced further is a separate question. The examples below focus on finding and interpreting the factorization itself.
Worked Example: Bernoulli Observations
Let \(X_1,\ldots,X_n\) be independent Bernoulli\((p)\) observations, with \(0<p<1\). Each observation has mass function \(p^x(1-p)^{1-x}\) for \(x\in\{0,1\}\). For a sample \(x_1,\ldots,x_n\), the joint mass is $$ f_p(x_1,\ldots,x_n) = \prod_{i=1}^n p^{x_i}(1-p)^{1-x_i} = p^{\sum_i x_i}(1-p)^{n-\sum_i x_i}. $$
Set \(T=\sum_{i=1}^n X_i\), the number of successes. The mass factors into \(g_p(T)=p^T(1-p)^{n-T}\) and \(h(x_1,\ldots,x_n)=1\) on the sample space \(\{0,1\}^n\). Thus \(T\) is sufficient for \(p\).
The conditional distribution gives a direct check. For any binary sample with \(k\) successes, its unconditional probability is \(p^k(1-p)^{n-k}\). There are \(\binom{n}{k}\) such samples, so \(P_p(T=k)=\binom{n}{k}p^k(1-p)^{n-k}\). Consequently, conditional on \(T=k\), each such sample has probability $$ \frac{p^k(1-p)^{n-k}}{\binom{n}{k}p^k(1-p)^{n-k}} = \frac{1}{\binom{n}{k}}, $$ which does not depend on \(p\).
Worked Example: Normal Observations with Both Parameters Unknown
Let \(X_1,\ldots,X_n\) be independent \(N(\mu,\sigma^2)\) observations, with \(\mu\in\mathbb R\) and \(\sigma^2>0\) both unknown. The joint density is $$ f_{\mu,\sigma^2}(x) = (2\pi\sigma^2)^{-n/2} \exp\!\left\{-\frac{1}{2\sigma^2} \sum_{i=1}^n(x_i-\mu)^2\right\}. $$ Expanding the square gives $$ \sum_{i=1}^n(x_i-\mu)^2 = \sum_{i=1}^n x_i^2-2\mu\sum_{i=1}^n x_i+n\mu^2. $$
Therefore the density can be written as $$ f_{\mu,\sigma^2}(x) = (2\pi\sigma^2)^{-n/2} \exp\!\left\{-\frac{\sum_i x_i^2}{2\sigma^2} +\frac{\mu\sum_i x_i}{\sigma^2} -\frac{n\mu^2}{2\sigma^2}\right\}. $$ The parameter-dependent terms involving the sample are functions of \(T=(\sum_i X_i,\sum_i X_i^2)\). The factorization theorem shows that this pair is sufficient for \((\mu,\sigma^2)\). Equivalently, because \(n\) is fixed, one can use the sample mean together with the sum of squares.
For \(n\ge 2\), the sample mean alone does not determine the sum-of-squares term. For instance, with \(n=2\), the samples \((0,2)\) and \((1,1)\) both have mean \(1\), but their sums of squares are \(4\) and \(2\), respectively. For any \(n>2\), appending \(n-2\) zeros gives the same contrast at a common mean. This is not merely a display issue: under an orthogonal transformation of the normal sample, one coordinate is proportional to the sample mean and the other \(n-1\) coordinates describe residual variation. Conditional on the mean, those residual coordinates have a normal distribution whose variance is \(\sigma^2\); their conditional law still depends on the unknown variance. Thus the mean alone is not sufficient when \(n\ge2\) and both parameters are unknown.
The case \(n=1\) is different. Then \(T=(X_1,X_1^2)\), and the second component is determined by the first. The sample mean is \(X_1\) itself, so it determines the entire sample and is sufficient. A claim that both summaries are always needed would therefore be false.
Worked Example: Poisson Observations
Suppose \(X_1,\ldots,X_n\) are independent Poisson\((\lambda)\) observations, with \(\lambda>0\). For nonnegative integer observations, $$ f_\lambda(x_1,\ldots,x_n) = \prod_{i=1}^n\frac{e^{-\lambda}\lambda^{x_i}}{x_i!} = \frac{e^{-n\lambda}\lambda^{\sum_i x_i}}{\prod_i x_i!}. $$ The parameter-dependent sample term is determined by \(T=\sum_i X_i\), and the denominator is parameter-free. Hence \(T\) is sufficient for \(\lambda\).
To verify the conditional interpretation, put \(s=\sum_i x_i\). The sum \(T\) has a Poisson\((n\lambda)\) distribution, so \(P_\lambda(T=s)=e^{-n\lambda}(n\lambda)^s/s!\). Dividing the joint mass by this probability gives $$ P_\lambda(X_1=x_1,\ldots,X_n=x_n\mid T=s) = \frac{s!}{n^s\prod_i x_i!}, \qquad \sum_i x_i=s. $$ This conditional probability contains no \(\lambda\). The factorial terms still distinguish allocations of the total among observations, but their conditional probabilities are parameter-free.
Common Mistakes and Why Sufficiency Matters
- Keeping only the most familiar summary. The sample mean is sufficient for a normal mean when the variance is known, but when both normal parameters are unknown, the likelihood also involves the sum of squares. For \(n\ge2\), the mean alone does not capture that term.
- Forgetting exceptional sample sizes. In the unknown-variance normal example, when \(n=1\), the sample mean is the observation and determines its square. The two components of the displayed sufficient statistic are then redundant.
- Confusing a parameter-free factor with an irrelevant likelihood term before conditioning. A factor such as \(\prod_i x_i!^{-1}\) is needed to define the joint distribution. It is parameter-free, which is why it belongs in \(h(x)\), not why it can be discarded from every calculation.
- Assuming every sufficient statistic is a single number. In a multiparameter family, the natural statistic can be a vector, as in the normal example.
Sufficiency is useful because it identifies a reduction of the raw sample that preserves all information about the parameter in the conditional-distribution sense. In exponential-family models, the log-likelihood makes this reduction visible: its sample-dependent, parameter-varying terms are functions of the natural statistic. Factorization supplies a direct proof, while the conditional-law calculation helps interpret what the reduction means.
Check Your Understanding
Use the factorization and conditional-distribution ideas to answer the following.
- For an independent Bernoulli sample, what statistic carries the parameter-dependent part of the joint likelihood?
- State the common-support condition used in the factorization theorem, and explain why the parameter-free factor must not depend on the parameter.
- For independent normal observations with both \(\mu\) and \(\sigma^2\) unknown, identify the two sample summaries in the likelihood factorization.
- Why is the sample mean alone sufficient for the one-observation normal sample even though the likelihood expression includes a square?
- For independent Poisson observations, derive the conditional probability of an allocation \((x_1,\ldots,x_n)\) given its total \(s\), and identify which parameter cancels.