The Likelihood Factorization Criterion

Why Factor a Likelihood?

A sample likelihood can be a complicated expression in all the observed values. Yet the parameter may enter that expression through only one or a few summaries. The likelihood factorization criterion makes this observation precise: it tells us when a statistic retains all the sample’s information about the parameter, in the sense that the data’s conditional distribution given that statistic no longer depends on the parameter.

The criterion is especially useful for independent observations. Their joint likelihood is a product of individual contributions, and collecting terms often reveals which function of the sample carries the parameter dependence. The remaining factors may depend on the observed data, but they must not depend on the parameter.

Definition: Let $X$ have distribution $P_\theta$, with $\theta$ in a parameter space $\Theta$, and let $T=T(X)$ be a measurable statistic. The statistic $T$ is sufficient for $\theta$ if the conditional distribution of $X$ given $T$ can be chosen to be the same for every $\theta$. Equivalently, observing $T$ leaves no further information in $X$ about $\theta$, as expressed by that conditional distribution.

We use the standard regular setting in which the sample space and the range of $T$ are standard Borel spaces, so regular conditional distributions exist. For the full “if and only if” criterion and proof below, assume that the model has densities $f_\theta$ with respect to a common sigma-finite measure $\mu$, and that $f_\theta(x)>0$ for $\mu$-almost every $x$, for every $\theta$. The positivity assumption gives the distributions a common support up to null sets. Factorizations are understood to hold $\mu$-almost everywhere for each fixed $\theta$.

The Factorization Criterion

Theorem (Neyman–Fisher factorization criterion): Under the common positive-density and regularity assumptions just stated, $T$ is sufficient for $\theta$ if and only if there are nonnegative measurable functions $g_\theta$ and $h$ such that $$ f_\theta(x)=g_\theta(T(x))h(x). $$ Here $g_\theta$ may depend on the parameter, but its data dependence is only through $T(x)$; $h$ may depend on the full data $x$, but not on $\theta$.

The two factors have distinct roles. The factor $g_\theta(T(x))$ contains all the way the likelihood can vary with the data and the parameter together. The factor $h(x)$ can alter the relative likelihood of different samples, but the alteration is the same for every parameter value. The criterion does not require $h$ to be constant.

Proof of the Criterion

First suppose the factorization holds. Fix a reference value $\theta_0$. Since the densities are positive almost everywhere, $h(x)$ and $g_{\theta_0}(T(x))$ are positive almost everywhere where the factorization applies. Therefore $$ \frac{f_\theta(x)}{f_{\theta_0}(x)} = \frac{g_\theta(T(x))}{g_{\theta_0}(T(x))} =:q_\theta(T(x)) $$ almost everywhere. Thus the likelihood ratio between $P_\theta$ and $P_{\theta_0}$ is a function of $T$ alone.

Let $K_0(t,B)$ be a regular conditional probability for $X\in B$ given $T=t$ under $P_{\theta_0}$. For a measurable set $A$ in the range of $T$ and a measurable sample-space set $B$, change measure using the likelihood ratio: $$ P_\theta(X\in B,\ T\in A) = E_{\theta_0}\!\left[1_{\{X\in B,T\in A\}}q_\theta(T)\right] = \int_A q_\theta(t)K_0(t,B)\,P_{\theta_0,T}(dt). $$ Taking $B$ to be the whole sample space shows that $P_{\theta,T}(dt)=q_\theta(t)P_{\theta_0,T}(dt)$. Substitution into the preceding display gives $$ P_\theta(X\in B,\ T\in A) = \int_A K_0(t,B)\,P_{\theta,T}(dt). $$ This is the defining conditional-probability identity for $K_0$ to be a conditional distribution of $X$ given $T$ under $P_\theta$. It is the same kernel for every $\theta$, so $T$ is sufficient.

For the converse, suppose $T$ is sufficient, and again use $\theta_0$ as a reference value. Write $r_\theta(x)=f_\theta(x)/f_{\theta_0}(x)$, which exists almost everywhere by positivity. The distribution of $T$ under $P_\theta$ is absolutely continuous with respect to its distribution under $P_{\theta_0}$; let $a_\theta(t)$ be its density ratio. Sufficiency gives a common conditional kernel $K(t,B)$ for $X$ given $T=t$. For every measurable $B$ and every measurable set $A$ in the range of $T$, $$ \int_{\{T\in A\}\cap B}r_\theta(x)\,P_{\theta_0}(dx) = \int_A K(t,B)a_\theta(t)\,P_{\theta_0,T}(dt). $$ By the conditional-probability identity under $P_{\theta_0}$, the right-hand side is also $$ \int_{\{T\in A\}\cap B}a_\theta(T(x))\,P_{\theta_0}(dx). $$ Since this holds for every $A$ and $B$, the two integrands agree $P_{\theta_0}$-almost surely: $r_\theta(X)=a_\theta(T)$. Hence $$ f_\theta(x)=a_\theta(T(x))f_{\theta_0}(x) $$ almost everywhere. Taking $g_\theta=a_\theta$ and $h=f_{\theta_0}$ gives the required factorization. $\square$

Applying the Criterion to Independent Samples

For independent observations $X_1,\ldots,X_n$, the joint density or mass function is the product of the individual contributions: $$ f_\theta(x_1,\ldots,x_n)=\prod_{i=1}^n f_{i,\theta}(x_i). $$ To apply the criterion, multiply these terms and regroup them. A useful statistic is one for which the parameter-dependent part of the product can be written using that statistic alone. Any remaining factor may depend on every observed value, provided it is parameter-free.

Write the joint likelihood.
Use independence to express it as a product of the individual densities or mass functions.

Collect parameter-dependent terms.
Look for a function of the observations that summarizes how these terms vary across possible samples.

Check both factors.
Confirm that the parameter-dependent factor uses the data only through the proposed statistic and that the other factor contains no unknown parameter.

Worked Example: Bernoulli Observations

Let $X_1,\ldots,X_n$ be independent Bernoulli observations with success probability $p$, where $0<p<1$. Each observation is either 0 or 1, with mass function $f_p(x_i)=p^{x_i}(1-p)^{1-x_i}$. For a realized sample $x=(x_1,\ldots,x_n)$, independence gives $$ f_p(x)=\prod_{i=1}^n p^{x_i}(1-p)^{1-x_i}. $$

Collect the exponents by summing: $$ f_p(x) =p^{\sum_{i=1}^n x_i}(1-p)^{\sum_{i=1}^n(1-x_i)} =p^{\sum_{i=1}^n x_i}(1-p)^{n-\sum_{i=1}^n x_i}. $$ Thus, with $T(x)=\sum_{i=1}^n x_i$, we have $g_p(T)=p^T(1-p)^{n-T}$ and $h(x)=1$. The factorization criterion shows that the number of successes is sufficient for $p$. The criterion establishes sufficiency; it does not, by itself, assert that this is the only possible sufficient statistic.

Worked Example: Exponential Observations

Let $X_1,\ldots,X_n$ be independent exponential observations with rate $\lambda>0$, each having density $f_\lambda(x_i)=\lambda e^{-\lambda x_i}$ for $x_i>0$. On the common sample support $x_i>0$ for all $i$, their joint density is $$ f_\lambda(x) =\prod_{i=1}^n \lambda e^{-\lambda x_i} =\lambda^n e^{-\lambda\sum_{i=1}^n x_i}. $$

Set $T(x)=\sum_{i=1}^n x_i$. The joint density is $g_\lambda(T)=\lambda^n e^{-\lambda T}$ times $h(x)=1$, so $T$ is sufficient for $\lambda$. The support condition matters: outside the sample support, the joint density is zero for every $\lambda$. If support restrictions depend on the parameter, their indicators must be included in the likelihood and cannot be silently discarded as parameter-free constants.

Worked Example: Normal Observations With Known Variance

Let $X_1,\ldots,X_n$ be independent with $X_i\sim N(\mu,\sigma^2)$, where $\sigma^2>0$ is known and $\mu$ is unknown. The joint density is $$ f_\mu(x) =(2\pi\sigma^2)^{-n/2} \exp\!\left(-\frac{1}{2\sigma^2}\sum_{i=1}^n(x_i-\mu)^2\right). $$

Expand the square inside the sum: $$ \sum_{i=1}^n(x_i-\mu)^2 =\sum_{i=1}^n x_i^2-2\mu\sum_{i=1}^n x_i+n\mu^2. $$ Substituting and separating the terms gives $$ f_\mu(x) = \underbrace{\exp\!\left(\frac{\mu}{\sigma^2}\sum_{i=1}^n x_i-\frac{n\mu^2}{2\sigma^2}\right)}_{g_\mu(T(x))} \underbrace{(2\pi\sigma^2)^{-n/2} \exp\!\left(-\frac{1}{2\sigma^2}\sum_{i=1}^n x_i^2\right)}_{h(x)}, \qquad T(x)=\sum_{i=1}^n x_i. $$ The second factor depends on the full sample but not on $\mu$; the first depends on the sample only through the sum. Therefore the sum is sufficient for the unknown mean in this model.

What the Criterion Does—and Does Not—Say

A factorization is an equality for the likelihood, not a claim that the statistic is numerically equivalent to the data. For example, in the normal calculation the parameter-free factor still changes with $\sum_i x_i^2$. It is allowed to do so: sufficiency says that the conditional distribution of the full data given $T$ is parameter-free, not that the remaining data are determined by $T$.

The statistic must also be a function of the observed sample that does not depend on the unknown parameter. A proposed summary such as $T_\theta(x)$, defined using the unknown $\theta$, does not qualify as a statistic for this purpose. Once a candidate statistic is chosen, inspect every term in the likelihood, including any indicator describing where the density is positive.

Mistake: Treating any term that does not involve the observations as the only permissible $h(x)$. Correction: $h$ may depend on the entire sample; it must simply be independent of $\theta$.
Mistake: Calling a statistic sufficient because it appears somewhere in the likelihood. Correction: The parameter-dependent factor must depend on the data only through that statistic.
Mistake: Dropping a parameter-dependent support indicator because it is not written as an ordinary algebraic factor. Correction: The support is part of the likelihood and must be included in the factorization.

Key takeaway: Factor the full likelihood, including support restrictions. A statistic is sufficient when the likelihood can be written as a parameter-dependent function of that statistic multiplied by a parameter-free function of the data.

Check Your Understanding

For each question, focus on how the parameter and the observations enter the factorization.

In the Bernoulli example, which part of the joint mass function depends on the sample only through the number of successes?
Why is it permissible for the parameter-free factor in the normal example to depend on the sum of squared observations?
For independent observations with densities $f_{i,\theta}(x_i)$, what is the first likelihood expression to write before looking for a sufficient statistic?
What role does the common positive-density assumption play in the stated “if and only if” theorem?
If a model’s support changes with the parameter, what part of the likelihood must be checked before applying a factorization argument?