Sufficiency in Exponential-Family Models

Why Exponential-Family Structure Helps

A sample can contain many individual observations, yet its likelihood may depend on those observations through only a few summaries. In an exponential-family model, the density or mass function displays those summaries directly: each one is multiplied by a component of the natural parameter. This makes the family’s sufficient statistics visible from the joint likelihood.

The key is to separate terms according to what they depend on. A factor that depends on the parameter but not on the observed sample does not require a data summary. A factor that depends on the data but not on the parameter does not affect the likelihood’s parameter dependence. The remaining data-dependent terms indicate which summaries to retain.

Definition: A statistic \(T(X)\) is sufficient for \(\theta\) if the conditional distribution of the data \(X\), given \(T(X)\), does not depend on \(\theta\). For a dominated model, the likelihood factorization criterion says that \(T\) is sufficient when the joint density or mass function can be written as \(f_\theta(x)=g_\theta(T(x))h(x)\), where \(g_\theta\) and \(h\) are nonnegative functions and \(h\) does not depend on \(\theta\).

The factorization criterion is the tool used here. In particular, an exponential-family factorization will identify a sufficient statistic, but it does not by itself establish that the statistic is minimal or that every component is necessary.

From the One-Observation Form to the Sample Statistic

Consider an independent, identically distributed sample \(X_1,\ldots,X_n\), with each observation having a density or mass function of the form

f_\theta(x)=h(x)c(\theta)\exp\left\{\sum_{j=1}^{k}\eta_j(\theta)t_j(x)\right\}.

Assume that the model has a common support that does not depend on \(\theta\), and that the densities or mass functions are defined with respect to the same dominating measure. Here \(h\) is parameter-free, \(c(\theta)>0\), and the \(k\) functions \(\eta_j(\theta)\) are the natural-parameter components. The functions \(t_j(x)\) specify which features of an observation are paired with those components.

Independence makes the joint density or mass function a product of the one-observation terms. Define the sample sums \(S_j=\sum_{i=1}^{n}t_j(X_i)\). Multiplying the factors and collecting terms gives the following result.

Theorem: Under the assumptions above, the vector \(S(X)=(S_1,\ldots,S_k)\) is sufficient for \(\theta\).

Proof. Independence gives the joint density or mass function

f_\theta(x_1,\ldots,x_n) =\prod_{i=1}^{n}h(x_i)c(\theta) \exp\left\{\sum_{j=1}^{k}\eta_j(\theta)t_j(x_i)\right\}.

Since the product of exponentials is the exponential of the sum of their exponents, this becomes

f_\theta(x_1,\ldots,x_n) = \left[\prod_{i=1}^{n}h(x_i)\right] c(\theta)^n \exp\left\{\sum_{j=1}^{k}\eta_j(\theta) \left(\sum_{i=1}^{n}t_j(x_i)\right)\right\}.

The expression in square brackets depends on the data but not on \(\theta\). The other factor depends on the data only through \(S_1,\ldots,S_k\). Thus the joint density or mass function has the factorization \(g_\theta(S(x))h_n(x)\), with \(h_n(x)=\prod_i h(x_i)\). By the likelihood factorization criterion, \(S\) is sufficient. \(\square\)

This derivation identifies the role of each component: \(\eta_j(\theta)\) multiplies \(S_j=\sum_i t_j(X_i)\). The factor \(c(\theta)^n\) can affect inference about \(\theta\), but it is the same for every sample of size \(n\); it does not require an additional data summary.

Write the one-observation form.
Identify the parameter-free factor, the parameter-only factor, and each product \(\eta_j(\theta)t_j(x)\).

Multiply across observations.
Use independence to form the joint density or mass function.

Collect the sample terms.
For each component \(j\), replace the individual terms \(t_j(X_i)\) by their sum \(S_j\).

Apply the factorization criterion.
Check that the remaining data-only factor is parameter-free and that the parameter-dependent factor uses the data only through the proposed summaries.

Worked Derivations

Worked Example: A Geometric Sample

Suppose \(X_1,\ldots,X_5\) are independent with \(P(X_i=x)=p(1-p)^x\) for \(x=0,1,2,\ldots\), where \(0<p<1\). For observed values \(0,1,0,3,2\), derive a sufficient statistic and write the likelihood.

For one observation, the mass function can be written as \(p\exp\{x\log(1-p)\}\). Thus the parameter-free factor is \(h(x)=1\), the parameter-only factor is \(c(p)=p\), and the data term \(t(x)=x\) is paired with \(\eta(p)=\log(1-p)\). The theorem gives \(S=\sum_i X_i\) as sufficient.

For the observed sample, \(S=0+1+0+3+2=6\). Multiplying the five probabilities gives

L(p)=p^5(1-p)^{0+1+0+3+2} =p^5(1-p)^6.

The likelihood depends on the observations through their sum, not through their order. For example, any other sample of size five with nonnegative integer entries summing to six has this same likelihood for every \(p\). The value \(p^5\) depends on the parameter but not on the observed values, so it does not call for a further statistic.

Worked Example: Two Components in a Finite-Support Model

Let each \(X_i\) take values \(0,1,2\), with mass function

P_{\alpha,\beta}(X_i=x) =\frac{\exp(\alpha x+\beta x^2)} {1+\exp(\alpha+\beta)+\exp(2\alpha+4\beta)}, \qquad x\in\{0,1,2\},

where \(\alpha,\beta\in\mathbb{R}\). For an independent sample of size four with observed values \(0,1,1,2\), find the sufficient-statistic vector and the likelihood.

The denominator is the sum of the three unnormalized probabilities, so the masses over \(0,1,2\) sum to one. In the exponential-family form, \(t_1(x)=x\) and \(t_2(x)=x^2\), paired respectively with \(\alpha\) and \(\beta\). The sample summaries are

S_1=0+1+1+2=4,\qquad S_2=0^2+1^2+1^2+2^2=6.

Writing \(Z(\alpha,\beta)=1+\exp(\alpha+\beta)+\exp(2\alpha+4\beta)\), the joint likelihood is

L(\alpha,\beta) =\frac{\exp\{\alpha(4)+\beta(6)\}}{Z(\alpha,\beta)^4}.

The data enter this expression through the pair \((S_1,S_2)=(4,6)\), so that pair is sufficient. Keeping only \(S_1\) would not retain the displayed \(\beta S_2\) contribution in general. This observation identifies the summary paired with each parameter component; by itself, it is not a proof that each summary is necessary or that the pair is minimal.

Worked Example: A Continuous One-Parameter Family

Suppose \(X_1,X_2,X_3\) are independent with density \(f_\theta(x)=(\theta+1)x^\theta\) for \(0<x<1\), where \(\theta>-1\). For observations \(0.2,0.5,0.8\), derive a sufficient statistic and evaluate the likelihood at \(\theta=1\).

The density is normalized because \(\int_0^1(\theta+1)x^\theta\,dx=1\) for \(\theta>-1\). Its exponential-family form is \((\theta+1)\exp\{\theta\log x\}\). Thus the sample sum of \(t(x)=\log x\), equivalently the product of the observations, is sufficient:

S=\log(0.2)+\log(0.5)+\log(0.8)=\log(0.08).

The likelihood is \(L(\theta)=(\theta+1)^3(0.08)^\theta\). At \(\theta=1\), it equals \(2^3(0.08)=0.64\). Direct substitution into the three densities gives \((2\cdot0.2)(2\cdot0.5)(2\cdot0.8)=0.64\), confirming the calculation. Samples with the same product have the same likelihood for every \(\theta>-1\).

What the Summaries Do—and Do Not—Tell You

The components of the sufficient-statistic vector are determined by the data functions \(t_j\) in the chosen exponential-family representation. If a model has several natural-parameter components, the likelihood displays the corresponding sample sums together. A statistic can be vector-valued: sufficiency does not require reducing the data to a single number.

The resulting vector is a sufficient statistic, not automatically the smallest possible one. Components can be redundant if the parameter space restricts the natural parameters or if the data functions have relationships on the support. Different valid representations can also package the same likelihood dependence in different ways. Establishing minimal sufficiency requires an additional argument; it does not follow just from counting the terms in an exponential expression.

A common mistake is to treat every factor involving the parameter as though it demanded a data summary. For example, \(c(\theta)^n\) varies with \(\theta\), but it is not a function of the observed values. The correction is to inspect the joint likelihood and ask which parts of the data appear inside its parameter-dependent factor. Conversely, ignoring a term such as \(\eta_j(\theta)S_j\) can discard data dependence relevant to the parameter.

Key takeaway: For an independent, identically distributed sample in the stated exponential-family form, collect each data function \(t_j(X_i)\) into its sample sum \(S_j\). The resulting vector is sufficient by likelihood factorization; whether it can be reduced further is a separate question.

Check Your Understanding

Use the factorization argument to answer these questions.

In the stated exponential-family form, what sample summary is paired with the natural-parameter component \(\eta_j(\theta)\)?
Why does \(c(\theta)^n\) not require an additional data statistic?
If a likelihood includes \(\eta_1(\theta)S_1+\eta_2(\theta)S_2\), what sufficient-statistic vector does the direct factorization provide?
Why does identifying that vector not, on its own, prove that it is minimal?
What common-support condition was assumed in deriving the sample factorization?

Sufficiency in Exponential-Family Models

What You'll Learn