Why Sufficient Statistics Matter
A sample can contain many observations, but a likelihood may depend on those observations through only a few summaries. When that happens, the summaries retain all the sample information relevant to the parameter in the likelihood. Identifying them is useful for understanding how an exponential-family model compresses data without losing information about its natural parameter.
The key is to inspect the likelihood, not to guess that the sample mean, total, or variance must be sufficient. In an exponential-family representation, the statistic paired with a natural parameter is the starting point. The number of required statistics depends on how many distinct parameter-dependent terms appear.
Here, a statistic is any function of the sample that does not depend on unknown parameters. Sufficiency is always stated relative to a parameter or parameter vector. A statistic sufficient for \(\theta\) when another parameter is held fixed need not be sufficient when that other parameter is also unknown.
Reading the Statistic from the Exponential-Family Form
A one-parameter exponential-family model can often be written as \(f(y;\theta,\phi)=\exp\{[y\theta-b(\theta)]/a(\phi)+c(y,\phi)\}\). Here \(\theta\) is the natural parameter, \(\phi\) is a dispersion or scale parameter, and the remaining functions specify the model. If \(\phi\) is fixed and \(Y_1,\ldots,Y_n\) are independent observations from this same model, their joint likelihood is
The data enter the parameter-dependent term through \(\sum_i y_i\). The remaining data-dependent term, \(\sum_i c(y_i,\phi)\), does not depend on \(\theta\) when \(\phi\) is fixed. Therefore \(\sum_iY_i\) is sufficient for \(\theta\). If \(n\) is fixed, the sample mean is an equivalent statistic because it is a one-to-one rescaling of the sum.
The result does not say that a sum is always sufficient. It applies when the likelihood has the stated form and the other parameters are fixed. If a second unknown parameter multiplies another data summary, that summary may also be needed. Likewise, for independent observations with different known weights, the relevant summary can be a weighted sum rather than the ordinary total.
Why the Factorization Gives Sufficiency
The factorization criterion explains why the likelihood’s data summary is sufficient. Suppose the joint likelihood has the form \(g_\theta(T(y))h(y)\). For two possible samples \(y\) and \(z\) with the same statistic, \(T(y)=T(z)\), their likelihood ratio is
The ratio no longer depends on \(\theta\). Thus, once \(T\) is known, the remaining differences among samples with that same value do not change their relative likelihood as \(\theta\) varies. This is the likelihood-based reason that \(T\) retains all sample information about \(\theta\).
A sufficient statistic need not be the smallest possible summary. In a regular exponential family, the statistic formed from the data terms paired with the natural parameters is often minimal sufficient, but sufficiency alone does not establish minimality. For this tutorial, the main task is to identify a valid sufficient statistic from the likelihood factorization.
Use independence to multiply the individual probability functions or densities.
Group the observations that multiply each natural parameter.
Terms may depend on the observed values, but must not depend on the parameter for which sufficiency is claimed.
Specify whether dispersion or other nuisance parameters are fixed or included in the parameter vector.
Worked Examples
Worked Example: A Total Is Sufficient for One Natural Parameter
Suppose \(Y_1,\ldots,Y_4\) are independent and each takes values \(0,1,2\), with probability \(P(Y_i=y\mid\theta)=e^{\theta y}/Z(\theta)\), where \(Z(\theta)=1+e^\theta+e^{2\theta}\). The parameter \(\theta\) is unknown. For observed values \(0,0,2,2\), identify a sufficient statistic and write the likelihood. Name the method and a common mistake it avoids.
Method. Multiply the four probabilities and collect the terms involving \(\theta\):
Calculate the data summary. The observed total is \(0+0+2+2=4\). Therefore \(L(\theta)=e^{4\theta}/(1+e^\theta+e^{2\theta})^4\). The likelihood depends on the sample values through \(T=\sum_iY_i\), so \(T\) is sufficient for \(\theta\).
Check. The alternative sample \(0,1,1,2\) also has total \(4\), and its likelihood is the same expression. More generally, any samples of size four with the same total have the same likelihood as a function of \(\theta\). The common mistake is to claim sufficiency merely because a total is familiar; here it is justified by the factorization. The sample size is fixed, so the sample mean \(T/4\) is equivalent.
Worked Example: Two Natural Parameters Require Two Summaries
Consider independent observations \(Y_i\in\{0,1,2\}\) with \(P(Y_i=y\mid\alpha,\beta)=e^{\alpha y+\beta y^2}/Z(\alpha,\beta)\), where \(Z(\alpha,\beta)=1+e^{\alpha+\beta}+e^{2\alpha+4\beta}\). For the sample \(0,1,2,2\), find sufficient statistics for the parameter pair \((\alpha,\beta)\), then write the likelihood. The method is factorization; the trap is retaining only the ordinary total.
First summary. Add the observations: \(T_1=\sum_iY_i=0+1+2+2=5\).
Second summary. Square each observation before adding: \(T_2=\sum_iY_i^2=0^2+1^2+2^2+2^2=0+1+4+4=9\).
Likelihood factorization. Independence gives
The likelihood depends on the observed values through the pair \((T_1,T_2)\), so this vector is sufficient for \((\alpha,\beta)\). The total alone does not preserve the likelihood’s dependence on \(\beta\): the squared-value total is also needed. For example, samples can share the same total but have different sums of squares, producing different likelihoods as \(\beta\) varies.
Check. The normalizing factor appears four times because there are four independent observations. The trap is to treat “one exponential-family model” as implying “one sufficient statistic”; the number of required summaries follows the distinct data terms in the exponent.
Worked Example: Unequal Known Weights Produce a Weighted Total
Suppose three independent observations each take values \(0,1,2\), with \(P(Y_i=y\mid\theta)=e^{\theta w_i y}/Z_i(\theta)\), where the known weights are \((w_1,w_2,w_3)=(1,2,1)\) and \(Z_i(\theta)=1+e^{\theta w_i}+e^{2\theta w_i}\). The observed values are \((2,1,0)\). Identify a sufficient statistic for \(\theta\) and factor the likelihood. Avoid assuming the unweighted total must suffice.
Calculate the weighted summary. The contribution from observation 1 is \(1(2)=2\); from observation 2 it is \(2(1)=2\); from observation 3 it is \(1(0)=0\). Hence \(T=\sum_iw_iY_i=2+2+0=4\).
Form the likelihood. The numerator is \(e^{\theta\sum_iw_iy_i}=e^{4\theta}\). The denominator is the product of the three normalizing factors. Since observations 1 and 3 have weight 1, their factors are both \(1+e^\theta+e^{2\theta}\); observation 2 has weight 2, giving \(1+e^{2\theta}+e^{4\theta}\). Therefore
The weights are known, so the denominator depends on the design and \(\theta\), not on the observed outcomes. The data enter the likelihood through \(T=4\), making the weighted sum sufficient. The unweighted total is \(2+1+0=3\), but it is not the summary appearing in the parameter-dependent numerator. The common mistake is to ignore the known observation weights.
When Dispersion or Other Parameters Are Unknown
The phrase “sufficient for the natural parameter” requires attention to what is held fixed. In the common form \(f(y;\theta,\phi)=\exp\{[y\theta-b(\theta)]/a(\phi)+c(y,\phi)\}\), if \(\phi\) is fixed, the sample total is sufficient for \(\theta\). If \(\phi\) is also unknown and sufficiency is required for the joint parameter \((\theta,\phi)\), the factor \(\sum_i c(y_i,\phi)\) may carry additional information about \(\phi\). The total alone is not automatically sufficient for the parameter pair.
The correct procedure is to factor the likelihood with respect to every unknown parameter in the question. Include every data summary that remains in a parameter-dependent factor. A statistic can be sufficient for one parameter under a stated condition and insufficient for a larger parameter vector; there is no contradiction because the sufficiency claims concern different targets.
| Likelihood feature | Candidate summary | Condition to state |
|---|---|---|
| One term \(\theta\sum_i y_i\) | \(\sum_iY_i\) | Other parameters fixed, unless shown otherwise |
| Terms \(\alpha\sum_i y_i+\beta\sum_i y_i^2\) | \((\sum_iY_i,\sum_iY_i^2)\) | Both natural parameters are of interest |
| Term \(\theta\sum_iw_i y_i\) | \(\sum_iw_iY_i\) | Weights are known and treated as fixed |
Common Exam Traps and Practical Interpretation
- Guessing the mean. A sample mean is sufficient only if the likelihood factorization supports it. With fixed sample size, it is equivalent to the total when the total is sufficient; it is not a universal rule.
- Dropping a parameter-dependent summary. If distinct natural parameters multiply different functions of the observations, retain the corresponding vector of summaries.
- Ignoring nuisance parameters. State whether dispersion and other parameters are fixed or included in the sufficiency claim.
- Confusing a statistic with an estimate. A sufficient statistic is a function of the data; it need not itself be an estimate of the parameter.
- Keeping terms that do not depend on the target parameter. Such terms can be part of \(h(y)\) in the factorization for that parameter, although they may matter when another parameter is unknown.
The practical value of sufficiency is disciplined data reduction: the likelihood for the target parameter can be analyzed through the statistic rather than the entire ordered sample. For models in the exponential-family framework, this also gives a direct way to connect the probability model’s natural parameters to the summaries of the observations. This treatment is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
Check Your Understanding
For each problem, identify the statistic or statistics required by the likelihood and state the parameter conditions. The answers are provided for checking.
- Independent \(Y_i\in\{0,1,2\}\) have probabilities proportional to \(e^{\theta y}\), with a common normalizing constant \(Z(\theta)=1+e^\theta+e^{2\theta}\). For sample \((1,2,0)\), what statistic is sufficient for \(\theta\), and what is its observed value? (Answer: \(\sum_iY_i\), observed value \(3\), for fixed sample size.)
- A likelihood for four observations is proportional to \(e^{\alpha\sum_i y_i+\beta\sum_i y_i^2}\) times a parameter-only factor. For data \((0,1,1,2)\), give the sufficient statistic vector. (Answer: \((\sum_iY_i,\sum_iY_i^2)=(4,6)\).)
- Independent observations have likelihood \(L(\theta)\propto e^{\theta(3y_1+y_2)}q(\theta)\), where \(q(\theta)\) does not depend on the observations. What summary is sufficient, and what is its value for \((y_1,y_2)=(1,2)\)? (Answer: \(3Y_1+Y_2\), with observed value \(3(1)+2=5\).)
- A common-dispersion model has likelihood \(L(\theta,\phi;y)\propto\exp\{\theta\sum_i y_i/a(\phi)-nb(\theta)/a(\phi)+\phi\sum_i y_i^2\}\), with both \(\theta\) and \(\phi\) unknown. What pair of data summaries should be retained from these terms? (Answer: \((\sum_iY_i,\sum_iY_i^2)\); the total alone does not capture the displayed \(\phi\)-dependent data term.)
- Two samples of size three have totals both equal to \(3\): \(A=(0,1,2)\) and \(B=(1,1,1)\). A model’s likelihood is proportional to \(e^{\theta\sum_i y_i+\gamma\sum_i y_i^2}\), with \(\theta\) and \(\gamma\) both unknown. Do the equal totals make the samples have the same likelihood for all parameter values? (Answer: No. Their sums of squares are \(5\) and \(3\), respectively, so the likelihood ratio includes \(e^{2\gamma}\), which generally differs from \(1\).)