Sufficiency in Independent Nonidentical Samples

Why Independence Does Not Require Identical Distributions

A sample is often introduced as a collection of independent observations from one common distribution. In a generalized linear model, however, the observations can be independent while having different distributions: their known exposures, offsets, or covariate values may differ. The relevant question for sufficiency is not whether the observations have the same distribution, but how the parameter enters their joint likelihood.

When the design quantities are fixed and known, they can affect the likelihood differently for each observation. The data may nevertheless enter that likelihood through a much smaller summary, often a weighted sum. This tutorial derives that summary directly from the likelihood.

Definition: Let $Y_1,\ldots,Y_n$ be independent observations with joint density or mass function $f_\theta(y_1,\ldots,y_n)$, relative to a common dominating measure, and let $T(Y_1,\ldots,Y_n)$ be a statistic. The statistic $T$ is sufficient for $\theta$ if the joint likelihood factors as $$ f_\theta(y_1,\ldots,y_n)=g_\theta\!\left(T(y_1,\ldots,y_n)\right)h(y_1,\ldots,y_n), $$ where $g_\theta$ may depend on the parameter and the data only through $T$, while $h$ does not depend on $\theta$. In the applications below, the support is assumed not to depend on $\theta$, and any design quantities used in $T$ are fixed and known.

The factorization criterion, covered in the earlier material on likelihood factorization, applies whether or not the observations are identically distributed. Independence lets us write the joint likelihood as a product of observation-specific likelihoods. We then collect every data-dependent term that multiplies an unknown parameter.

A Scalar Natural-Parameter Result

Suppose the independent observations have densities or mass functions of the form $$ f_i(y_i;\theta)=h_i(y_i)\exp\!\left\{y_i\bigl(a_i\eta(\theta)+c_i\bigr)-b_i\bigl(a_i\eta(\theta)+c_i\bigr)\right\},\qquad i=1,\ldots,n. $$ Here $a_i$ and $c_i$ are known constants, $\eta(\theta)$ is a function of the unknown parameter, and $h_i$ and $b_i$ are known functions. Assume the displayed expression defines the observation distributions on a support that does not depend on $\theta$. The functions $h_i$ and $b_i$ are allowed to differ across observations.

Theorem: Under the conditions above, the statistic $$ T(Y_1,\ldots,Y_n)=\sum_{i=1}^n a_iY_i $$ is sufficient for $\theta$. The observations need not be identically distributed.

Proof. By independence, the joint density or mass function is the product of the individual terms. Substituting the assumed form gives

\begin{aligned} f_\theta(y_1,\ldots,y_n) &=\prod_{i=1}^n h_i(y_i) \exp\!\left\{y_i\bigl(a_i\eta(\theta)+c_i\bigr) -b_i\bigl(a_i\eta(\theta)+c_i\bigr)\right\}\\ &=\exp\!\left\{\eta(\theta)\sum_{i=1}^n a_i y_i -\sum_{i=1}^n b_i\bigl(a_i\eta(\theta)+c_i\bigr)\right\} \prod_{i=1}^n h_i(y_i)e^{c_i y_i}. \end{aligned}

The first factor depends on the observed values only through $\sum_i a_i y_i$; the second factor has no dependence on $\theta$. This is the required factorization, so $T$ is sufficient. $\square$

The coefficients $a_i$ explain why the sufficient summary is weighted. If all $a_i=1$, the summary is the ordinary sum. If the parameter enters observation $i$'s natural parameter with a different known coefficient, its response contributes in proportion to that coefficient.

Worked Example: Bernoulli Responses With Different Known Slopes

Let $Y_i$ be independent Bernoulli variables with success probabilities satisfying $$ \operatorname{logit}(p_i)=a_i\theta, $$ where $\operatorname{logit}(p)=\log\{p/(1-p)\}$, $\theta$ is unknown, and the known coefficients are $a_1=1$, $a_2=2$, and $a_3=3$. For observed responses $(y_1,y_2,y_3)=(1,0,1)$, find a sufficient statistic and write the likelihood.

For a Bernoulli variable with natural parameter $\eta_i=a_i\theta$, its mass function is $$ P(Y_i=y_i)=\exp\{y_i a_i\theta-\log(1+e^{a_i\theta})\},\qquad y_i\in\{0,1\}. $$ Thus the observations are independent but have different success probabilities in general. Multiplying their mass functions yields

\begin{aligned} L(\theta) &=\exp\!\left\{\theta(y_1+2y_2+3y_3) -\log(1+e^\theta)-\log(1+e^{2\theta})-\log(1+e^{3\theta})\right\}. \end{aligned}

The data enter the parameter-dependent part through $T=y_1+2y_2+3y_3$, so $T$ is sufficient. For the observed responses, $T=1+2(0)+3(1)=4$, and the likelihood is $$ L(\theta)=\frac{e^{4\theta}}{(1+e^\theta)(1+e^{2\theta})(1+e^{3\theta})}. $$ The weights come from how $\theta$ enters each observation's log odds, not from the observed success probabilities.

Worked Example: Poisson Counts With Exposure and Different Coefficients

Suppose independent counts $Y_i$ have means $\lambda_i=e_i e^{a_i\theta}$, with known exposures $(e_1,e_2,e_3)=(1,2,4)$ and known coefficients $(a_1,a_2,a_3)=(1,2,1)$. For observed counts $(2,1,0)$, derive a sufficient statistic and evaluate the likelihood.

The Poisson mass function gives $$ P(Y_i=y_i)=\frac{\lambda_i^{y_i}e^{-\lambda_i}}{y_i!} =\frac{e_i^{y_i}}{y_i!}\exp\{a_i y_i\theta-e_i e^{a_i\theta}\}. $$ Consequently, the joint likelihood is $$ L(\theta)= \left(\prod_{i=1}^3\frac{e_i^{y_i}}{y_i!}\right) \exp\!\left\{\theta\sum_{i=1}^3a_i y_i-\sum_{i=1}^3e_i e^{a_i\theta}\right\}. $$ The data-dependent term involving $\theta$ uses $T=\sum_i a_iY_i$; the exposures affect the likelihood's parameter-only term but not this statistic. For these data, $T=1(2)+2(1)+1(0)=4$. The remaining data factor is $$ \prod_{i=1}^3\frac{e_i^{y_i}}{y_i!} =\frac{1^2}{2!}\frac{2^1}{1!}\frac{4^0}{0!}=1, $$ and the parameter-only sum in the exponent is $1e^\theta+2e^{2\theta}+4e^\theta=5e^\theta+2e^{2\theta}$. Therefore $$ L(\theta)=\exp\{4\theta-5e^\theta-2e^{2\theta}\}. $$ The displayed likelihood confirms directly that the observations enter through the weighted sum $T=4$.

Several Parameters and Several Weighted Sums

With more than one unknown parameter, the same reasoning produces a vector of sufficient summaries. Consider independent observations with densities or mass functions $$ f_i(y_i;\boldsymbol{\beta})=h_i(y_i)\exp\!\left\{q_i y_i\,\mathbf{x}_i^{\mathsf T}\boldsymbol{\beta} -B_i\!\left(\mathbf{x}_i^{\mathsf T}\boldsymbol{\beta}\right)\right\}. $$ Here $\boldsymbol{\beta}$ is a $p$-dimensional parameter, $\mathbf{x}_i$ is a known $p$-vector, $q_i$ is a known scalar, and $B_i$ is a known normalizing function. Assume the support is independent of $\boldsymbol{\beta}$. The observation-specific functions $h_i$ and $B_i$ may differ.

Theorem: Under these conditions, the $p$-vector $$ \mathbf{T}=\sum_{i=1}^n q_i\mathbf{x}_iY_i $$ is sufficient for $\boldsymbol{\beta}$.

Proof. Independence and the specified form imply

\begin{aligned} f_{\boldsymbol{\beta}}(y_1,\ldots,y_n) &=\prod_{i=1}^n h_i(y_i) \exp\!\left\{q_i y_i\mathbf{x}_i^{\mathsf T}\boldsymbol{\beta} -B_i(\mathbf{x}_i^{\mathsf T}\boldsymbol{\beta})\right\}\\ &=\exp\!\left\{\boldsymbol{\beta}^{\mathsf T} \sum_{i=1}^n q_i\mathbf{x}_i y_i -\sum_{i=1}^n B_i(\mathbf{x}_i^{\mathsf T}\boldsymbol{\beta})\right\} \prod_{i=1}^n h_i(y_i). \end{aligned}

The first factor depends on the data only through $\sum_i q_i\mathbf{x}_i y_i$, and the second is parameter-free. The factorization criterion proves the assertion. $\square$

This result includes canonical exponential-family models in which each observation's natural parameter is $\mathbf{x}_i^{\mathsf T}\boldsymbol{\beta}$. When the known $q_i$ are all one, each coordinate of the summary adds the corresponding coordinate of $\mathbf{x}_i$, weighted by the observed response.

Worked Example: Two Poisson Regression Parameters

Let $Y_1,Y_2,Y_3$ be independent Poisson variables with means $\lambda_i=\exp(\beta_0+x_i\beta_1)$, where the known covariate values are $x=(0,1,2)$. The observed counts are $y=(1,3,2)$. Find the sufficient summaries and factor the likelihood.

For a Poisson observation, the log-likelihood contribution is $y_i(\beta_0+x_i\beta_1)-\exp(\beta_0+x_i\beta_1)-\log(y_i!)$. Summing the coefficients of the two unknown parameters gives $$ T_0=\sum_i y_i=1+3+2=6,\qquad T_1=\sum_i x_i y_i=0(1)+1(3)+2(2)=7. $$ Thus the data enter the parameter-dependent linear term through $(T_0,T_1)=(6,7)$. The full likelihood is $$ \begin{aligned} L(\beta_0,\beta_1) &=\frac{1}{1!\,3!\,2!} \exp\!\left\{6\beta_0+7\beta_1 -\sum_{i=1}^3e^{\beta_0+x_i\beta_1}\right\}\\ &=\frac{1}{12} \exp\!\left\{6\beta_0+7\beta_1 -e^{\beta_0}(1+e^{\beta_1}+e^{2\beta_1})\right\}. \end{aligned} $$ The factorization shows that the pair $(\sum_iY_i,\sum_i x_iY_i)$ is sufficient for $(\beta_0,\beta_1)$. The individual counts need not be identical in distribution: the means differ because the known covariate values differ.

What the Summaries Do—and Do Not—Mean

A sufficient statistic preserves the dependence of the likelihood on the unknown parameter; it does not say that the observations are interchangeable or have the same distribution. In the examples, the known design quantities determine how each response contributes to the summary, while the other likelihood factors may still depend on those design quantities.

The design quantities must be known for this argument. If a coefficient or exposure is unknown, it cannot simply be treated as a fixed weight in the statistic. Likewise, if the support of the observations depends on the unknown parameter, the support restriction must be included in the likelihood factorization; the results above explicitly assume parameter-independent support.

The same logic applies to ordinary sums as a special case. If the parameter enters every observation with the same coefficient, the sufficient weighted sum reduces to a multiple of $\sum_iY_i$. When the coefficients differ, replacing $\sum_i a_iY_i$ by the unweighted sum generally discards how the responses align with the known design values.

Common Mistake

Wrong reasoning: “The observations are independent, so their sum is sufficient.” Independence only makes the joint likelihood a product; it does not guarantee that this product depends on the data through the unweighted sum. In the Bernoulli example, $\theta$ multiplies $y_1+2y_2+3y_3$, not $y_1+y_2+y_3$. The correction is to write the joint likelihood and collect the coefficients of the unknown parameter. The resulting statistic is the data summary that appears in the parameter-dependent factor.

Key takeaway: For independent, nonidentically distributed observations with fixed known design quantities, derive sufficiency from the product likelihood. In canonical exponential-family forms, the parameter-dependent linear term reveals the sufficient summary: usually a weighted sum, or a vector of weighted sums when there are several parameters.

Check Your Understanding

Use the likelihood factorization, rather than independence alone, to answer these questions.

In the scalar model above, what sufficient statistic results when every $a_i=1$?
For independent Bernoulli observations with $\operatorname{logit}(p_i)=a_i\theta+c_i$, explain why the offsets $c_i$ do not change the weighted statistic.
Suppose independent Poisson counts have means $\lambda_i=e_i e^{a_i\theta}$, with known $e_i$ and $a_i$. What statistic carries the data dependence involving $\theta$?
In the two-parameter Poisson example, calculate the sufficient pair if the observed counts change to $(2,0,1)$, keeping $x=(0,1,2)$.
Why is independence alone not enough to conclude that the unweighted sample sum is sufficient?