Reproductive Closure Under Independent Sampling

Why Sums Need Not Stay in the Same Family

Independent sampling often turns a simple observation into a sum: a count of successes, a total number of events, or an accumulated measurement. The distribution of that sum is governed by the exponential-family form of each observation, but two different claims must be kept separate. The sum may have an exponential-family representation without belonging to the same named distribution family as one observation.

The key structural condition is that the canonical statistic adds across observations. This lets us combine the exponential factors directly. The base measure also changes: for a sum, it becomes a convolution of the one-observation base measure. Whether that new base measure has the form required by the original family determines whether the sum remains in that family.

Definition: Suppose $X$ has density or mass function $f_\eta(x)=h(x)\exp\{\eta^\mathsf{T}T(x)-A(\eta)\}$ with respect to a measure $\mu$, where $T$ is measurable and $A(\eta)$ is finite. If $X_1,\ldots,X_n$ are independent observations from this same distribution, their canonical statistic is additive when the statistic for the sample is $S_n=\sum_{i=1}^n T(X_i)$. A family is reproductive under summation for a specified class of sample sizes if the distribution of $\sum_i X_i$ remains in that family, possibly with transformed parameters.

The Distribution of an Additive Canonical Statistic

Here is the general result. Let $\eta$ be a fixed vector in the natural parameter space, and assume the observations are identically distributed as well as independent. Define a measure $\nu$ on the range of $T$ by $\nu(B)=\int_{\{x:T(x)\in B\}}h(x)\,\mu(dx)$ for measurable sets $B$. Assume $\nu$ is a well-defined, sigma-finite measure. For measures on the range of $T$, write $\nu^{*n}$ for the $n$-fold convolution: it is the distribution of a sum under the product measure $\nu^{\otimes n}$, without requiring that $\nu$ itself be a probability measure.

Theorem: If $X_1,\ldots,X_n$ are independent with common density or mass function $h(x)\exp\{\eta^\mathsf{T}T(x)-A(\eta)\}$ with respect to $\mu$, and $S_n=\sum_{i=1}^nT(X_i)$, then for every measurable set $B$, $$ P_\eta(S_n\in B)=\int_B \exp\{\eta^\mathsf{T}s-nA(\eta)\}\,\nu^{*n}(ds). $$ Thus $S_n$ has an exponential-family representation with canonical parameter $\eta$, base measure $\nu^{*n}$, and log normalizing constant $nA(\eta)$, relative to that base measure.

Proof. For a measurable set $B$, express the event $S_n\in B$ as the condition that the sum of the $n$ canonical-statistic values lies in $B$. Independence gives the product base measure for the $n$ observations. The exponential factors combine because the statistic is additive:

\begin{aligned} P_\eta(S_n\in B) &=\int \mathbf{1}\!\left\{\sum_{i=1}^nT(x_i)\in B\right\} \prod_{i=1}^n h(x_i) \exp\!\left\{\eta^\mathsf{T}\sum_{i=1}^nT(x_i)-nA(\eta)\right\} \,\mu^{\otimes n}(d x_1\cdots d x_n)\\ &=\int \mathbf{1}\!\left\{\sum_{i=1}^nt_i\in B\right\} \exp\!\left\{\eta^\mathsf{T}\sum_{i=1}^nt_i-nA(\eta)\right\} \,\nu^{\otimes n}(d t_1\cdots d t_n)\\ &=\int_B\exp\{\eta^\mathsf{T}s-nA(\eta)\}\,\nu^{*n}(ds). \end{aligned}

The second equality uses the definition of the pushforward measure $\nu$; the final equality is the definition of convolution. The integrands are nonnegative, so the product integrations and pushforward are valid by Tonelli's theorem. Taking $B$ to be the whole range shows that the displayed measure has total mass one. $\square$

The theorem describes the law of the sum statistic. When $T(x)=x$, it directly gives the distribution of the sum of the observations. For a different statistic, it gives the distribution of the sum of the statistic values, which need not be the same object as the sum of the original observations.

Worked Sums from Familiar Families

Worked Example: Bernoulli Observations Give a Binomial Count

Let $X_i$ be independent Bernoulli observations with success probability $p$, where $0<p<1$. Write $\eta=\log\{p/(1-p)\}$. The mass function can be written $f_\eta(x)=\exp\{\eta x-A(\eta)\}$ for $x\in\{0,1\}$, where $A(\eta)=\log(1+e^\eta)$. The canonical statistic is $T(x)=x$, so $S_n=\sum_iX_i$ counts the successes.

The one-observation base measure assigns mass one to each of 0 and 1. Its $n$-fold convolution assigns mass $\binom{n}{s}$ to each integer $s=0,\ldots,n$, since that many sequences of zeros and ones have sum $s$. Therefore

P(S_n=s)=\binom{n}{s}\exp\{\eta s-nA(\eta)\} =\binom{n}{s}p^s(1-p)^{n-s},\qquad s=0,\ldots,n.

The last equality follows from $e^\eta=p/(1-p)$ and $e^{A(\eta)}=1+e^\eta=1/(1-p)$. Thus the sum is binomial with size $n$ and success probability $p$. It is not Bernoulli when $n>1$, but it remains in the binomial family when the size parameter is allowed to change.

Worked Example: Poisson Sums Stay Poisson

Let $X_i$ be independent Poisson observations with mean $\lambda>0$. With $\eta=\log\lambda$, the mass function is $f_\eta(x)=\exp\{\eta x-e^\eta\}/x!$ for nonnegative integers $x$. Hence $T(x)=x$, $h(x)=1/x!$, and $A(\eta)=e^\eta$.

At a nonnegative integer $s$, the $n$-fold convolution of the base measure has mass

\begin{aligned} \nu^{*n}(\{s\}) &=\sum_{k_1+\cdots+k_n=s}\frac{1}{k_1!\cdots k_n!}\\ &=\frac{1}{s!}\sum_{k_1+\cdots+k_n=s} \binom{s}{k_1,\ldots,k_n} =\frac{n^s}{s!}. \end{aligned}

The multinomial coefficients in the sum count all assignments of $s$ labelled items to $n$ groups, so their sum is $n^s$. The theorem now gives

P(S_n=s)=e^{-ne^\eta}\frac{e^{\eta s}n^s}{s!} =\exp\{-n\lambda\}\frac{(n\lambda)^s}{s!}.

Therefore $S_n$ is Poisson with mean $n\lambda$. In canonical coordinates its parameter is $\eta+\log n$, since $e^{\eta+\log n}=n\lambda$.

Worked Example: Gamma Sums Preserve a Common Rate

Suppose $X_i$ are independent gamma observations with shape $\alpha>0$ and rate $\beta>0$, using the density $f(x)=\beta^\alpha x^{\alpha-1}e^{-\beta x}/\Gamma(\alpha)$ for $x>0$. For two observations, convolution gives, for $s>0$,

\begin{aligned} f_{X_1+X_2}(s) &=\frac{\beta^{2\alpha}e^{-\beta s}}{\Gamma(\alpha)^2} \int_0^s x^{\alpha-1}(s-x)^{\alpha-1}\,dx\\ &=\frac{\beta^{2\alpha}e^{-\beta s}s^{2\alpha-1}}{\Gamma(\alpha)^2} \int_0^1u^{\alpha-1}(1-u)^{\alpha-1}\,du\\ &=\frac{\beta^{2\alpha}s^{2\alpha-1}e^{-\beta s}}{\Gamma(2\alpha)}. \end{aligned}

In the second line, substitute $u=x/s$. The integral is the beta integral $B(\alpha,\alpha)=\Gamma(\alpha)^2/\Gamma(2\alpha)$, giving the final density. It is the gamma density with shape $2\alpha$ and rate $\beta$. Repeating the convolution argument with shapes $k\alpha$ and $\alpha$ proves by induction that $\sum_{i=1}^nX_i$ is gamma with shape $n\alpha$ and rate $\beta$.

This is closure within the two-parameter gamma family. If the family under consideration holds the shape fixed at $\alpha$ and varies only the rate, then the sum generally leaves that narrower family because its shape becomes $n\alpha$.

Worked Example: A Uniform Sum Is Not Uniform

Let $X_1,X_2$ be independent uniform observations on $(0,1)$. The density of $S=X_1+X_2$ is the length of the set of $x\in(0,1)$ for which $s-x\in(0,1)$. For $0<s<1$, this set has length $s$; for $1\le s<2$, it has length $2-s$. Thus

f_S(s)= \begin{cases} s,&0<s<1,\\ 2-s,&1\le s<2,\\ 0,&\text{otherwise}. \end{cases}

This triangular density is not uniform. Indeed, it is not constant on its support $(0,2)$. The uniform family is therefore not reproductive under summation, even though the sum has an exponential-family representation with the convolution of the uniform base measure.

When Is the Sum in the Original Family?

For a scalar observation whose canonical statistic is $T(x)=x$, write the family as $f_\eta(x)=e^{\eta x-A(\eta)}\nu(dx)$, with $\nu$ the base measure. The theorem says that the sum has base measure $\nu^{*n}$. A useful sufficient condition for the sum to use the original base measure again is an exponential change of that measure.

Closure criterion: Suppose that for a fixed $n$, the convolution base measure satisfies $\nu^{*n}(ds)=c_n e^{a_ns}\nu(ds)$, where $c_n>0$ and $a_n$ do not depend on $\eta$. If $\eta+a_n$ is in the natural parameter space, then the sum has density with respect to $\nu$ $$ c_n\exp\{(\eta+a_n)s-nA(\eta)\}. $$ When this density is normalized by the same family, its log normalizer must satisfy $A(\eta+a_n)=nA(\eta)-\log c_n$. Under these conditions, the sum belongs to the original family at canonical parameter $\eta+a_n$.

To verify the criterion, substitute the assumed expression for $\nu^{*n}$ into the theorem's formula. The resulting density is the displayed expression. Its integral is one because it is the probability law of the sum; the original family's density at $\eta+a_n$ is also normalized. Comparing their normalizing factors gives the stated identity. The condition is sufficient, not necessary for every possible parameterization: a distribution might match a family after a more general reparameterization or a change in family parameters such as binomial size or gamma shape.

Normal sums provide another familiar closure property: if $X_i\sim N(\mu,\sigma^2)$ independently, then their sum is $N(n\mu,n\sigma^2)$. For example, the moment generating function of the sum is the product of the individual moment generating functions:

\prod_{i=1}^n \exp\{\mu t+\tfrac12\sigma^2t^2\} =\exp\{n\mu t+\tfrac12n\sigma^2t^2\}.

This is the moment generating function of the stated normal law. The sum stays in the full normal family, but not in a subfamily that requires the variance to remain fixed at $\sigma^2$.

Common Mistake and Why the Distinction Matters

A common mistake is to reason, “The observations are exponential-family members and their canonical statistics add, so their sum must be from the same distribution family.” The theorem does not say this. It says the sum has an exponential factor involving the same canonical parameter, while the base measure changes from $\nu$ to $\nu^{*n}$. For the sum to remain in a particular family, that changed measure and the resulting support must also match the family's form, perhaps after a parameter update.

This distinction is useful in sampling theory and distribution calculations. It tells us exactly what independence and a shared parameter contribute, and it prevents an unjustified family label. For example, independent Bernoulli observations yield a binomial count, not a Bernoulli observation; gamma sums retain a common rate but change shape; and uniform sums need not be uniform. Also, identical distributions matter: independent observations with different canonical parameters contribute different terms in the exponent, and the common-parameter conclusion of the theorem does not apply as stated.

When assessing a proposed sum law, check the statistic, the base measure, the support, and which parameters the family allows to change. Additivity handles the exponential factor; it does not by itself establish reproductive closure.

Check Your Understanding

Use the theorem and examples to answer the following questions.

In the general theorem, which assumption allows the canonical exponential factors to combine into $\eta^\mathsf{T}S_n-nA(\eta)$?
For $n$ Bernoulli observations with success probability $p$, what is the distribution of their sum, and which parameter changes from the one-observation family?
Why does the sum of independent Poisson observations with common mean $\lambda$ have mean $n\lambda$?
For independent gamma observations with common shape $\alpha$ and rate $\beta$, which parameter is preserved and which is multiplied by $n$?
What feature of the density of the sum of two independent uniform $(0,1)$ observations shows that it is not uniform?