Mixtures of Exponential Families Need Not Be Exponential

Why Mixing Changes the Form

A finite mixture describes a population whose members come from one of several component distributions. The component distributions may each have an exponential-family representation, yet averaging their densities usually changes the algebraic structure: the sum of exponentials appears inside a logarithm. That distinction matters whenever a model is chosen for its canonical representation, not merely because its component distributions have one.

We will use “exponential family” in the common-support sense. This keeps the comparison meaningful: the carrier measure and statistic are fixed across the parameter values being represented.

Definition: A family of densities or mass functions is a finite-dimensional exponential family with fixed support if, relative to a common dominating measure, its members can be written as $$ f_\eta(x)=h(x)\exp\{\eta^\mathsf{T}T(x)-A(\eta)\}, $$ where $h$ and the finite-dimensional statistic $T$ do not depend on $\eta$, and $A(\eta)$ normalizes the density or mass function. The natural parameter $\eta$ ranges over a specified set for which the expression is integrable. A finite mixture has weights $\pi_j\geq 0$, $\sum_{j=1}^m\pi_j=1$, and density $f(x)=\sum_{j=1}^m\pi_j f_j(x)$, all relative to the same dominating measure.

The Mixture Formula

Suppose each component belongs to the same exponential family, with natural parameter $\eta_j$. Substitution into the definition gives the mixture density directly.

Result: If $f_j(x)=h(x)\exp\{\eta_j^\mathsf{T}T(x)-A(\eta_j)\}$ for $j=1,\ldots,m$, then $$ f(x)=h(x)\sum_{j=1}^m\pi_j \exp\{\eta_j^\mathsf{T}T(x)-A(\eta_j)\}. $$ Where $h(x)>0$, its log density is $$ \log f(x)=\log h(x)+ \log\left(\sum_{j=1}^m\pi_j \exp\{\eta_j^\mathsf{T}T(x)-A(\eta_j)\}\right). $$

Derivation. By the definition of a mixture, $f(x)=\sum_j\pi_j f_j(x)$. Replacing each $f_j(x)$ by its exponential-family expression and factoring out the common $h(x)$ gives the first equation. Taking logarithms gives the second. The logarithm of a sum of exponentials is generally not an affine function of $T(x)$. Thus, the mixture generally does not retain the original family’s canonical form with the same statistic. The word “generally” is important: special choices of components and weights can produce a member of the family, as one example below shows.

Worked Example: A Mixture of Two Exponential Distributions

Let $X$ have density $e^{-x}$ for $x\geq 0$ with probability $1/2$, and density $3e^{-3x}$ for $x\geq 0$ with probability $1/2$. Find the mixture density and show that it is not an exponential distribution.

Both component densities use the same support $[0,\infty)$, so the mixture density is $$ f(x)=\frac12 e^{-x}+\frac12(3e^{-3x}) =\frac12e^{-x}+\frac32e^{-3x},\qquad x\geq 0. $$ Each component integrates to one, so the mixture does too: $$ \int_0^\infty f(x)\,dx =\frac12(1)+\frac32\left(\frac13\right)=1. $$

An exponential distribution has a constant hazard rate, where the hazard is $f(x)/S(x)$ and $S(x)=P(X>x)$ is the survival function. Here, $$ S(x)=\frac12e^{-x}+\frac12e^{-3x}, \qquad \frac{f(x)}{S(x)} =\frac{e^{-x}+3e^{-3x}}{e^{-x}+e^{-3x}}. $$ At $x=0$, this hazard is $(1+3)/(1+1)=2$. As $x$ tends to infinity, dividing numerator and denominator by $e^{-x}$ shows that it tends to $1$. Since the hazard is not constant, the mixture is not exponential.

Worked Example: A Symmetric Normal Mixture

Let $X$ be an equal mixture of $N(-2,1)$ and $N(2,1)$. Show explicitly how the density differs from a single normal density.

Writing $\phi(x)=(2\pi)^{-1/2}e^{-x^2/2}$ for the standard normal density, the mixture density is $$ f(x)=\frac12\phi(x+2)+\frac12\phi(x-2). $$ Since $\phi(x\mp2)=\phi(x)e^{\pm 2x-2}$, we can also write $$ f(x)=\phi(x)e^{-2}\frac{e^{2x}+e^{-2x}}{2} =\phi(x)e^{-2}\cosh(2x). $$ Thus its log density includes $\log\cosh(2x)$, rather than just the quadratic expression in $x$ that appears in a normal log density.

There is also a direct shape check. The density is symmetric, so its derivative at zero is zero. For a normal density with mean $\mu$ and variance one, the second derivative at $x$ is $((x-\mu)^2-1)\phi(x-\mu)$. Therefore, $$ f''(0)=\frac12(4-1)\phi(2)+\frac12(4-1)\phi(-2) =3\phi(2)>0. $$ The density has a local minimum at zero. A nondegenerate normal density has a single maximum and cannot have a local minimum at its center. The mixture is not normal.

A Mixture That Does Remain in the Family

Non-closure is not a claim that every mixture leaves every exponential family. For some component families, averaging the component probabilities simply changes the parameter.

Worked Example: Mixing Bernoulli Distributions

Choose a component with distribution $\operatorname{Bernoulli}(0.2)$ with probability $0.4$, and a component with distribution $\operatorname{Bernoulli}(0.7)$ with probability $0.6$. Find the distribution of the resulting binary observation $X$.

The probability of success is $$ P(X=1)=0.4(0.2)+0.6(0.7)=0.08+0.42=0.50. $$ The probability of failure is $$ P(X=0)=0.4(0.8)+0.6(0.3)=0.32+0.18=0.50. $$ These probabilities sum to one, and the mixture is $\operatorname{Bernoulli}(0.5)$. The Bernoulli family is therefore closed under this mixing operation. More generally, mixing Bernoulli component probabilities $p_j$ with weights $\pi_j$ gives a Bernoulli distribution with success probability $\sum_j\pi_jp_j$.

A Stronger Non-Closure Result for Poisson Mixtures

A single mixture that is not Poisson shows that Poisson distributions are not closed under mixing. We can make a stronger statement: as the mixing weight varies, these mixtures cannot all be represented by any one finite-dimensional exponential family with fixed support, carrier, and statistic. This rules out a different finite collection of canonical statistics rescuing the whole family of mixtures.

Worked Example: The Mass Function of a Two-Poisson Mixture

Let $X$ be $\operatorname{Poisson}(1)$ with probability $w$, and $\operatorname{Poisson}(2)$ with probability $1-w$, where $0<w<1$. For $x=0,1,2,\ldots$, its mass function is $$ p_w(x)=w e^{-1}\frac{1}{x!} +(1-w)e^{-2}\frac{2^x}{x!} =\frac{1}{x!}\left(we^{-1}+(1-w)e^{-2}2^x\right). $$

The ratio of the two component masses at $x$ is $$ R_x=\frac{e^{-1}/x!}{e^{-2}2^x/x!}=e\,2^{-x}. $$ Consequently $p_w(x)=p_2(x)\{(1-w)+wR_x\}$, where $p_2(x)=e^{-2}2^x/x!$ is the mass function of $\operatorname{Poisson}(2)$. The log mass contains $\log\{(1-w)+wR_x\}$, a log-sum term that depends on $x$ in a way that is not linear in the Poisson statistic $x$.

For example, at $w=1/2$, the masses at zero and one are $$ p_{1/2}(0)=\frac12(e^{-1}+e^{-2}),\qquad p_{1/2}(1)=\frac12(e^{-1}+2e^{-2}). $$ The mean is $w(1)+(1-w)(2)=1.5$, but the mass function is not Poisson with that mean: a $\operatorname{Poisson}(1.5)$ law has mass $e^{-1.5}$ at zero, which is not $\frac12(e^{-1}+e^{-2})$.

Theorem: The family $\{p_w:0<w<1\}$ of mixtures of $\operatorname{Poisson}(1)$ and $\operatorname{Poisson}(2)$ cannot be represented as a finite-dimensional exponential family with a fixed carrier and fixed statistic on $\{0,1,2,\ldots\}$.

Proof. Suppose, to the contrary, that such a representation exists with a statistic $T(x)$ taking values in $\mathbb{R}^d$. Then for each $w$, $$ \log p_w(x)=\log h(x)+\eta(w)^\mathsf{T}T(x)-A(\eta(w)). $$ Fix $w_0=1/2$ and form the log-mass differences $g_w(x)=\log p_w(x)-\log p_{w_0}(x)$. Every $g_w$ belongs to the finite-dimensional vector space $V$ spanned by the constant function $1$ and the $d$ coordinate functions of $T$; the carrier term cancels.

For a finite-dimensional space of functions on a countable set, pointwise limits of sequences in that space remain in the space. To see this, choose a basis $v_1,\ldots,v_r$ and points $x_1,\ldots,x_r$ such that the matrix with entries $v_i(x_j)$ is invertible. If a sequence of functions in the space converges at every $x$, its coefficients in this basis converge, because they are determined by its values at these $r$ points. The pointwise limit is therefore also a linear combination of the basis functions. Difference quotients of $g_w$, and their successive limits, must consequently belong to $V$.

Set $R_x=e\,2^{-x}$. Apart from the term $\log p_2(x)$, which does not depend on $w$, the log mass is $\log\{1+w(R_x-1)\}$. For every integer $k\geq1$, its $k$th derivative at $w_0$ is $$ (-1)^{k-1}(k-1)! \left(\frac{R_x-1}{1+w_0(R_x-1)}\right)^k. $$ These derivatives must all belong to $V$. But the values $$ q_x=\frac{R_x-1}{1+w_0(R_x-1)} $$ are infinitely many distinct real numbers: $R_x=e\,2^{-x}$ takes infinitely many distinct values, and the displayed transformation is strictly increasing in $R_x$ wherever its denominator is positive. It is positive in the denominator here because $1+\tfrac12(R_x-1)=(1+R_x)/2>0$.

The functions $x\mapsto q_x^k$, $k=1,2,\ldots$, are linearly independent. Indeed, if a finite linear combination vanished for all $x$, the polynomial $\sum_{k=1}^K c_k q^k$ would vanish at infinitely many distinct $q_x$. A nonzero polynomial has only finitely many roots, so every coefficient $c_k$ would have to be zero. Thus $V$ would contain infinitely many linearly independent functions, contradicting its finite dimension. The assumed representation cannot exist. $\square$

Reading the Result Correctly

The theorem concerns a whole family of mixture distributions and requires one fixed, finite-dimensional statistic and carrier for that family. It does not say that an individual distribution can never be written in an exponential form with a specially chosen statistic, or that no special mixture can remain in its component family. The Bernoulli calculation is one such exception.

A common mistake is to average the component parameters and assume that the mixture belongs to the family at the averaged parameter. In the Poisson example, the mean does average to $1.5$, but matching the mean does not establish equality of distributions. Check the mass function itself: the mixture’s probability at zero differs from that of $\operatorname{Poisson}(1.5)$. More generally, averaging canonical parameters does not turn a sum of exponential terms into a single exponential term.

When assessing a proposed mixture representation, first write the weighted sum of the component densities or masses. Then inspect whether its log can be expressed using a fixed finite list of statistics, with parameter dependence confined to their coefficients and the normalizing term. A log-sum-exp is a warning that this may fail, not by itself a proof of failure; a structural argument or a direct check, as above, completes the conclusion.

Check Your Understanding

Use the definitions and calculations in this tutorial to answer the following questions.

Write the density of a finite mixture when its components share a carrier function and canonical statistic. Which part of the resulting log density usually obstructs the original canonical form?
Why does the equal mixture of $N(-2,1)$ and $N(2,1)$ have a local minimum at zero, and why does that rule out a normal distribution?
Mix $\operatorname{Bernoulli}(0.3)$ and $\operatorname{Bernoulli}(0.8)$ with weights $0.25$ and $0.75$. What is the resulting success probability?
In the Poisson non-closure proof, why do infinitely many distinct values of $q_x$ imply that the functions $q_x^k$, viewed as functions of $x$, are linearly independent?
Does the Poisson theorem rule out every possible exponential representation for every individual Poisson mixture? Explain what family-level condition the theorem actually rules out.