Mean and Variance of Mixtures

Why Mixture Moments Need Two Sources of Variation

A mixture distribution describes a population whose members may come from different component distributions. For instance, patients might enter one of several treatment groups, or events might arise under one of several operating conditions. Knowing the average variability inside each group does not, by itself, determine the variability across the combined population. The component means may also differ, adding variation between groups.

A latent component label makes this distinction precise. First a component is selected according to its weight; then an observation is drawn from that component’s distribution. The resulting mixture has a mean equal to the weighted average of component means. Its variance includes both the average component variance and the variance of the component means under the weights.

Definition: Let \(Z\) take values \(1,\ldots,k\), with \(P(Z=j)=p_j\), where \(p_j\geq 0\) and \(\sum_{j=1}^k p_j=1\). Given \(Z=j\), let \(X\) have component distribution \(F_j\). The distribution of \(X\) is the finite mixture with weights \(p_1,\ldots,p_k\) and component distributions \(F_1,\ldots,F_k\). For each component with \(p_j>0\), write \(m_j=E[X\mid Z=j]\) and \(v_j=\operatorname{Var}(X\mid Z=j)\), when these moments exist.

The label \(Z\) is a device for describing how the mixture is formed; it need not be observed. The weights are probabilities, not arbitrary multipliers. Components with weight zero have no effect on the mixture and can be left out of moment calculations.

The Mixture Mean and Variance

For the mean, assume \(E[|X|]<\infty\), so the component means and their weighted sum are well-defined. For the variance decomposition below, assume \(E[X^2]<\infty\). With finitely many components, this is equivalent to every component with positive weight having a finite second moment.

Theorem: For a finite mixture with \(E[|X|]<\infty\), its mean is

E[X]=\sum_{j=1}^k p_jm_j.

If, in addition, \(E[X^2]<\infty\), its variance is

\operatorname{Var}(X) = \underbrace{\sum_{j=1}^k p_jv_j}_{\text{within-component variance}} + \underbrace{\sum_{j=1}^k p_j(m_j-\mu)^2}_{\text{between-component variance}}, \qquad \mu=\sum_{j=1}^k p_jm_j.

The first term averages the component variances using the mixture weights. It measures variation remaining among observations from the same component. The second term is the weighted variance of the component means around the overall mixture mean. It measures how much component membership separates observations into groups with different centers.

Proof of the Mean and Variance Formulas

For any integrable function \(g\), conditioning on the finite-valued label partitions the expectation across components: \(E[g(X)]=\sum_{j=1}^k p_j E[g(X)\mid Z=j]\). Taking \(g(X)=X\) gives the mean formula directly. To prove the variance formula, let \(\mu=E[X]\), and within component \(j\) write \(X-\mu=(X-m_j)+(m_j-\mu)\).

Expand within a component.
For each \(j\) with \(p_j>0\), square the centered observation and take its conditional expectation given \(Z=j\).

Use the component mean.
The conditional expectation of \(X-m_j\) is zero, so the cross-product term vanishes.

Average over component labels.
Weight the resulting conditional squared deviations by \(p_j\) and sum to obtain the total variance.

In detail, conditional on \(Z=j\), the cross-product has expectation \(2(m_j-\mu)E[X-m_j\mid Z=j]=0\). Therefore,

\begin{aligned} E[(X-\mu)^2\mid Z=j] &=E[(X-m_j)^2\mid Z=j] +2(m_j-\mu)E[X-m_j\mid Z=j] +(m_j-\mu)^2\\ &=v_j+(m_j-\mu)^2. \end{aligned}

Averaging these conditional expectations gives

\begin{aligned} \operatorname{Var}(X) &=E[(X-\mu)^2]\\ &=\sum_{j=1}^k p_j E[(X-\mu)^2\mid Z=j]\\ &=\sum_{j=1}^k p_jv_j+\sum_{j=1}^k p_j(m_j-\mu)^2. \end{aligned}

All quantities are finite under the assumed second-moment condition. This proves both formulas. \(\square\)

A related expression follows by expanding the definition of variance: \(\operatorname{Var}(X)=E[X^2]-\mu^2\). In a mixture, \(E[X^2]=\sum_j p_j(v_j+m_j^2)\), because \(E[X^2\mid Z=j]=v_j+m_j^2\). Hence the same variance can be calculated as

\operatorname{Var}(X)=\sum_{j=1}^k p_j(v_j+m_j^2) -\left(\sum_{j=1}^k p_jm_j\right)^2.

This raw-moment formula is convenient for checking calculations. The decomposition is usually more informative because it displays the two distinct sources of variation.

Worked Mixture Calculations

Worked Example: A Mixture of Bernoulli Components

Let \(P(Z=1)=0.7\) and \(P(Z=2)=0.3\). Conditional on \(Z=1\), let \(X\sim\operatorname{Bernoulli}(0.2)\); conditional on \(Z=2\), let \(X\sim\operatorname{Bernoulli}(0.8)\). A Bernoulli variable with success probability \(q\) has mean \(q\) and variance \(q(1-q)\).

The component means are \(m_1=0.2\) and \(m_2=0.8\). Thus the mixture mean is

\mu=0.7(0.2)+0.3(0.8)=0.14+0.24=0.38.

The component variances are \(v_1=0.2(0.8)=0.16\) and \(v_2=0.8(0.2)=0.16\), so the within-component term is

$$ 0.7(0.16)+0.3(0.16)=0.112+0.048=0.16. $$

The between-component term is

$$ 0.7(0.2-0.38)^2+0.3(0.8-0.38)^2 =0.7(0.0324)+0.3(0.1764) =0.02268+0.05292=0.0756. $$

Therefore \(\operatorname{Var}(X)=0.16+0.0756=0.2356\). As a check, \(X\) can only be zero or one, and its overall success probability is \(0.38\). Its variance is consequently \(0.38(0.62)=0.2356\), matching the decomposition.

Worked Example: Three Components with Different Means and Variances

Suppose the component weights are \(0.2,0.5,0.3\). Component 1 assigns probability \(1/2\) to each of \(1\) and \(3\); component 2 assigns probability \(1/2\) to each of \(3\) and \(7\); component 3 assigns probability \(1/2\) to each of \(7\) and \(9\). Their means are \(2,5,8\), and their variances are \(1,4,1\), respectively. For example, component 2 has variance \(\{(3-5)^2+(7-5)^2\}/2=4\).

The overall mean is

\mu=0.2(2)+0.5(5)+0.3(8)=0.4+2.5+2.4=5.3.

The within-component contribution is \(0.2(1)+0.5(4)+0.3(1)=0.2+2+0.3=2.5\). The between-component contribution is

\begin{aligned} &0.2(2-5.3)^2+0.5(5-5.3)^2+0.3(8-5.3)^2\\ &=0.2(10.89)+0.5(0.09)+0.3(7.29)\\ &=2.178+0.045+2.187=4.41. \end{aligned}

Thus \(\operatorname{Var}(X)=2.5+4.41=6.91\). A raw-moment check gives component second moments \(5,29,65\), since each equals its variance plus the square of its mean. Their weighted average is \(0.2(5)+0.5(29)+0.3(65)=1+14.5+19.5=35\). Subtracting \(\mu^2=5.3^2=28.09\) gives \(35-28.09=6.91\).

Worked Example: A Two-Rate Poisson Mixture

Let \(P(Z=1)=0.25\) and \(P(Z=2)=0.75\). Given \(Z=1\), let \(X\) have a Poisson distribution with parameter \(2\); given \(Z=2\), let \(X\) have a Poisson distribution with parameter \(8\). A Poisson distribution with parameter \(\lambda\) has mean and variance both equal to \(\lambda\).

The mean of the mixture is

\mu=0.25(2)+0.75(8)=0.5+6=6.5.

The within-component contribution is \(0.25(2)+0.75(8)=6.5\). The between-component contribution is

$$ 0.25(2-6.5)^2+0.75(8-6.5)^2 =0.25(20.25)+0.75(2.25) =5.0625+1.6875=6.75. $$

The mixture variance is \(6.5+6.75=13.25\). In particular, it is greater than the mixture mean, even though each component separately has variance equal to its mean. The difference comes from the spread between the two component means. For a numerical check, the component second moments are \(2+2^2=6\) and \(8+8^2=72\). The mixture second moment is \(0.25(6)+0.75(72)=1.5+54=55.5\), so the variance is \(55.5-6.5^2=55.5-42.25=13.25\).

Interpretation and a Common Pitfall

The decomposition shows that the total variance is at least the weighted average of the component variances, because the between-component term is a sum of nonnegative terms. Equality holds exactly when all component means with positive weight equal the overall mean. If the means differ, mixing them adds variance, even if the components themselves are individually tightly concentrated.

For continuous mixing, the same principle uses the random mixing label in place of a finite sum: the mean is the expectation of the conditional mean, and the variance is the expectation of the conditional variance plus the variance of the conditional mean, provided the relevant moments are finite. The finite formulas here are those conditional-moment identities written as weighted sums.

Wrong reasoning: “The mixture variance is the weighted average of the component variances.” This drops all variation due to differences among component means.
Correction: Add the weighted variance of the component means around the mixture mean. In the Poisson example, averaging the component variances gives \(6.5\), but the full variance is \(13.25\) because the between-component term is \(6.75\).
Another check: A variance cannot be negative, and the between-component contribution cannot be negative. If a calculation violates either fact, recheck the weights, component moments, or centering value.

The weighted average of component means gives the correct mixture mean, but the variance formula has an additional term because a mixture can vary both within components and across their centers. Keeping those sources separate is useful for describing heterogeneity and for checking any numerical calculation of mixture moments.

Check Your Understanding

Use the component weights and moments to answer each question.

A mixture has weights \(0.4\) and \(0.6\), with component means \(2\) and \(7\). Calculate its mean.
For the mixture in Question 1, suppose the component variances are \(3\) and \(5\). Calculate the within-component and between-component contributions, then find the total variance.
Under what condition is the mixture variance equal to the weighted average of component variances?
A mixture of two Poisson components has positive weights and different rate parameters. Explain why its variance exceeds the weighted average of its component variances.
What error is made by using only \(\sum_j p_jv_j\) as the mixture variance?

What You'll Learn