Tutorials › Exam P - Probability › Mean and Variance Decomposition in Mixture Models

Home  /  Tutorials  /  Exam P / Conditional Expectation and Decomposition

Exam P · Conditional Expectation and Decomposition · Tutorial 783 of 915

Mean and Variance Decomposition in Mixture Models

Combine category-specific means and variances correctly by separating variation within categories from variation between their means.

Advanced 9 min read

What You'll Learn

  • Define a mixture model using a latent categorical variable and category probabilities.
  • Calculate an overall mean as a probability-weighted average of conditional means.
  • Decompose total variance into within-category and between-category components.
  • Derive the decomposition by expanding squared deviations from the overall mean.
  • Check mixture calculations using conditional second moments.
  • Recognize why averaging conditional variances alone understates mixture variance.

Why Mixtures Need Two Sources of Variation

A population may combine distinct types of cases even when the type is not observed until after a case is selected. For example, a hypothetical insurer might receive claims from several risk categories, each with its own average claim amount and variability. The overall claim distribution is then a mixture of the category-specific distributions.

The mean of the mixture reflects the category means and how likely each category is. Its variance reflects more than the average variability within categories: it also reflects how far the category means lie from one another. Keeping these two sources separate is the central calculation in this topic.

Definition: Let \(C\) be a latent categorical variable taking values \(1,\ldots,k\), with \(P(C=j)=p_j\), where \(p_j\geq 0\) and \(\sum_{j=1}^{k}p_j=1\). A random variable \(X\) has a mixture distribution when its conditional distribution depends on the selected category \(C\). Write \(\mu_j=E[X\mid C=j]\) and \(\sigma_j^2=\operatorname{Var}(X\mid C=j)\). Assume the conditional means and variances are finite for every category with positive probability.

The category is selected according to the probabilities \(p_j\), and then \(X\) is generated according to the distribution for that category. The categories need not be equally likely, and their conditional distributions need not belong to the same distribution family. The mixture formulas require only the category probabilities and the conditional first and second moments.

The Mean and Variance Formulas

The overall mean is the weighted average of the conditional means. The overall variance has a within-category part and a between-category part:

$$ E[X]=\sum_{j=1}^{k}p_j\mu_j $$
$$ \operatorname{Var}(X) = \underbrace{\sum_{j=1}^{k}p_j\sigma_j^2}_{\text{within-category variance}} + \underbrace{\sum_{j=1}^{k}p_j(\mu_j-E[X])^2}_{\text{between-category variance}}. $$

The within-category term averages the conditional variances using the category probabilities. The between-category term measures the weighted spread of the conditional means around the overall mean. Both terms are nonnegative. In particular, the overall variance is at least the probability-weighted average of the conditional variances.

Theorem: For a finite-variance mixture, \(E[X]=\sum_j p_j\mu_j\) and \(\operatorname{Var}(X)=\sum_j p_j\sigma_j^2+\sum_jp_j(\mu_j-E[X])^2\). The first term is variation remaining within a category; the second is variation attributable to differences between category means.

Why the Decomposition Holds

Let \(\mu=E[X]\). Within category \(j\), add and subtract \(\mu_j\) from the deviation \(X-\mu\). Squaring creates a within-category squared deviation, a between-category squared deviation, and a cross term:

$$ (X-\mu)^2 = \bigl((X-\mu_j)+(\mu_j-\mu)\bigr)^2 = (X-\mu_j)^2+2(X-\mu_j)(\mu_j-\mu)+(\mu_j-\mu)^2. $$

Now take the conditional expectation given \(C=j\). The conditional mean of \(X-\mu_j\) is zero, so the cross term vanishes. The conditional expectation of \((X-\mu_j)^2\) is \(\sigma_j^2\), while \(\mu_j-\mu\) is constant within category \(j\). Thus:

$$ E[(X-\mu)^2\mid C=j]=\sigma_j^2+(\mu_j-\mu)^2. $$

Averaging this identity over categories weights each conditional expectation by its category probability. Since variance is \(E[(X-\mu)^2]\), the stated formula follows. This argument also explains why averaging only the conditional variances is incomplete: it omits the squared differences between category means.

For two categories with probabilities \(p\) and \(1-p\), the between-category term can be written more compactly. If the conditional means are \(\mu_1\) and \(\mu_2\), then:

$$ \sum_{j=1}^{2}p_j(\mu_j-\mu)^2 = p(1-p)(\mu_1-\mu_2)^2. $$

To see this, the overall mean is \(\mu=p\mu_1+(1-p)\mu_2\). Therefore, \(\mu_1-\mu=(1-p)(\mu_1-\mu_2)\) and \(\mu_2-\mu=-p(\mu_1-\mu_2)\). Substituting these two deviations gives \(p(1-p)^2(\mu_1-\mu_2)^2+(1-p)p^2(\mu_1-\mu_2)^2\), which simplifies to the formula above.

Worked Example: A Three-Category Claim Mixture

A hypothetical portfolio selects one of three claim categories. The category probabilities are \(0.50\), \(0.30\), and \(0.20\). Conditional on the selected category, the claim amount \(X\), measured in thousands of dollars, has the following means and variances:

  • Category 1: conditional mean \(100\), conditional variance \(400\).
  • Category 2: conditional mean \(200\), conditional variance \(900\).
  • Category 3: conditional mean \(400\), conditional variance \(1{,}600\).

Find the overall mean and variance. The method is to calculate the weighted mean first, then add the within-category and between-category variance terms. This avoids the common mistake of treating the weighted conditional variance as the entire variance.

Step 1: Check the weights. They are nonnegative and sum to \(0.50+0.30+0.20=1.00\), so they form a probability distribution over the categories.

Step 2: Calculate the overall mean.

$$ E[X]=0.50(100)+0.30(200)+0.20(400)=50+60+80=190. $$

The overall mean claim amount is \(190\) thousand dollars.

Step 3: Calculate within-category variance.

$$ 0.50(400)+0.30(900)+0.20(1{,}600)=200+270+320=790. $$

Step 4: Calculate between-category variance. The category means differ from the overall mean \(190\) by \(-90\), \(10\), and \(210\), respectively. Thus:

$$ 0.50(-90)^2+0.30(10)^2+0.20(210)^2 = 0.50(8{,}100)+0.30(100)+0.20(44{,}100) = 4{,}050+30+8{,}820=12{,}900. $$

Step 5: Add the two components.

$$ \operatorname{Var}(X)=790+12{,}900=13{,}690. $$

The overall variance is \(13{,}690\) thousand-dollars-squared. As a check, the conditional second moments are \(400+100^2=10{,}400\), \(900+200^2=40{,}900\), and \(1{,}600+400^2=161{,}600\). Their weighted average is \(0.50(10{,}400)+0.30(40{,}900)+0.20(161{,}600)=49{,}790\). Subtracting the squared overall mean gives \(49{,}790-190^2=49{,}790-36{,}100=13{,}690\), as required. The wrong calculation \(790\) would omit the substantial variation between category means.

Applying the Formula Carefully

The category probabilities are essential weights in both parts of the variance formula. A rare category with a mean far from the overall mean can matter greatly, because its between-category contribution includes a squared difference. However, its effect is still multiplied by its probability.

The variance terms also have units squared. If \(X\) is measured in thousands of dollars, then its means are in thousands of dollars and its variances are in thousands-of-dollars-squared. The mean and variance should not be added to each other; only the two variance components are added.

Worked Example: Separating Within- and Between-Category Variation

A hypothetical equipment-repair program classifies each service request as routine with probability \(0.25\) or complex with probability \(0.75\). The repair cost \(X\), in thousands of dollars, has conditional mean \(12\) and conditional variance \(16\) for routine requests, and conditional mean \(4\) and conditional variance \(4\) for complex requests. Calculate the overall mean and variance, and identify how much of the variance comes from differences between category means.

Step 1: Calculate the mean.

$$ E[X]=0.25(12)+0.75(4)=3+3=6. $$

Step 2: Calculate the within-category component.

$$ 0.25(16)+0.75(4)=4+3=7. $$

Step 3: Calculate the between-category component. The deviations of the conditional means from \(6\) are \(12-6=6\) and \(4-6=-2\). Therefore:

$$ 0.25(6)^2+0.75(-2)^2=0.25(36)+0.75(4)=9+3=12. $$

Step 4: Combine the components.

$$ \operatorname{Var}(X)=7+12=19. $$

The overall mean is \(6\) thousand dollars, the overall variance is \(19\) thousands-of-dollars-squared, and \(12\) of the variance units come from the difference between category means. As a check, the conditional second moments are \(16+12^2=160\) and \(4+4^2=20\). Their weighted average is \(0.25(160)+0.75(20)=40+15=55\), and \(55-6^2=19\). Using \(7\) as the total variance would incorrectly discard the between-category contribution.

Worked Example: A Mixture with Three Waiting-Time Groups

A hypothetical service center assigns each incoming request to one of three latent workload groups. The probabilities are \(0.20\), \(0.50\), and \(0.30\). Conditional on the group, the waiting time \(X\), in minutes, has means \(10\), \(14\), and \(20\), and variances \(4\), \(9\), and \(16\), respectively. Find the overall mean, variance, and standard deviation. Do not round until the final step.

Step 1: Verify the mixture weights and calculate the mean. The probabilities sum to \(0.20+0.50+0.30=1.00\), and:

$$ E[X]=0.20(10)+0.50(14)+0.30(20)=2+7+6=15\text{ minutes}. $$

Step 2: Calculate the within-category component.

$$ 0.20(4)+0.50(9)+0.30(16)=0.8+4.5+4.8=10.1. $$

Step 3: Calculate the between-category component. The conditional-mean deviations from \(15\) are \(-5\), \(-1\), and \(5\):

$$ 0.20(-5)^2+0.50(-1)^2+0.30(5)^2 = 5+0.5+7.5=13. $$

Step 4: Find the overall variance and standard deviation.

$$ \operatorname{Var}(X)=10.1+13=23.1\text{ minutes}^2, \qquad \sqrt{\operatorname{Var}(X)}=\sqrt{23.1}\approx 4.806\text{ minutes}. $$

The mean and variance are exact as given; the standard deviation is rounded to three decimal places. A second-moment check gives \(0.20(4+10^2)+0.50(9+14^2)+0.30(16+20^2)=0.20(104)+0.50(205)+0.30(416)=20.8+102.5+124.8=248.1\). Then \(248.1-15^2=248.1-225=23.1\), confirming the variance.

Common Exam Traps

  • Using only the average conditional variance. This gives the within-category component, not the overall variance. Add the between-category component as well.
  • Weighting the category means incorrectly. Use the probabilities of the latent categories, not equal weights unless the probabilities actually are equal.
  • Centering on the wrong value. In the between-category term, subtract the overall mean \(E[X]\) from each conditional mean \(\mu_j\), not a conditional mean from another category.
  • Forgetting to square deviations. The between-category contribution uses \(p_j(\mu_j-E[X])^2\). A signed weighted deviation would not measure variance.
  • Mixing units. Conditional variances and the overall variance are in squared units of \(X\); conditional and overall means are in the original units.
Key takeaway: In a mixture, calculate the weighted conditional mean first. Then add the probability-weighted conditional variances to the probability-weighted squared deviations of conditional means from that overall mean.

Check Your Understanding

For each problem, use the category probabilities as mixture weights and include both variance components.

  1. A two-category mixture has probabilities \(0.40\) and \(0.60\), conditional means \(5\) and \(9\), and conditional variances \(1\) and \(4\). Find the overall mean and variance. (Answer: mean \(7.4\), variance \(6.64\).)
  2. A hypothetical two-category payment has probabilities \(0.25\) and \(0.75\), conditional means \(20\) and \(8\), and conditional variances \(9\) and \(16\). Find the overall mean and variance. (Answer: mean \(11\), variance \(41.25\).)
  3. A three-group mixture has probabilities \(0.20\), \(0.30\), and \(0.50\), conditional means \(2\), \(6\), and \(10\), and conditional variances \(1\), \(4\), and \(9\). Find the overall mean and variance. (Answer: mean \(7.2\), variance \(15.66\).)
  4. A two-category mixture has conditional means \(3\) and \(11\), conditional variances both equal to \(4\), and overall mean \(7\). Find the probability of the category with mean \(3\), then find the overall variance. (Answer: probability \(0.50\), variance \(20\).)
  5. A three-category mixture has probabilities \(0.10\), \(0.40\), and \(0.50\), conditional means \(30\), \(50\), and \(70\), and conditional variances \(25\), \(100\), and \(225\). Find the overall mean and variance. (Answer: mean \(58\), variance \(331\).)

Previous

Next

Back to Tutorials
Clinical Biostats