Tutorials › VEE Mathematical Statistics › Variance Decomposition in Mixture Models

Home  /  Tutorials  /  VEE Mathematical Statistics / Conditional Distributions and Expectations

VEE Mathematical Statistics · Conditional Distributions and Expectations · Tutorial 298 of 880

Variance Decomposition in Mixture Models

Separate a mixture’s total variance into within-group uncertainty and between-group differences in conditional means.

Intermediate 9 min read

What You'll Learn

  • State the conditions under which conditional group means and variances are defined.
  • Apply the law of total variance to a finite mixture.
  • Calculate within-group and between-group variance separately.
  • Verify a mixture variance using its marginal distribution or second moment.
  • Recognize why averaging conditional variances alone understates total variance.

Why Mixtures Have Two Sources of Variation

Suppose a portfolio contains several types of risks, or a study combines outcomes from distinct populations. An observation can vary because outcomes are uncertain within its group, and because the groups themselves may have different average outcomes. A variance calculation that considers only one of these sources will generally miss part of the total variability.

A mixture model describes this situation by first selecting a group and then drawing an outcome from that group’s conditional distribution. The law of total variance gives an exact way to separate the resulting variation. It is useful both for calculating a marginal variance and for understanding what drives that variance.

Definition: Let \(G\) be a group label taking values \(1,\ldots,k\), with \(P(G=j)=p_j\). Omit any group with \(p_j=0\); for each remaining group, \(p_j>0\). Suppose \(E[X^2]\) is finite, and define \(\mu_j=E[X\mid G=j]\), \(\sigma_j^2=\operatorname{Var}(X\mid G=j)\), and \(\mu=E[X]\). The within-group variance is \(\sum_{j=1}^k p_j\sigma_j^2\). The between-group variance is \(\sum_{j=1}^k p_j(\mu_j-\mu)^2\), the variance of the conditional means.

The positive-probability condition matters: a conditional mean or variance given an event of probability zero is not determined by ordinary conditioning on that event. Omitting zero-probability groups avoids assigning such groups conditional moments that are not defined.

The Law of Total Variance

The central result applies to any finite-variance random variable \(X\) and any finite-valued group label \(G\). It does not require normal conditional distributions, equal group sizes, or any particular shape for the mixture.

Theorem: Under the conditions above,
$$ \operatorname{Var}(X) = E[\operatorname{Var}(X\mid G)] + \operatorname{Var}(E[X\mid G]) = \sum_{j=1}^k p_j\sigma_j^2 + \sum_{j=1}^k p_j(\mu_j-\mu)^2. $$
The first term is within-group variance; the second is between-group variance.

The notation \(E[\operatorname{Var}(X\mid G)]\) means that the conditional variance is averaged using the group probabilities. Similarly, \(E[X\mid G]\) is itself a random variable: it equals \(\mu_j\) when \(G=j\). Its variance is therefore the weighted variance of the group means.

Why the Decomposition Holds

The key is to add and subtract the conditional mean. Write \(\mu_G=E[X\mid G]\). Then \(X-\mu=(X-\mu_G)+(\mu_G-\mu)\). When the square is expanded, the cross term has conditional expectation zero because \(E[X-\mu_G\mid G]=0\).

1
Center within each group.
For each possible value of \(G\), \(\mu_G\) is a constant and \(E[X-\mu_G\mid G]=0\) by the definition of conditional expectation.
2
Expand the squared deviation.
The identity \(X-\mu=(X-\mu_G)+(\mu_G-\mu)\) gives three terms after squaring.
3
Take expectations.
The cross term vanishes, leaving the average conditional squared deviation plus the squared deviation of the conditional mean.
$$ \begin{aligned} \operatorname{Var}(X) &=E[(X-\mu)^2]\\ &=E[(X-\mu_G)^2]+2E[(X-\mu_G)(\mu_G-\mu)] +E[(\mu_G-\mu)^2]\\ &=E[\operatorname{Var}(X\mid G)] +\operatorname{Var}(E[X\mid G]). \end{aligned} $$

For the last equality, condition the cross term on \(G\). Since \(\mu_G-\mu\) is determined once \(G\) is known,

$$ E[(X-\mu_G)(\mu_G-\mu)\mid G] =(\mu_G-\mu)E[X-\mu_G\mid G]=0. $$

Also, \(E[(X-\mu_G)^2\mid G]=\operatorname{Var}(X\mid G)\), and \(E[(\mu_G-\mu)^2]=\operatorname{Var}(\mu_G)\). Taking expectations proves the formula. This argument uses finite second moments so the variances and expectations in the identity exist.

Worked Examples

Worked Example: A Two-Group Bernoulli Mixture

A hypothetical insurer classifies policies into groups A and B. A randomly selected policy is in A with probability \(0.25\) and in B with probability \(0.75\). Let \(X=1\) if the policy has a claim during the year and \(X=0\) otherwise. Conditional on A, \(X\sim\operatorname{Bernoulli}(0.20)\); conditional on B, \(X\sim\operatorname{Bernoulli}(0.60)\). Find the overall variance of \(X\) and verify it from the marginal distribution. No interest rate or mortality assumption is involved.

Method. Use the law of total variance. For a Bernoulli random variable with success probability \(q\), the mean is \(q\) and the variance is \(q(1-q)\).

Step 1: Conditional means and variances. Group A has \(\mu_A=0.20\) and \(\sigma_A^2=0.20(0.80)=0.16\). Group B has \(\mu_B=0.60\) and \(\sigma_B^2=0.60(0.40)=0.24\).

Step 2: Overall mean.

$$ \mu=E[X]=0.25(0.20)+0.75(0.60)=0.05+0.45=0.50. $$

Step 3: Within-group variance.

$$ E[\operatorname{Var}(X\mid G)] =0.25(0.16)+0.75(0.24) =0.04+0.18=0.22. $$

Step 4: Between-group variance.

$$ \begin{aligned} \operatorname{Var}(E[X\mid G]) &=0.25(0.20-0.50)^2+0.75(0.60-0.50)^2\\ &=0.25(0.09)+0.75(0.01)=0.0225+0.0075=0.03. \end{aligned} $$

Thus \(\operatorname{Var}(X)=0.22+0.03=0.25\). To check, \(P(X=1)=0.25(0.20)+0.75(0.60)=0.50\), so marginally \(X\) is Bernoulli with variance \(0.50(0.50)=0.25\), as required. The common mistake is to report \(0.22\), the average conditional variance, as the total; that omits the \(0.03\) due to different group claim probabilities.

Worked Example: A Mixture of Normal Outcomes

For a hypothetical operational study, let \(X\) be an annual measurement in thousands of dollars. Group A occurs with probability \(0.60\), with \(X\mid G=A\sim N(10,4)\). Group B occurs with probability \(0.40\), with \(X\mid G=B\sim N(16,9)\). The second parameter is the variance, so the conditional standard deviations are 2 and 3 thousand dollars. Find the marginal mean and variance. No interest rate or mortality assumption is involved.

Method. Calculate the weighted conditional mean, then add the weighted conditional variances and the variance of the two conditional means. Do not assume the mixture itself is normal.

Step 1: Overall mean.

$$ \mu=0.60(10)+0.40(16)=6+6.4=12.4 $$

The overall mean is \(12.4\) thousand dollars.

Step 2: Within-group variance.

$$ E[\operatorname{Var}(X\mid G)] =0.60(4)+0.40(9)=2.4+3.6=6. $$

Step 3: Between-group variance.

$$ \begin{aligned} \operatorname{Var}(E[X\mid G]) &=0.60(10-12.4)^2+0.40(16-12.4)^2\\ &=0.60(5.76)+0.40(12.96)\\ &=3.456+5.184=8.64. \end{aligned} $$

Therefore \(\operatorname{Var}(X)=6+8.64=14.64\) in squared thousands of dollars. As a check, the conditional second moments are \(4+10^2=104\) and \(9+16^2=265\). Their weighted average is \(0.60(104)+0.40(265)=62.4+106=168.4\). Subtracting \(\mu^2=12.4^2=153.76\) gives \(168.4-153.76=14.64\). The calculation verifies the variance without treating the overall mixture as a single normal distribution.

Worked Example: Discrete Claim Payments in Two Segments

A hypothetical portfolio reports covered claim payments in thousands of dollars. A randomly selected claim belongs to segment A with probability \(1/3\) and segment B with probability \(2/3\). Conditional on A, the payment \(X\) is 0 or 12, each with probability \(1/2\). Conditional on B, \(X\) is 8 or 20, each with probability \(1/2\). Find the overall mean and variance, and check the variance using the second moment. There is no interest rate or mortality model in this payment calculation.

Method. First calculate each group’s mean and variance from its two possible payments. Then apply the decomposition. All payment amounts are in thousands of dollars.

Step 1: Group A moments.

$$ \mu_A=\frac{0+12}{2}=6,\qquad \sigma_A^2=\frac{(0-6)^2+(12-6)^2}{2} =\frac{36+36}{2}=36. $$

Step 2: Group B moments.

$$ \mu_B=\frac{8+20}{2}=14,\qquad \sigma_B^2=\frac{(8-14)^2+(20-14)^2}{2} =\frac{36+36}{2}=36. $$

Step 3: Overall mean and within-group variance.

$$ \mu=\frac13(6)+\frac23(14)=2+\frac{28}{3}=\frac{34}{3}, \qquad E[\operatorname{Var}(X\mid G)]=\frac13(36)+\frac23(36)=36. $$

Step 4: Between-group variance.

$$ \begin{aligned} \operatorname{Var}(E[X\mid G]) &=\frac13\left(6-\frac{34}{3}\right)^2 +\frac23\left(14-\frac{34}{3}\right)^2\\ &=\frac13\left(\frac{256}{9}\right) +\frac23\left(\frac{64}{9}\right) =\frac{256}{27}+\frac{128}{27} =\frac{128}{9}. \end{aligned} $$

Hence

$$ \operatorname{Var}(X)=36+\frac{128}{9} =\frac{324}{9}+\frac{128}{9} =\frac{452}{9}\approx 50.22 $$

in squared thousands of dollars. To check, \(E[X^2\mid A]=(0^2+12^2)/2=72\), and \(E[X^2\mid B]=(8^2+20^2)/2=(64+400)/2=232\). Thus \(E[X^2]=(1/3)(72)+(2/3)(232)=24+464/3=536/3\). Since \(\mu^2=(34/3)^2=1156/9\), the variance is \(536/3-1156/9=1608/9-1156/9=452/9\), matching the decomposition. The variance is rounded to two decimals only in the displayed approximation.

Interpreting the Two Components

The decomposition is more than a computational shortcut. It distinguishes uncertainty that remains even when the group is known from variation attributable to group membership. In the first example, knowing the group reduces uncertainty about the claim indicator, but the groups still have different claim probabilities. In the second example, the conditional variances contribute 6 of the total 14.64, while the differences in group means contribute 8.64.

The between-group term is nonnegative, so

$$ \operatorname{Var}(X)\geq E[\operatorname{Var}(X\mid G)]. $$

Equality holds precisely when the conditional mean \(E[X\mid G]\) is constant with probability one across the groups with positive probability. This does not require the conditional distributions to be identical: their variances or other features may differ while their means agree.

Common Exam Traps

  • Averaging conditional variances and stopping. This gives only the within-group component. Add the variance of the conditional means to obtain the total variance.
  • Taking an unweighted average across groups. The weights are the probabilities \(p_j\), not equal weights unless the groups actually have equal probabilities.
  • Using the wrong center in the between-group term. Each conditional mean is centered at the overall mean \(\mu\), not at zero or at another group’s mean.
  • Mixing variances and standard deviations. The theorem averages conditional variances. If a problem gives conditional standard deviations, square them before using the formula.
  • Defining moments for an impossible group. A conditional mean or variance given a zero-probability group is not specified by ordinary conditioning; omit that group from the mixture calculation.
Key takeaway: To find the variance of a finite mixture, calculate the probability-weighted average of the conditional variances, then add the probability-weighted variance of the conditional means. Both components are needed.

Check Your Understanding

For each problem, identify the within-group and between-group contributions before combining them.

  1. A random observation comes from group A with probability \(0.30\) and group B with probability \(0.70\). The conditional means are 4 and 10, and the conditional variances are 5 and 2, respectively. Find the overall mean and variance. (Answer: mean \(8.2\); within variance \(3.5\), between variance \(7.56\), total variance \(11.06\).)
  2. A hypothetical claim indicator \(X\) is Bernoulli with probability \(0.25\) in group A and \(0.75\) in group B. The group probabilities are \(0.40\) and \(0.60\). Find the overall claim probability and variance by decomposition. (Answer: claim probability \(0.55\); within variance \(0.1875\), between variance \(0.06\), total variance \(0.2475\).)
  3. Three groups have probabilities \(0.20,0.50,0.30\), conditional means \(2,5,9\), and conditional variances \(1,4,0\). Calculate the overall mean and total variance. (Answer: mean \(5.6\); within variance \(0.4\), between variance \(6.24\), total variance \(6.64\).)
  4. A measurement is conditionally uniform on \((0,4)\) in group A and on \((4,10)\) in group B. The group probabilities are \(0.25\) and \(0.75\). Use the conditional uniform means and variances to find the overall mean and variance. (Answer: mean \(5.75\); within variance \(97/48\), between variance \(75/16\), total variance \(161/24\).)
  5. Group A has probability \(1/3\) and conditional outcomes 2 and 6 with equal probabilities. Group B has probability \(2/3\) and conditional outcomes 4 and 10 with equal probabilities. Find the overall mean and variance, then verify the variance using \(E[X^2]-E[X]^2\). (Answer: mean \(20/3\); within variance \(20/3\), between variance \(32/9\), total variance \(92/9\).)

Previous

Next

Back to Tutorials
Clinical Biostats