Why These Errors Matter
Conditioning can make a difficult calculation manageable, but it also introduces a second layer of randomness. A common error is to calculate what happens within each condition correctly and then forget that the conditional mean itself can vary from one condition to another. Another is to use an unconditional mean or variance where a conditional moment is required. Both mistakes can produce answers that look plausible while describing a different quantity.
The key distinction is between a conditional quantity for a specified condition and a conditional quantity that is still random because the condition is random. If \(Y\) is a random variable, then \(E[X\mid Y]\) is generally a random variable: its value depends on the realized value of \(Y\). The same point applies to \(\operatorname{Var}(X\mid Y)\).
The formulas below require the relevant expectations to exist; the variance decomposition requires \(E[X^2]\) to be finite. For a discrete conditioning variable, an expectation over \(Y\) is a probability-weighted sum over its possible values. For a continuous conditioning variable, it is an expectation with respect to the distribution of \(Y\).
The Two Decompositions to Keep Straight
The first formula says to average the conditional means using the distribution of the conditioning variable. The second has two parts: the average conditional variance is the within-condition variation, while the variance of the conditional means is the between-condition variation. The second term is not optional just because the conditional variances have already been calculated.
A useful identity for checking work is $$ E[X^2\mid Y]=\operatorname{Var}(X\mid Y)+\bigl(E[X\mid Y]\bigr)^2. $$ Consequently, \(E[X^2]\) is found by averaging the entire right-hand side over \(Y\). In particular, the square of a conditional mean must be averaged as a square; it is not generally the square of its average.
Why Total Variance Includes Both Parts
Here is a short derivation that identifies exactly where the between-condition term comes from. Write \(M=E[X\mid Y]\) and \(\mu=E[X]\). Add and subtract \(M\) inside the squared deviation from \(\mu\):
Squaring and taking expectations gives a within-condition squared deviation, a between-condition squared deviation, and a cross-product. The cross-product has expectation zero. To see why, condition on \(Y\): \(M-\mu\) is fixed given \(Y\), while $$ E[X-M\mid Y]=E[X\mid Y]-M=0. $$ Therefore,
The remaining terms are \(E[(X-M)^2]=E[\operatorname{Var}(X\mid Y)]\) and \(E[(M-\mu)^2]=\operatorname{Var}(M)\). Since \(M=E[X\mid Y]\), this proves the total variance formula.
Worked Example: Do Not Drop Variation Between Risk Groups
A hypothetical portfolio classifies a randomly selected contract as standard with probability \(0.75\) or elevated-risk with probability \(0.25\). Let \(X\) be the number of covered incidents in a year. Given standard risk, \(X\) is equally likely to be 0 or 2. Given elevated risk, \(X\) is equally likely to be 2 or 6. Find \(E[X]\) and \(\operatorname{Var}(X)\). The method is total expectation followed by total variance; the common mistake to avoid is reporting only the average conditional variance.
Step 1: Find the conditional means and variances. For standard risk, $$ E[X\mid \text{standard}]=\frac{0+2}{2}=1,\qquad \operatorname{Var}(X\mid \text{standard})=\frac{(0-1)^2+(2-1)^2}{2}=1. $$ For elevated risk, $$ E[X\mid \text{elevated}]=\frac{2+6}{2}=4,\qquad \operatorname{Var}(X\mid \text{elevated})=\frac{(2-4)^2+(6-4)^2}{2}=4. $$
Step 2: Average the conditional means. $$ E[X]=0.75(1)+0.25(4)=0.75+1=1.75. $$
Step 3: Calculate the within-group part. $$ E[\operatorname{Var}(X\mid Y)]=0.75(1)+0.25(4)=0.75+1=1.75. $$
Step 4: Calculate the between-group part. The conditional mean is 1 with probability \(0.75\) and 4 with probability \(0.25\), and its mean is 1.75. Thus $$ \operatorname{Var}(E[X\mid Y]) =0.75(1-1.75)^2+0.25(4-1.75)^2 =0.75(0.5625)+0.25(5.0625) =0.421875+1.265625=1.6875. $$
Step 5: Combine and check. $$ \operatorname{Var}(X)=1.75+1.6875=3.4375. $$ As a direct check, \(E[X^2\mid\text{standard}]=(0^2+2^2)/2=2\), and \(E[X^2\mid\text{elevated}]=(2^2+6^2)/2=20\). Hence \(E[X^2]=0.75(2)+0.25(20)=6.5\), so \(\operatorname{Var}(X)=6.5-(1.75)^2=6.5-3.0625=3.4375\). The incorrect answer \(1.75\) keeps only the within-group term and omits the variation between the two conditional means.
Do Not Substitute an Unconditional Moment
A conditional variance must be calculated from the distribution within the condition. In particular, for a Bernoulli random variable with conditional success probability \(p_y=P(X=1\mid Y=y)\), its conditional variance is \(p_y(1-p_y)\). Replacing \(p_y\) by the unconditional success probability changes the conditional distribution being used.
Worked Example: Use Each Group’s Bernoulli Probability
A hypothetical screening process assigns each case to group A or group B with probability \(0.5\) each. Let \(X=1\) indicate that the case produces a positive result, and let \(X=0\) otherwise. The conditional positive-result probabilities are \(P(X=1\mid A)=0.2\) and \(P(X=1\mid B)=0.8\). Find the unconditional mean and variance. The method is to calculate conditional Bernoulli moments first and then decompose the variance; the trap is to use the unconditional success probability inside each group’s variance formula.
Step 1: Find the conditional means and variances. In group A, the mean is \(0.2\), and $$ \operatorname{Var}(X\mid A)=0.2(1-0.2)=0.2(0.8)=0.16. $$ In group B, the mean is \(0.8\), and $$ \operatorname{Var}(X\mid B)=0.8(1-0.8)=0.8(0.2)=0.16. $$
Step 2: Find the unconditional mean. $$ E[X]=0.5(0.2)+0.5(0.8)=0.1+0.4=0.5. $$
Step 3: Calculate both variance components. The average conditional variance is $$ E[\operatorname{Var}(X\mid Y)]=0.5(0.16)+0.5(0.16)=0.16. $$ The conditional mean is \(0.2\) or \(0.8\), each with probability \(0.5\), so $$ \operatorname{Var}(E[X\mid Y]) =0.5(0.2-0.5)^2+0.5(0.8-0.5)^2 =0.5(0.09)+0.5(0.09)=0.09. $$ Therefore \(\operatorname{Var}(X)=0.16+0.09=0.25\).
Check and diagnose. Unconditionally, \(X\) is Bernoulli with success probability \(0.5\), so its variance is \(0.5(1-0.5)=0.25\), matching the decomposition. A mistaken calculation might insert \(0.5\) into the conditional Bernoulli formula in each group, obtaining \(0.25\) as the average conditional variance, and then add the between-group term \(0.09\). That gives \(0.34\), which double-counts variation: \(0.25\) is already the unconditional variance, not the within-group variance.
Keep the Conditional Mean Random When Squaring
Another frequent error is to replace \(E[(E[X\mid Y])^2]\) by \((E[X])^2\). The two expressions are generally different because the conditional mean varies with \(Y\). Their difference is precisely \(\operatorname{Var}(E[X\mid Y])\). This distinction also appears when using conditional second moments to check a variance calculation.
Worked Example: Average the Squared Conditional Means
A hypothetical equipment program has two operating states. State A occurs with probability \(0.25\), and conditional on A, the measured repair index \(X\) is equally likely to be \(-1\) or 1. State B occurs with probability \(0.75\), and conditional on B, \(X\) is equally likely to be 1 or 5. Find \(E[X]\) and \(\operatorname{Var}(X)\), and show why the squared conditional means cannot be replaced by the square of the unconditional mean.
Step 1: Calculate conditional moments. In state A, $$ E[X\mid A]=\frac{-1+1}{2}=0,\qquad \operatorname{Var}(X\mid A)=\frac{(-1-0)^2+(1-0)^2}{2}=1. $$ In state B, $$ E[X\mid B]=\frac{1+5}{2}=3,\qquad \operatorname{Var}(X\mid B)=\frac{(1-3)^2+(5-3)^2}{2}=4. $$
Step 2: Find the unconditional mean. $$ E[X]=0.25(0)+0.75(3)=2.25. $$
Step 3: Apply total variance. The average conditional variance is $$ E[\operatorname{Var}(X\mid Y)]=0.25(1)+0.75(4)=0.25+3=3.25. $$ The between-state variance is $$ \operatorname{Var}(E[X\mid Y]) =0.25(0-2.25)^2+0.75(3-2.25)^2 =0.25(5.0625)+0.75(0.5625) =1.265625+0.421875=1.6875. $$ Thus \(\operatorname{Var}(X)=3.25+1.6875=4.9375\).
Step 4: Check using second moments. The conditional second moments are \(E[X^2\mid A]=(1+1)/2=1\) and \(E[X^2\mid B]=(1+25)/2=13\). Therefore $$ E[X^2]=0.25(1)+0.75(13)=0.25+9.75=10, \qquad \operatorname{Var}(X)=10-(2.25)^2=10-5.0625=4.9375. $$ Also, \(E[(E[X\mid Y])^2]=0.25(0^2)+0.75(3^2)=6.75\), whereas \((E[X])^2=(2.25)^2=5.0625\). Their difference, \(6.75-5.0625=1.6875\), is the between-state variance. Using the latter in place of the former would omit that component.
A Reliable Error-Checking Routine
When a problem asks for an unconditional mean or variance after conditioning, write the conditional quantities as functions of the conditioning variable before taking an expectation. This makes it harder to mistake a random conditional mean for a constant.
List the possible values or groups for \(Y\) and their probabilities. Check that the probabilities sum to 1.
Find \(E[X\mid Y=y]\) and, when needed, \(\operatorname{Var}(X\mid Y=y)\) from the conditional distribution for that particular \(y\).
Use \(E[X]=E[E[X\mid Y]]\). Do not treat the conditional mean as a fixed value before averaging over \(Y\).
Add the average conditional variance and the variance of the conditional means. Do not substitute the unconditional variance for either term.
Compute \(E[X^2]=E[\operatorname{Var}(X\mid Y)+(E[X\mid Y])^2]\), then subtract \((E[X])^2\). Agreement is a useful check for omitted or misapplied terms.
These checks are especially valuable in insurance and other mixture settings, where a hidden risk class or operating state can create substantial between-group variation even when variation within each group is modest. The formulas are aligned with the SOA Exam P (Probability) syllabus.
Check Your Understanding
For each problem, identify the conditional moments before combining them. Answers are included for checking.
- A condition \(Y=A\) occurs with probability \(0.4\), and \(Y=B\) occurs with probability \(0.6\). Given A, \(E[X\mid A]=2\) and \(\operatorname{Var}(X\mid A)=3\). Given B, \(E[X\mid B]=5\) and \(\operatorname{Var}(X\mid B)=4\). Find \(E[X]\) and \(\operatorname{Var}(X)\). (Answer: \(E[X]=3.8\), \(\operatorname{Var}(X)=5.76\))
- A case is equally likely to be in group A or group B. Given A, \(X\) is Bernoulli with success probability \(0.1\); given B, it is Bernoulli with success probability \(0.7\). Find the unconditional mean and variance, including both variance components. (Answer: \(E[X]=0.4\), within-group variance \(=0.15\), between-group variance \(=0.09\), \(\operatorname{Var}(X)=0.24\))
- Let \(P(Y=A)=0.25\) and \(P(Y=B)=0.75\). Suppose \(E[X\mid A]=1\), \(E[X\mid B]=3\), and both conditional variances equal 2. Find the unconditional mean and variance. (Answer: \(E[X]=2.5\), \(\operatorname{Var}(X)=2.75\))
- A high-cost state occurs with probability \(0.2\), and a low-cost state occurs with probability \(0.8\). Given the high-cost state, \(X\) has mean 10 and variance 4; given the low-cost state, \(X\) has mean 3 and variance 1. Find the unconditional mean and variance. (Answer: \(E[X]=4.4\), within-group variance \(=1.6\), between-group variance \(=7.84\), \(\operatorname{Var}(X)=9.44\))
- The two values of \(Y\) are equally likely. Given \(Y=A\), \(E[X\mid A]=0\) and \(E[X^2\mid A]=2\). Given \(Y=B\), \(E[X\mid B]=4\) and \(E[X^2\mid B]=20\). Find \(E[X]\), \(E[X^2]\), and \(\operatorname{Var}(X)\). (Answer: \(E[X]=2\), \(E[X^2]=11\), \(\operatorname{Var}(X)=7\))