Conditional Variance Within a Group

Why Variability Within a Group Matters

A conditional mean describes the average value of $X$ among outcomes that meet a specified condition. But two groups can have the same mean and very different spreads. For example, two categories of hypothetical policyholders might have the same average claim count, while one category has much more variation from member to member. Conditional variance measures that spread within the specified group.

The key is to calculate variance from the distribution of $X$ after conditioning, not from its unconditional distribution. For a discrete conditioning variable, this means selecting the appropriate conditional probabilities. For a continuous conditioning variable, it means using the conditional density at the specified value. The conditional variance is not generally the same as $\operatorname{Var}(X)$.

Definition: For a fixed value $y$ with a well-defined conditional distribution of $X$, the conditional variance of $X$ given $Y=y$ is $$ \operatorname{Var}(X\mid Y=y) =E\left[(X-E[X\mid Y=y])^2\mid Y=y\right]. $$ It is the expected squared deviation from the conditional mean, where the expectation is taken using the conditional distribution of $X$ given $Y=y$.

Calculating Conditional Variance

Suppose $X$ is discrete. If its possible values are $x_1,x_2,\ldots$, and the conditional probabilities are $P(X=x_j\mid Y=y)$, first find the conditional mean. Then average the squared deviations from that mean using those same conditional probabilities.

E[X\mid Y=y]=\sum_j x_jP(X=x_j\mid Y=y), \qquad \operatorname{Var}(X\mid Y=y) =\sum_j (x_j-E[X\mid Y=y])^2P(X=x_j\mid Y=y).

An equivalent calculation uses the conditional second moment. It is often faster, but it requires careful arithmetic and the same conditional probabilities in both moments.

\operatorname{Var}(X\mid Y=y) =E[X^2\mid Y=y]-\big(E[X\mid Y=y]\big)^2.

Worked Example: Variance From Conditional Deviations

A hypothetical service center classifies each repair by the workload group $Y$. For repairs in group $Y=\text{standard}$, the number of technician-hours $X$ has conditional probabilities $P(X=0\mid Y=\text{standard})=0.20$, $P(X=2\mid Y=\text{standard})=0.50$, and $P(X=5\mid Y=\text{standard})=0.30$. Find the conditional mean and variance of technician-hours in this group.

Method. Use the conditional probabilities for the standard group, first finding the mean and then the weighted squared deviations.

The probabilities sum to $0.20+0.50+0.30=1.00$. The conditional mean is

E[X\mid Y=\text{standard}] =0(0.20)+2(0.50)+5(0.30) =0+1.00+1.50 =2.50.

The squared deviations from $2.50$ are $6.25$, $0.25$, and $6.25$. Weight each by its conditional probability:

\begin{aligned} \operatorname{Var}(X\mid Y=\text{standard}) &=(0-2.50)^2(0.20)+(2-2.50)^2(0.50)+(5-2.50)^2(0.30)\\ &=6.25(0.20)+0.25(0.50)+6.25(0.30)\\ &=1.25+0.125+1.875\\ &=3.25\text{ hours}^2. \end{aligned}

Check. The probabilities used sum to one, and the variance is nonnegative. The result is a variance in squared hours, not hours. A common mistake is to average the possible values without conditioning; that would mix repairs from other workload groups into the calculation.

Why the Conditional-Moment Formula Works

The shorter formula follows by expanding the squared deviation around the conditional mean. Write $m_y=E[X\mid Y=y]$. Within the conditional distribution, $m_y$ is a constant, and the conditional expectation of $X-m_y$ is zero. Therefore:

\begin{aligned} \operatorname{Var}(X\mid Y=y) &=E[(X-m_y)^2\mid Y=y]\\ &=E[X^2-2m_yX+m_y^2\mid Y=y]\\ &=E[X^2\mid Y=y]-2m_yE[X\mid Y=y]+m_y^2\\ &=E[X^2\mid Y=y]-2m_y^2+m_y^2\\ &=E[X^2\mid Y=y]-m_y^2. \end{aligned}

The third line uses the fact that a constant can be taken outside a conditional expectation. The fourth uses $E[X\mid Y=y]=m_y$. This identity is useful computationally, but the centered-deviation definition explains what the answer measures.

Worked Example: Conditional Density at a Fixed Value

Suppose the joint density of two variables $X$ and $Y$ is $f_{X,Y}(x,y)=2$ on $0<x<y<1$, and zero elsewhere. Find $\operatorname{Var}(X\mid Y=0.6)$. The joint density integrates to one over this triangular region, since its area is $1/2$ and its constant height is $2$.

Method. Derive the conditional density by dividing the joint density by the marginal density of $Y$. For $0<y<1$,

f_Y(y)=\int_0^y 2\,dx=2y.

Thus, for $0<x<y$,

f_{X\mid Y}(x\mid y)=\frac{f_{X,Y}(x,y)}{f_Y(y)} =\frac{2}{2y} =\frac{1}{y}.

At $y=0.6$, the conditional density is $1/0.6=5/3$ over $0<x<0.6$. This is a uniform density on that interval. Its conditional first and second moments can be calculated directly:

\begin{aligned} E[X\mid Y=0.6] &=\int_0^{0.6}x\left(\frac{5}{3}\right)dx =\frac{5}{3}\left(\frac{0.6^2}{2}\right) =0.30,\\ E[X^2\mid Y=0.6] &=\int_0^{0.6}x^2\left(\frac{5}{3}\right)dx =\frac{5}{3}\left(\frac{0.6^3}{3}\right) =0.12. \end{aligned}

Therefore,

\operatorname{Var}(X\mid Y=0.6) =0.12-(0.30)^2 =0.12-0.09 =0.03.

Check. The result also agrees with the variance of a uniform distribution on an interval of length $0.6$: $0.6^2/12=0.03$. Since $Y$ is continuous, $P(Y=0.6)$ is zero; the calculation uses the conditional density at $y=0.6$, not division by $P(Y=0.6)$.

Comparing Variance Across Groups

A group comparison requires one conditional distribution and one conditional variance for each group. The groups may have different conditional means, different conditional variances, or both. To compare spread, calculate each variance around that group’s own conditional mean.

Worked Example: Comparing Two Conditional Distributions

In a hypothetical portfolio, let $X$ be the number of service events recorded for an account, and let $G$ identify its account group. The joint probabilities for $X$ and the two groups are shown below. Find and compare the conditional variances for groups $A$ and $B$.

$X$	$P(X=x,G=A)$	$P(X=x,G=B)$
0	0.12	0.10
2	0.36	0.12
4	0.12	0.18

Step 1: Find the probability of each group. The columns sum to $P(G=A)=0.60$ and $P(G=B)=0.40$. The entire table sums to $1.00$.

Step 2: Normalize within group $A$. Divide each joint probability in the $A$ column by $0.60$. The conditional probabilities for $X=0,2,4$ are $0.20,0.60,0.20$. Their sum is $1.00$. Thus,

E[X\mid G=A]=0(0.20)+2(0.60)+4(0.20)=2.00, \qquad E[X^2\mid G=A]=0+4(0.60)+16(0.20)=5.60.

It follows that

\operatorname{Var}(X\mid G=A)=5.60-(2.00)^2=1.60.

Step 3: Normalize within group $B$. Divide the $B$-column probabilities by $0.40$. The conditional probabilities are $0.25,0.30,0.45$, which sum to $1.00$. Then

E[X\mid G=B]=0(0.25)+2(0.30)+4(0.45)=0+0.60+1.80=2.40,

E[X^2\mid G=B]=0^2(0.25)+2^2(0.30)+4^2(0.45)=0+1.20+7.20=8.40.

Therefore,

\operatorname{Var}(X\mid G=B)=8.40-(2.40)^2=8.40-5.76=2.64.

Conclusion and check. Group $B$ has the larger conditional variance: $2.64>1.60$. The calculation uses each group’s own conditional distribution and mean. The probabilities in each normalized group sum to one, and both variances are nonnegative.

Common Mistakes and Exam Traps

Using joint probabilities as if they were conditional probabilities. In the table example, the $B$-column entries sum to $0.40$, not $1$. Divide each by $0.40$ before calculating conditional moments.
Subtracting the wrong squared mean. The formula is $E[X^2\mid Y=y]-(E[X\mid Y=y])^2$. The square applies to the conditional mean, not to each outcome before averaging.
Mixing groups when centering deviations. For each group, deviations are measured from that group’s conditional mean. Using an overall mean does not calculate that group’s conditional variance.
Making an arithmetic slip after the moments are correct. In group $B$, $8.40-(2.40)^2=8.40-5.76=2.64$, not $2.44$. Recheck both the second moment and the final subtraction.
Confusing variance with standard deviation. Conditional variance is expressed in squared units. Its square root is the conditional standard deviation, a different quantity.

Conditional variance answers a focused question: once the group or conditioning value is specified, how dispersed is $X$ under that conditional distribution? It does not by itself describe how much the groups differ from one another in their means. Keeping the conditioning event, conditional probabilities, and group-specific center aligned is essential for a correct calculation.

Check Your Understanding

For each problem, calculate only from the stated conditional distribution or conditional density.

For $Y=a$, suppose $X$ takes values $1$ and $4$ with conditional probabilities $0.25$ and $0.75$. Find $\operatorname{Var}(X\mid Y=a)$. (Answer: $1.6875$)
The conditional density of $X$ given $Y=y$ is uniform on $0<x<y$. Find $\operatorname{Var}(X\mid Y=0.5)$. (Answer: $1/48$, or approximately $0.0208$)
Given $G=c$, $X$ takes values $0,3,6$ with probabilities $0.20,0.50,0.30$. Find the conditional mean and variance. (Answer: $E[X\mid G=c]=3.3$, $\operatorname{Var}(X\mid G=c)=4.41$)
In group $C$, $X$ is 1 or 5 with equal probabilities. In group $D$, $X$ is 0, 2, or 4 with probabilities $0.25,0.50,0.25$. Which group has the larger conditional variance, and what are both variances? (Answer: $C$ has the larger variance; $\operatorname{Var}(X\mid C)=4$, $\operatorname{Var}(X\mid D)=2$)

What You'll Learn