Conditional Moments and Conditional Variance

Why Conditional Variability Matters

An unconditional variance describes spread across all outcomes in a population or experiment. But when information is available—such as a known group, event, or value of another variable—the relevant spread may change. Conditional variance measures the remaining variability in $X$ after that information is fixed. It is not the same quantity as the variability of the conditional mean across different conditions.

For example, claim amounts might have one spread among low-risk policies and another among high-risk policies. The conditional mean answers what the average claim is within a specified group; the conditional variance describes how widely individual claims in that group vary around that mean. These calculations are aligned with the SOA VEE Mathematical Statistics syllabus.

Definition: Suppose $X$ has a finite conditional second moment given $Y$. The conditional raw moment of order $r$ is $E[X^r\mid Y]$. The conditional mean is $E[X\mid Y]$, and the conditional variance and standard deviation are $$ \operatorname{Var}(X\mid Y)=E\!\left[(X-E[X\mid Y])^2\mid Y\right], \qquad \operatorname{SD}(X\mid Y)=\sqrt{\operatorname{Var}(X\mid Y)}. $$ When conditioning on a particular value $Y=y$, the corresponding quantities are written $E[X^r\mid Y=y]$, $\operatorname{Var}(X\mid Y=y)$, and $\operatorname{SD}(X\mid Y=y)$.

For a discrete conditional distribution, the conditional raw moment is a probability-weighted sum. For a continuous conditional distribution, it is an integral against the conditional density. In either case, the condition must identify a valid conditional law: for conditioning on an event $B$, require $P(B)>0$; for conditioning on a value of a continuous variable, use a specified conditional distribution or density. Conditional quantities given a random variable are themselves functions of that variable and are defined up to sets of probability zero.

From Conditional Raw Moments to Variance

The first conditional raw moment is the conditional mean. The second raw moment is not the variance: it must be adjusted for the conditional mean. Provided the conditional second moment is finite, the useful calculation is

\operatorname{Var}(X\mid Y) =E[X^2\mid Y]-\big(E[X\mid Y]\big)^2.

The square applies to the conditional mean as a whole. In particular, $E[X^2\mid Y]$ and $\big(E[X\mid Y]\big)^2$ are generally different. The same formula applies when conditioning on a fixed value $Y=y$.

Find the conditional mean.
Use the conditional probabilities or density to calculate $m=E[X\mid Y=y]$.

Find the conditional second raw moment.
Calculate $E[X^2\mid Y=y]$, squaring each possible value of $X$ before weighting or integrating.

Subtract the squared mean.
Compute $\operatorname{Var}(X\mid Y=y)=E[X^2\mid Y=y]-m^2$.

Take the square root if requested.
The conditional standard deviation is the nonnegative square root of the conditional variance.

Why the Raw-Moment Formula Works

Fix a condition $Y=y$, and let $m=E[X\mid Y=y]$. By definition, conditional variance is the conditional expectation of the squared deviation from $m$. Expanding the square and using linearity of conditional expectation gives

\begin{aligned} \operatorname{Var}(X\mid Y=y) &=E[(X-m)^2\mid Y=y]\\ &=E[X^2-2mX+m^2\mid Y=y]\\ &=E[X^2\mid Y=y]-2mE[X\mid Y=y]+m^2\\ &=E[X^2\mid Y=y]-2m^2+m^2\\ &=E[X^2\mid Y=y]-m^2. \end{aligned}

The third line treats $m$ as a fixed number under the specified condition. The fourth line uses $E[X\mid Y=y]=m$. For a random conditioning variable $Y$, the same argument applies conditionally, with $m=E[X\mid Y]$ retained as a function of $Y$.

Worked Example: Two Possible Claim Counts Given a Review Category

Method: Calculate the first two conditional raw moments from the conditional probability mass function, then apply the raw-moment variance formula. In a hypothetical claims review, given that a file is in category $B$, the number $X$ of flagged items is 2 with probability 0.75 and 6 with probability 0.25. Find the conditional mean, second raw moment, variance, and standard deviation.

Conditional mean. Weight each possible value by its conditional probability:

E[X\mid B]=2(0.75)+6(0.25)=1.5+1.5=3.

Second raw moment. Square the possible values before weighting:

E[X^2\mid B]=2^2(0.75)+6^2(0.25) =4(0.75)+36(0.25)=3+9=12.

Variance. Subtract the square of the conditional mean:

\operatorname{Var}(X\mid B)=12-3^2=12-9=3.

Standard deviation. Take the nonnegative square root:

\operatorname{SD}(X\mid B)=\sqrt{3}\approx 1.7321.

Check. Directly from the centered definition, the deviations from 3 are $-1$ and 3, so the variance is $0.75(1^2)+0.25(3^2)=0.75+2.25=3$, agreeing with the shortcut. A common mistake is to report the second raw moment, 12, as the variance; the squared conditional mean, 9, must be subtracted.

Conditional Moments from a Density

For a continuous $X$, if the conditional density of $X$ given $Y=y$ is $f_{X\mid Y}(x\mid y)$, then, when the integral exists,

E[X^r\mid Y=y]=\int_{-\infty}^{\infty}x^r f_{X\mid Y}(x\mid y)\,dx.

The density must be integrated over its conditional support. A quick normalization check—its integral over that support equals 1—can catch an error before moments are calculated.

Worked Example: Conditional Moments from a Scaled Density

Method: Integrate powers of $X$ against the conditional density, then use the second raw moment to obtain the variance. Suppose a hypothetical continuous measurement $X$, given a positive scale value $Y=y$, has conditional density

f_{X\mid Y}(x\mid y)=\frac{2x}{y^2},\qquad 0<x<y,\quad y>0,

and density zero elsewhere. Find the conditional mean, second raw moment, variance, and standard deviation when $y=6$.

Check the density. Its integral over the support is

\int_0^y \frac{2x}{y^2}\,dx =\frac{1}{y^2}[x^2]_0^y=1.

Conditional mean. For any fixed $y>0$,

E[X\mid Y=y] =\int_0^y x\frac{2x}{y^2}\,dx =\frac{2}{y^2}\left[\frac{x^3}{3}\right]_0^y =\frac{2y}{3}.

Second raw moment.

E[X^2\mid Y=y] =\int_0^y x^2\frac{2x}{y^2}\,dx =\frac{2}{y^2}\left[\frac{x^4}{4}\right]_0^y =\frac{y^2}{2}.

At $y=6$, the mean is $2(6)/3=4$, and the second raw moment is $6^2/2=18$. Therefore,

\operatorname{Var}(X\mid Y=6)=18-4^2=18-16=2, \qquad \operatorname{SD}(X\mid Y=6)=\sqrt{2}\approx1.4142.

Check. The general variance is $y^2/2-(2y/3)^2=y^2(1/2-4/9)=y^2/18$, which at $y=6$ is $36/18=2$. The standard deviation is thus $\sqrt{2}$, consistent with the variance.

Conditional Spread Versus Variation in Conditional Means

A conditional variance describes spread within one specified condition. By contrast, conditional means can differ from one condition to another. Their variation describes how the predicted or average value of $X$ changes across conditions. Neither quantity is a substitute for the other.

For example, two groups can have the same conditional mean but different conditional variances. Conversely, groups can have small within-group variances but very different conditional means. If $m(Y)=E[X\mid Y]$, then $\operatorname{Var}(X\mid Y)$ describes conditional spread, while the distribution of $m(Y)$ describes variation among conditional means. Calculating one does not, by itself, determine the other.

Worked Example: Comparing Within-Condition Spread and Conditional Means

Method: Calculate the conditional mean and variance separately for each condition, then calculate the variance of the conditional means across conditions. A hypothetical service center records daily call counts $X$ under weather condition $W$. Calm weather has probability 0.75; stormy weather has probability 0.25. Given calm weather, $X$ is 1 or 3 with probability 0.5 each. Given stormy weather, $X$ is 4 or 8 with probability 0.5 each. Find the conditional means and variances, and the variance of the conditional means.

Calm weather. The conditional mean is $0.5(1)+0.5(3)=2$. The second raw moment is $0.5(1^2)+0.5(3^2)=0.5+4.5=5$, so the conditional variance is $5-2^2=1$.

Stormy weather. The conditional mean is $0.5(4)+0.5(8)=6$. The second raw moment is $0.5(4^2)+0.5(8^2)=8+32=40$, so the conditional variance is $40-6^2=4$.

The conditional means are therefore 2 with probability 0.75 and 6 with probability 0.25. Their mean is $0.75(2)+0.25(6)=1.5+1.5=3$, and their variance is

$$ 0.75(2-3)^2+0.25(6-3)^2 =0.75(1)+0.25(9)=0.75+2.25=3. $$

Check and interpretation. The within-condition variances are 1 and 4; the variation among the conditional means is 3. These are distinct summaries of different features of the model. The value 3 above is the variance of the two conditional means with their stated weather probabilities; it is not either conditional variance.

Common Mistakes and Exam Traps

The raw-moment shortcut is efficient, but its notation makes several errors easy to commit. For the conditional distribution in the first example, the correct values are $E[X\mid B]=3$ and $E[X^2\mid B]=12$. A mistaken calculation might subtract the mean rather than its square:

12-3=9 \quad\text{(incorrect)},\qquad 12-3^2=3 \quad\text{(correct)}.

Another error is to square the possible values after averaging rather than before: $E[X^2\mid B]$ requires weighting $2^2$ and $6^2$, not squaring the mean. Keep these quantities distinct:

Key distinction: $E[X^2\mid Y]$ is the conditional expectation of the squared variable. $\big(E[X\mid Y]\big)^2$ is the square of the conditional mean. Their difference is the conditional variance, when the conditional second moment exists.

Do not confuse variance and standard deviation. Variance is in squared units; standard deviation is its square root and has the units of $X$.
Do not omit the condition. A mean or variance calculated within one group need not equal its unconditional counterpart.
For a density, use the conditional support. Integrating beyond that support can produce incorrect moments even when the integrand is otherwise correct.
Do not confuse two kinds of spread. Conditional variance measures spread within a condition; variation in conditional means measures differences across conditions.

Key takeaway: To calculate conditional variance, find the conditional second raw moment and subtract the square of the conditional mean. To interpret the result, state the condition clearly: it measures spread remaining within that condition, not variation among conditional means.

Check Your Understanding

For each problem, calculate the requested conditional moments and keep the conditioning information explicit.

Given event $C$, a hypothetical count $X$ equals 0 with probability 0.6 and 4 with probability 0.4. Find $E[X\mid C]$, $E[X^2\mid C]$, $\operatorname{Var}(X\mid C)$, and $\operatorname{SD}(X\mid C)$. (Answer: mean $1.6$; second raw moment $6.4$; variance $3.84$; standard deviation $\sqrt{3.84}\approx1.9596$.)
Given condition $D$, $X$ is 3 with probability 0.25 and 9 with probability 0.75. Find its conditional mean, variance, and standard deviation. (Answer: mean $7.5$; variance $6.75$; standard deviation $\sqrt{6.75}\approx2.5981$.)
Given $Y=a>0$, a continuous variable has conditional density $f_{X\mid Y}(x\mid a)=3x^2/a^3$ for $0<x<a$, and zero elsewhere. Find the conditional mean, second raw moment, and variance. (Answer: $E[X\mid Y=a]=3a/4$; $E[X^2\mid Y=a]=3a^2/5$; $\operatorname{Var}(X\mid Y=a)=3a^2/80$.)
Given event $E$, a hypothetical loss amount $X$, in thousands of dollars, is 1, 5, or 9 with conditional probabilities 0.2, 0.5, or 0.3, respectively. Find its conditional mean, variance, and standard deviation, keeping the units in mind. (Answer: mean $5.4$ thousand dollars; variance $7.84$ thousand-dollars-squared; standard deviation $2.8$ thousand dollars.)
A condition $W=A$ has probability 0.5 and gives $X=0$ or $2$ with equal probabilities; $W=B$ also has probability 0.5 and gives $X=5$ or $9$ with equal probabilities. Find each conditional mean and variance, then find the variance of the conditional means. (Answer: for $A$, mean $1$, variance $1$; for $B$, mean $7$, variance $4$; variance of conditional means $9$.)

What You'll Learn