Conditional Variance and Conditional Standard Deviation

Why Measure Variation After Conditioning?

A conditional mean describes the center of a distribution within a specified group or at a specified value of another variable. A conditional variance describes its spread. For instance, patients with different treatment histories may have different average recovery times, while patients with the same history may still differ from one another. These are distinct features of the conditional distributions: their locations and their within-group variability.

Conditional variance packages the second feature into a random quantity that depends on what we condition on. Its value can differ from group to group. The corresponding conditional standard deviation is on the original measurement scale, so it is often easier to interpret alongside the conditional mean.

Definition and Second-Moment Formula

Definition: Let $X$ be a random variable with $E[X^2]<\infty$, and let $Y$ be a random variable. The conditional variance of $X$ given $Y$ is $$ \operatorname{Var}(X\mid Y) = E\!\left[(X-E[X\mid Y])^2\mid Y\right]. $$ It is a nonnegative random variable measurable with respect to the information in $Y$, defined up to equality almost surely. The conditional standard deviation is $$ \operatorname{SD}(X\mid Y)=\sqrt{\operatorname{Var}(X\mid Y)}. $$

The square-integrability condition $E[X^2]<\infty$ ensures that the conditional second moment is finite almost surely. In particular, $E[X\mid Y]$ is square-integrable. Conditional expectations, and hence conditional variances, are only determined up to almost-sure equality; two versions that differ on a set of $Y$-values having probability zero represent the same conditional quantity.

The definition centers $X$ at its conditional mean, not at its unconditional mean. Expanding the square gives a convenient formula in terms of conditional first and second moments.

Result: If $E[X^2]<\infty$, then $$ \operatorname{Var}(X\mid Y) = E[X^2\mid Y]-\big(E[X\mid Y]\big)^2 \quad\text{almost surely}. $$

To verify the formula, write $m=E[X\mid Y]$. Since $m$ is a function of the conditioning information, it can be treated as known when taking the conditional expectation. The products in the expansion are integrable because $X$ and $m$ are square-integrable.

\begin{aligned} \operatorname{Var}(X\mid Y) &=E[(X-m)^2\mid Y]\\ &=E[X^2-2Xm+m^2\mid Y]\\ &=E[X^2\mid Y]-2mE[X\mid Y]+m^2\\ &=E[X^2\mid Y]-m^2. \end{aligned}

The last equality uses $E[X\mid Y]=m$. This proves the result. $\square$

Calculating from a Conditional Law

If $Y$ is discrete and $P(Y=y)>0$, the conditional law of $X$ given $Y=y$ yields ordinary conditional moments: $$ E[X\mid Y=y]=\sum_x xP(X=x\mid Y=y),\qquad E[X^2\mid Y=y]=\sum_x x^2P(X=x\mid Y=y). $$ Subtracting the square of the first from the second gives $\operatorname{Var}(X\mid Y=y)$. For a continuous conditional density $f_{X\mid Y}(x\mid y)$, the corresponding moments are integrals: $$ E[X^r\mid Y=y]=\int_{-\infty}^{\infty}x^r f_{X\mid Y}(x\mid y)\,dx,\qquad r=1,2, $$ when these integrals exist. These expressions describe a version of the conditional moments for almost every $y$ under the distribution of $Y$.

Worked Example: Conditional Variances in Two Groups

Let $Y$ take the values 0 and 1, with $P(Y=0)=0.4$ and $P(Y=1)=0.6$. Given $Y=0$, suppose $X$ is 0 or 2 with probability $1/2$ each. Given $Y=1$, suppose $X$ is 1 with probability $1/4$ and 5 with probability $3/4$. Find the conditional mean, variance, and standard deviation in each group.

Group $Y=0$. The conditional first and second moments are $$ E[X\mid Y=0]=0(1/2)+2(1/2)=1, $$ $$ E[X^2\mid Y=0]=0^2(1/2)+2^2(1/2)=2. $$ Thus $$ \operatorname{Var}(X\mid Y=0)=2-1^2=1,\qquad \operatorname{SD}(X\mid Y=0)=1. $$

Group $Y=1$. Here, $$ E[X\mid Y=1]=1(1/4)+5(3/4)=4, $$ $$ E[X^2\mid Y=1]=1^2(1/4)+5^2(3/4)=\frac14+\frac{75}{4}=19. $$ Therefore, $$ \operatorname{Var}(X\mid Y=1)=19-4^2=3,\qquad \operatorname{SD}(X\mid Y=1)=\sqrt{3}. $$

The groups have different conditional means, 1 and 4, and different within-group variances, 1 and 3. The conditional variance in either group measures the spread of $X$ among outcomes in that group; the difference between the group means is a separate feature.

Worked Example: A Continuous Conditional Distribution

Suppose $Y\sim\operatorname{Uniform}(1,3)$, and, conditional on $Y=y$, $X\sim\operatorname{Uniform}(0,y)$. Find the conditional variance and standard deviation as functions of $y$, and evaluate them at $y=2$.

For $y\in(1,3)$, the conditional density of $X$ is $1/y$ on $0<x<y$. Hence $$ E[X\mid Y=y]=\int_0^y x\frac{1}{y}\,dx =\frac{1}{y}\left[\frac{x^2}{2}\right]_0^y =\frac{y}{2}. $$ The second conditional moment is $$ E[X^2\mid Y=y]=\int_0^y x^2\frac{1}{y}\,dx =\frac{1}{y}\left[\frac{x^3}{3}\right]_0^y =\frac{y^2}{3}. $$ It follows that $$ \operatorname{Var}(X\mid Y=y)=\frac{y^2}{3}-\left(\frac{y}{2}\right)^2 =\frac{y^2}{12}, \qquad \operatorname{SD}(X\mid Y=y)=\frac{y}{\sqrt{12}}. $$ At $y=2$, the conditional mean is 1, the variance is $4/12=1/3$, and the standard deviation is $2/\sqrt{12}=1/\sqrt{3}$, approximately $0.577$. The conditional spread grows with $y$, because the interval supporting the conditional distribution grows with $y$.

Worked Example: A Binomial Conditional Law

Suppose that, given a group size $N=n$, the number of successes $X$ has a $\operatorname{Binomial}(n,0.25)$ distribution. Find the conditional mean, variance, and standard deviation when $N=8$.

Represent $X$, conditional on $N=8$, as the sum of eight independent Bernoulli indicators $Z_1,\ldots,Z_8$, each with success probability $0.25$. Then $$ E[X\mid N=8]=8(0.25)=2. $$ For the second moment, expand the square and use $Z_i^2=Z_i$ and conditional independence: $$ \begin{aligned} E[X^2\mid N=8] &=E\left[\sum_{i=1}^8 Z_i+2\sum_{1\leq i<j\leq 8}Z_iZ_j\mid N=8\right]\\ &=8(0.25)+2\binom{8}{2}(0.25)^2\\ &=2+2(28)(0.0625)=5.5. \end{aligned} $$ Therefore, $$ \operatorname{Var}(X\mid N=8)=5.5-2^2=1.5, \qquad \operatorname{SD}(X\mid N=8)=\sqrt{1.5}\approx1.225. $$ The variance here describes the spread in the number of successes among repetitions with the group size fixed at eight.

What Conditional Variance Says—and Does Not Say

Conditional variance is a within-group measure: it describes dispersion remaining in $X$ after the value of $Y$ is specified. It is itself random because the conditioning value can vary. In the first example, for instance, $\operatorname{Var}(X\mid Y)$ equals 1 when $Y=0$ and 3 when $Y=1$.

Variation in conditional means is different. If the groups have means far apart, that indicates that the center of $X$ changes across groups. It does not, by itself, tell us how spread out $X$ is within any one group. Conversely, groups can have similar means but quite different conditional variances. Keeping these questions separate prevents a common interpretive error: treating differences between group averages as if they measured within-group spread.

Result: For square-integrable $X$, $\operatorname{Var}(X\mid Y)\geq 0$ almost surely. Moreover, for real-valued $Y$, $\operatorname{Var}(X\mid Y)=0$ almost surely if and only if $X=g(Y)$ almost surely for some measurable function $g$.

Nonnegativity follows directly from the definition: a squared quantity is nonnegative, and the conditional expectation of a nonnegative random variable is nonnegative almost surely. To prove the zero-variance characterization, let $m=E[X\mid Y]$. By the tower property, $$ E[(X-m)^2]=E\!\left[E[(X-m)^2\mid Y]\right] =E[\operatorname{Var}(X\mid Y)]. $$ If the conditional variance is zero almost surely, then the right-hand side is zero. Since $(X-m)^2\geq0$, its expectation can be zero only if $(X-m)^2=0$ almost surely. Thus $X=m$ almost surely. Because $m$ is measurable with respect to $Y$, the Doob–Dynkin lemma gives a measurable $g$ such that $m=g(Y)$ almost surely. Conversely, if $X=g(Y)$ almost surely, then $X$ is already determined by the conditioning information, so $E[X\mid Y]=X$ almost surely and the conditional squared residual is zero. This proves both directions. $\square$

Common Mistake: Centering at the Wrong Mean

A tempting but incorrect calculation is to use $E[X^2\mid Y]-(E[X])^2$ as the conditional variance. This subtracts the square of the unconditional mean, even though conditional variance centers observations at their conditional mean. The correct expression is $E[X^2\mid Y]-(E[X\mid Y])^2$. For example, in the first worked example, the unconditional mean is $0.4(1)+0.6(4)=2.8$. Using it for the group $Y=0$ would give $2-2.8^2=-5.84$, an impossible negative variance. The conditional calculation instead gives $2-1^2=1$.

The conditional standard deviation is the nonnegative square root of the conditional variance, and it has the same units as $X$. The variance has squared units. Reporting both can be useful: variance is convenient for algebra, while standard deviation is often easier to interpret on the scale of the measured quantity.

Check Your Understanding

Use the definitions and calculations in this tutorial to answer the following questions.

If $E[X^2\mid Y=y]=18$ and $E[X\mid Y=y]=4$, what are the conditional variance and conditional standard deviation at that value of $y$?
For a conditional density $f_{X\mid Y}(x\mid y)$, write the integral that gives $E[X^2\mid Y=y]$.
Two groups have the same conditional mean but different conditional variances. What differs about their within-group distributions?
Why does $\operatorname{Var}(X\mid Y)=0$ almost surely imply that $X$ is determined by $Y$ almost surely?
Explain why subtracting $(E[X])^2$ from $E[X^2\mid Y]$ does not in general give conditional variance.