Conditional Variance Decomposition

Why Decompose Variance?

Variance measures the spread of a random variable around its mean. When information is available, part of that spread may be explained by what is known, while the remaining spread reflects variability even after conditioning on that information. Conditional variance makes this distinction precise.

The resulting decomposition is useful whenever data or outcomes fall into groups. One term measures average variability within the groups; the other measures variability between their conditional means. Both terms are nonnegative, so the decomposition also identifies exactly how much of the total variance is attributable to each source.

Definition and Basic Formula

Let \(X\) be a real-valued random variable with finite second moment, so that \(\mathbb{E}[X^2]<\infty\), and let \(\mathcal{G}\) be a sigma-algebra. The conditional expectation \(\mathbb{E}[X\mid\mathcal{G}]\) is square-integrable. The conditional variance is defined by centering \(X\) at this conditional expectation and then taking the conditional expectation of the squared deviation.

It is nonnegative almost surely, since it is the conditional expectation of a nonnegative random variable. Expanding the square gives a useful equivalent expression:

To verify the formula, put \(M=\mathbb{E}[X\mid\mathcal{G}]\). By the pull-out property, \(\mathbb{E}[XM\mid\mathcal{G}]=M\mathbb{E}[X\mid\mathcal{G}]=M^2\), and \(\mathbb{E}[M^2\mid\mathcal{G}]=M^2\). Therefore, conditioning the expansion \(X^2-2XM+M^2\) gives \(\mathbb{E}[X^2\mid\mathcal{G}]-M^2\), as claimed.

When \(\mathcal{G}\) represents known information, \(\mathbb{E}[X\mid\mathcal{G}]\) is the conditional center, and \(\operatorname{Var}(X\mid\mathcal{G})\) is the remaining squared spread around that center. Both are random variables: their values can depend on what the conditioning information reveals.

Conditional Variance Decomposition

Suppose \(\mathcal{G}\subseteq\mathcal{H}\), so that \(\mathcal{H}\) represents at least as much information as \(\mathcal{G}\). The decomposition compares the variation remaining after conditioning on the coarser information with two contributions: expected variation after the finer information is known, and variation in the finer conditional mean as seen from the coarser information.

Proof. Write \(U=\mathbb{E}[X\mid\mathcal{H}]\) and \(V=\mathbb{E}[X\mid\mathcal{G}]\). The tower property gives \(\mathbb{E}[U\mid\mathcal{G}]=V\). Decompose the deviation from the coarser conditional mean as \(X-V=(X-U)+(U-V)\), then square and condition on \(\mathcal{G}\).

All terms are integrable: \(X,U,V\) are square-integrable, and products of square-integrable random variables are integrable. Since \(U-V\) is \(\mathcal{H}\)-measurable, the pull-out property and \(\mathbb{E}[X-U\mid\mathcal{H}]=0\) imply \(\mathbb{E}[(X-U)(U-V)\mid\mathcal{H}]=0\). Applying the tower property shows that the cross term conditioned on \(\mathcal{G}\) is also zero.

For the first term, apply the tower property to condition first on \(\mathcal{H}\). Its inner conditional expectation is \(\operatorname{Var}(X\mid\mathcal{H})\). The last term is the conditional variance of \(U\) given \(\mathcal{G}\), because \(\mathbb{E}[U\mid\mathcal{G}]=V\). Substitution proves the identity:

The Law of Total Variance

The unconditional identity is a direct special case, but the choice of sigma-algebras matters. In the conditional theorem, take the coarser sigma-algebra \(\mathcal{G}=\{\varnothing,\Omega\}\), and take \(\mathcal{H}\) to be an arbitrary sigma-algebra. Conditioning on the trivial sigma-algebra gives constant random variables, equal almost surely to their expectations. Thus the conditional decomposition becomes the law of total variance.

Proof. Apply the conditional variance decomposition with the trivial sigma-algebra as \(\mathcal{G}\) and the given \(\mathcal{H}\) as the finer sigma-algebra. The left side is the constant \(\operatorname{Var}(X)\); the first term on the right is the constant \(\mathbb{E}[\operatorname{Var}(X\mid\mathcal{H})]\); and the second is the constant variance of \(\mathbb{E}[X\mid\mathcal{H}]\). Equating these constants gives the result.

The first term is the expected conditional variance, or average within-group variation. The second is the variance of the conditional mean, or between-group variation. The formula is also a useful check on calculations: omitting either contribution generally understates the total variance.

Discrete Calculations

For a finite partition, conditioning is especially concrete. The conditional expectation is constant on each partition cell with positive probability, and equals the average of \(X\) on that cell. The conditional variance on that cell is the average squared deviation from that cell's conditional mean. Cells of probability zero do not affect any of these identities; conditional expectations and variances are specified only up to almost-sure equality.

Interpretation and a Common Pitfall

The decomposition separates two kinds of uncertainty without asserting that either component is constant. In the conditional identity, each side is a \(\mathcal{G}\)-measurable random variable: the equality holds almost surely, cell by cell when conditioning on a finite partition. In the unconditional identity, taking expectations averages the within-group component, while the variance of the conditional mean quantifies differences between groups.

A common error is to use \(\mathbb{E}[X\mid\mathcal{H}]\) in the between-group term but forget that its variance is taken with respect to the original probability measure. Another is to confuse \(\mathbb{E}[\operatorname{Var}(X\mid\mathcal{H})]\) with \(\operatorname{Var}(X\mid\mathcal{H})\): the former is a number, while the latter is generally a random variable. Keep the levels of conditioning visible, and check that both terms have the same status as the left side.

Square-integrability is important: it ensures that the variances and the products in the proof are integrable. The identities are statements about equivalence classes up to almost-sure equality, as is standard for conditional expectations. They do not require a particular pointwise choice of versions.