Total Variance and Conditional Variance

Variance Within Groups and Between Groups

When information about a random variable is available, its remaining uncertainty can be measured by a conditional variance. This is itself a random variable: its value may depend on the information observed. The law of total variance connects this conditional uncertainty to the ordinary variance. It says that total variation has two parts: average variation remaining within groups, and variation between the groups’ conditional means.

Throughout, \((\Omega,\mathcal{F},\mathbb{P})\) is a probability space, and \(X:\Omega\to\mathbb{R}\) is a real-valued random variable with \(\mathbb{E}[X^2]<\infty\). Thus \(X\) is square-integrable, and \(\operatorname{Var}(X)=\mathbb{E}[(X-\mathbb{E}X)^2]\) is finite. Let \(\mathcal{G}\) be a sub-sigma-algebra of \(\mathcal{F}\), representing the information being conditioned on. Conditional expectations are defined up to almost-sure equality; all identities between them below hold \(\mathbb{P}\)-a.s.

The assumption \(\mathbb{E}[X^2]<\infty\) ensures that this definition is meaningful: \(Y\) is square-integrable by conditional Jensen’s inequality, so \((X-Y)^2\) is integrable. The conditional variance records the squared error after subtracting the conditional mean, rather than the squared error after subtracting the unconditional mean.

A Formula in Terms of Conditional Moments

Proof. Let \(Y=\mathbb{E}[X\mid\mathcal{G}]\). Expanding the square gives \((X-Y)^2=X^2-2XY+Y^2\). Each term is integrable: \(X^2\) and \(Y^2\) are integrable, and \(XY\) is integrable by the Cauchy–Schwarz inequality. By linearity of conditional expectation and the property of pulling out a \(\mathcal{G}\)-measurable factor, \(\mathbb{E}[XY\mid\mathcal{G}]=Y\mathbb{E}[X\mid\mathcal{G}]=Y^2\) and \(\mathbb{E}[Y^2\mid\mathcal{G}]=Y^2\). Therefore \(\mathbb{E}[(X-Y)^2\mid\mathcal{G}]=\mathbb{E}[X^2\mid\mathcal{G}]-2Y^2+Y^2=\mathbb{E}[X^2\mid\mathcal{G}]-Y^2\) almost surely. \(\square\)

In particular, the difference between the conditional second moment and the square of the conditional mean is nonnegative almost surely. This also follows directly from the definition as a conditional expectation of a nonnegative random variable. The formula is useful for calculations, but the definition explains what the quantity measures.

The Law of Total Variance

Proof. Put \(Y=\mathbb{E}[X\mid\mathcal{G}]\), \(Z=X-Y\), and \(m=\mathbb{E}[X]\). The tower property gives \(\mathbb{E}[Y]=m\) and \(\mathbb{E}[Z\mid\mathcal{G}]=\mathbb{E}[X\mid\mathcal{G}]-Y=0\) almost surely. By the definition of conditional variance and the tower property, \(\mathbb{E}[\operatorname{Var}(X\mid\mathcal{G})]=\mathbb{E}[\mathbb{E}[Z^2\mid\mathcal{G}]]=\mathbb{E}[Z^2]\). All these expectations are finite because \(X\) and \(Y\) are square-integrable.

Since \(X-m=Z+(Y-m)\), expanding the square and taking expectations gives \(\operatorname{Var}(X)=\mathbb{E}[Z^2]+2\mathbb{E}[Z(Y-m)]+\mathbb{E}[(Y-m)^2]\). The cross term is zero. Indeed, \(Y-m\) is \(\mathcal{G}\)-measurable, and the conditional-expectation pulling-out property gives \(\mathbb{E}[Z(Y-m)]=\mathbb{E}[(Y-m)\mathbb{E}[Z\mid\mathcal{G}]]=0\). This use of the property is valid for the possibly unbounded factor \(Y-m\), since both factors are square-integrable and their product is integrable. The last term is \(\operatorname{Var}(Y)\), because \(\mathbb{E}[Y]=m\). Substituting \(\mathbb{E}[Z^2]=\mathbb{E}[\operatorname{Var}(X\mid\mathcal{G})]\) proves the identity. \(\square\)

The identity includes degenerate cases. If \(\mathcal{G}\) contains no information beyond events of probability zero or one, then \(Y=m\) almost surely and the between-group term is zero. If \(X\) is \(\mathcal{G}\)-measurable, then \(Y=X\) almost surely, the conditional variance is zero, and all variation is in the conditional mean.

Finite Partitions and Worked Calculations

For a finite partition \(\{A_1,\ldots,A_k\}\) with \(\mathbb{P}(A_i)>0\) for every \(i\), let \(\mathcal{G}\) be the sigma-algebra generated by the partition. On \(A_i\), the conditional mean is \(\mathbb{E}[X\mid A_i]=\mathbb{E}[X\chi_{A_i}]/\mathbb{P}(A_i)\). The conditional variance on that cell is \(\mathbb{E}[X^2\mid A_i]-\mathbb{E}[X\mid A_i]^2\). Values assigned on zero-probability cells do not affect any almost-sure statement or expectation; the formulas using division by \(\mathbb{P}(A_i)\) are used only for positive-probability cells.

A Common Mistake: Omitting Between-Group Variation

A common error is to calculate the conditional variances and then report their average as the unconditional variance. That omits the variation in conditional means. In the finite-mixture example, the average conditional variance is \(1\), but the unconditional variance is \(17/9\). The missing term is the variance \(8/9\) of the group-specific conditional means.

The conditional-moment formula also prevents a related mistake: \(\operatorname{Var}(X\mid\mathcal{G})\) is not simply \(\mathbb{E}[X^2\mid\mathcal{G}]\). One must subtract the square of the conditional mean, not the conditional mean itself. The law of total variance is useful precisely because it keeps the two sources of variation separate. For a finite partition, it is often efficient to compute each cell’s mean and variance first, then combine the within-cell variances and the between-cell means.