Tutorials › Statistical Theory I - Probability and Distribution Theory › The Law of Total Covariance

Home  /  Tutorials  /  Variance and Covariance Structure

Variance and Covariance Structure · Tutorial 370 of 994

The Law of Total Covariance

Learn to derive the total covariance decomposition and use it to calculate how conditional dependence and changing conditional means jointly determine overall covariance.

Intermediate 9 min read

What You'll Learn

  • Define conditional covariance given a random variable using conditional expectations.
  • State the finite-moment conditions for the law of total covariance.
  • Prove the decomposition by separating conditional residuals from conditional means.
  • Calculate within-group and between-group covariance in a discrete mixture.
  • Apply the law to Bernoulli and Gaussian hierarchical models.
  • Recognize why averaging conditional covariances alone may not give the overall covariance.

Why Decompose Covariance?

A pair of variables can be associated for two distinct reasons. Within a group, the variables may tend to rise or fall together. Across groups, their conditional means may change in coordinated ways. A single overall covariance combines both sources, so it can hide how the association arises.

The law of total covariance makes this distinction precise. Given a grouping or conditioning variable \(Y\), it expresses the covariance of \(X\) and \(Z\) as the average covariance remaining within levels of \(Y\), plus the covariance of their conditional means. The result applies to discrete, continuous, and more general conditioning variables.

Conditional Covariance and the Decomposition

Definition: Suppose \(X\) and \(Z\) have finite second moments, and let \(\sigma(Y)\) denote the information generated by a random variable or random vector \(Y\). The conditional covariance of \(X\) and \(Z\) given \(Y\) is the \(\sigma(Y)\)-measurable random variable \( \operatorname{Cov}(X,Z\mid Y)=E[(X-E[X\mid Y])(Z-E[Z\mid Y])\mid Y] \). Equivalently, almost surely, \( \operatorname{Cov}(X,Z\mid Y)=E[XZ\mid Y]-E[X\mid Y]E[Z\mid Y] \).

Conditional expectations given \(Y\) are defined up to almost-sure equality, and so is conditional covariance. When \(Y\) is discrete, its value at a particular group is the ordinary covariance under the conditional distribution within that group. When \(Y\) is not discrete, the conditional covariance is still a random variable determined by \(Y\); it need not be described by conditioning on an event \(Y=y\) of positive probability.

Theorem (Law of Total Covariance): If \(E[X^2]<\infty\) and \(E[Z^2]<\infty\), then \( \operatorname{Cov}(X,Z)=E[\operatorname{Cov}(X,Z\mid Y)]+\operatorname{Cov}(E[X\mid Y],E[Z\mid Y]) \).

The first term is the within-group contribution: the conditional covariance averaged over the distribution of \(Y\). The second is the between-group contribution: the covariance between the two conditional means as \(Y\) varies. Either term may be positive, negative, or zero.

Proof of the Law

Write \(m_X=E[X\mid Y]\) and \(m_Z=E[Z\mid Y]\). The key fact is that a conditional residual has zero expectation when multiplied by a square-integrable quantity determined by \(Y\).

Lemma: If \(X\) is square-integrable, \(m_X=E[X\mid Y]\), and \(A\) is square-integrable and \(\sigma(Y)\)-measurable, then \(E[(X-m_X)A]=0\).

Proof. For bounded \(\sigma(Y)\)-measurable \(A\), the defining property of conditional expectation gives \( E[(X-m_X)A]=E[E[X-m_X\mid Y]A]=0 \), because \(E[X-m_X\mid Y]=0\) almost surely. For general square-integrable \(A\), define \(A_k=\max(-k,\min(A,k))\). Each \(A_k\) is bounded and \(\sigma(Y)\)-measurable, so \(E[(X-m_X)A_k]=0\). Also \(A_k\to A\) in \(L^2\), and \(X-m_X\) is square-integrable. By Cauchy–Schwarz, \( |E[(X-m_X)(A-A_k)]|\leq (E[(X-m_X)^2])^{1/2}(E[(A-A_k)^2])^{1/2}\to 0 \). Thus \(E[(X-m_X)A]=0\). \(\square\)

Proof of the theorem. Conditional expectation is a contraction in \(L^2\), so \(m_X\) and \(m_Z\) are square-integrable. All products below are integrable by Cauchy–Schwarz. Decompose each centered variable into a conditional residual and a centered conditional mean:

$$ X-E[X]=(X-m_X)+(m_X-E[X]), \qquad Z-E[Z]=(Z-m_Z)+(m_Z-E[Z]). $$

Multiply these expressions and take expectations. The two cross terms vanish by the lemma: use \(A=m_Z-E[Z]\) for the term involving \(X-m_X\), and \(A=m_X-E[X]\) for the term involving \(Z-m_Z\). The remaining terms are

$$ \begin{aligned} \operatorname{Cov}(X,Z) &=E[(X-m_X)(Z-m_Z)]\\ &\quad+E[(m_X-E[X])(m_Z-E[Z])]. \end{aligned} $$

By the tower property of conditional expectation, the first expectation is \(E[\operatorname{Cov}(X,Z\mid Y)]\). The second is \(\operatorname{Cov}(m_X,m_Z)\), since \(E[m_X]=E[X]\) and \(E[m_Z]=E[Z]\). Substitution proves the stated decomposition. \(\square\)

Worked Examples

Worked Example: A Discrete Mixture With Different Within-Group Covariances

Let \(P(Y=0)=3/4\) and \(P(Y=1)=1/4\). Conditional on \(Y=0\), let \((X,Z)\) equal \((0,0)\) or \((2,2)\), each with probability \(1/2\). Conditional on \(Y=1\), let \((X,Z)\) equal \((0,2)\) or \((2,0)\), each with probability \(1/2\). Find \(\operatorname{Cov}(X,Z)\) using the decomposition.

In either group, \(E[X\mid Y]=1\) and \(E[Z\mid Y]=1\). For \(Y=0\), \(E[XZ\mid Y=0]=(0+4)/2=2\), so \(\operatorname{Cov}(X,Z\mid Y=0)=2-(1)(1)=1\). For \(Y=1\), \(E[XZ\mid Y=1]=0\), so \(\operatorname{Cov}(X,Z\mid Y=1)=0-(1)(1)=-1\).

The average conditional covariance is therefore \( (3/4)(1)+(1/4)(-1)=1/2 \). Both conditional means are constant, so their covariance is zero. Hence the law gives \(\operatorname{Cov}(X,Z)=1/2\).

As a direct check, \(E[X]=E[Z]=1\), while \(E[XZ]=(3/4)(2)+(1/4)(0)=3/2\). Thus \(\operatorname{Cov}(X,Z)=E[XZ]-E[X]E[Z]=3/2-1=1/2\), as obtained from the decomposition.

Worked Example: A Bernoulli Hierarchical Model

Let \(Y\sim\operatorname{Bernoulli}(0.3)\). Given \(Y\), construct errors \((\varepsilon,\eta)\) with conditional probabilities \(3/8\) each for \((1,1)\) and \((-1,-1)\), and \(1/8\) each for \((1,-1)\) and \((-1,1)\). Set \(X=Y+\varepsilon\) and \(Z=2Y+\eta\). Find their covariance.

In each group, \(E[\varepsilon\mid Y]=E[\eta\mid Y]=0\). Their conditional product has expectation \(1\) for the equal-sign outcomes, whose total probability is \(3/4\), and \(-1\) for the opposite-sign outcomes, whose total probability is \(1/4\). Therefore \( \operatorname{Cov}(X,Z\mid Y)=E[\varepsilon\eta\mid Y]=(3/4)-(1/4)=1/2 \).

The conditional means are \(E[X\mid Y]=Y\) and \(E[Z\mid Y]=2Y\). Since \(\operatorname{Var}(Y)=0.3(0.7)=0.21\), their covariance is \(2\operatorname{Var}(Y)=0.42\). Consequently, \( \operatorname{Cov}(X,Z)=1/2+0.42=0.92 \).

For a direct check, \(E[X]=0.3\) and \(E[Z]=0.6\). Expanding \(XZ=(Y+\varepsilon)(2Y+\eta)\), the cross terms have expectation zero, while \(2E[Y^2]=0.6\) and \(E[\varepsilon\eta]=0.5\). Thus \(E[XZ]=1.1\), and \(\operatorname{Cov}(X,Z)=1.1-(0.3)(0.6)=0.92\).

Worked Example: A Gaussian Hierarchical Model

Let \(Y\sim N(0,1)\). Independently of \(Y\), let \((U,V)\) be jointly normal with means zero, variances one, and covariance \(-0.4\). This is a valid covariance specification because the matrix with diagonal entries \(1\) and off-diagonal entries \(-0.4\) has determinant \(1-0.16=0.84>0\). Set \(X=Y+U\) and \(Z=Y+V\). Find their covariance.

Given \(Y\), the only randomness in \(X\) and \(Z\) comes from \(U\) and \(V\). Thus \(E[X\mid Y]=Y\), \(E[Z\mid Y]=Y\), and \(\operatorname{Cov}(X,Z\mid Y)=\operatorname{Cov}(U,V)=-0.4\). The mean of the conditional covariance is \(-0.4\), and the covariance of the conditional means is \(\operatorname{Var}(Y)=1\). Therefore \(\operatorname{Cov}(X,Z)=-0.4+1=0.6\).

The direct calculation agrees: \(E[X]=E[Z]=0\), and \(X,Z\) each contain the shared term \(Y\). Independence of \(Y\) from \((U,V)\) gives \( \operatorname{Cov}(X,Z)=\operatorname{Var}(Y)+\operatorname{Cov}(U,V)=1-0.4=0.6 \). Here the within-group association is negative, but shared variation in the conditional means more than offsets it.

Interpretation and a Common Pitfall

The decomposition is useful whenever a distribution has a natural hierarchy: observations within treatment arms, sites, families, or latent classes can have one pattern of covariance, while differences in group means add another. It helps distinguish association that persists after conditioning from association induced by variation across groups. The same calculation works whether the conditional distributions are specified directly or are part of a hierarchical model.

Common mistake: averaging the conditional covariance and calling it the overall covariance. This omits \(\operatorname{Cov}(E[X\mid Y],E[Z\mid Y])\). For example, in the Bernoulli model above, the average conditional covariance is only \(0.5\), whereas the overall covariance is \(0.92\). The missing \(0.42\) comes from the coordinated changes in the two conditional means. Conversely, the between-group term alone is not generally enough: the discrete example has zero covariance between conditional means but a nonzero overall covariance.

Key takeaway: Overall covariance equals average within-condition covariance plus covariance between conditional means. To use the identity, check that both variables have finite second moments and calculate both contributions.

Check Your Understanding

Use the decomposition to answer each question. In every case, identify both contributions.

  1. If \(\operatorname{Cov}(X,Z\mid Y)=0\) almost surely, what does the law imply about \(\operatorname{Cov}(X,Z)\)?
  2. Suppose the conditional means of \(X\) and \(Z\) are constant in \(Y\), while the average conditional covariance is \(-0.6\). Find the overall covariance.
  3. In the discrete mixture example, why is the overall covariance not the covariance of the conditional means?
  4. For the Gaussian hierarchical model, which contribution is negative, and which contribution makes the overall covariance positive?
  5. What moment condition in the theorem guarantees that the covariances and products in the decomposition are integrable?

Previous

Next

Back to Tutorials