Within-Group and Between-Group Covariance

Why Covariance Has Two Sources

Two variables can be associated for two distinct reasons. Within a group, larger values of one variable may tend to occur with larger or smaller values of the other. In addition, the group averages themselves may move together or in opposite directions. Overall covariance combines these sources. Separating them helps explain why a covariance can be small even when there is strong association inside the groups, or why the overall association can differ in sign from the association seen in some groups.

Let $G$ denote a grouping variable, such as a category with finitely many possible values. For each group, consider the conditional means of $X$ and $W$, and their conditional covariance. Assume $X$ and $W$ have finite second moments, so the expectations and covariances below exist. The same decomposition also applies to more general conditioning variables, provided the relevant conditional moments exist.

Definition: The within-group covariance is the expected conditional covariance, $E[\operatorname{Cov}(X,W\mid G)]$. The between-group covariance is the covariance of the conditional means, $\operatorname{Cov}(E[X\mid G],E[W\mid G])$. The conditional means and conditional covariance are functions of the group and therefore are random variables before the group is observed.

For a discrete grouping variable with group probabilities $P(G=g)$, these terms are weighted averages across groups. The within-group term averages the covariance in each group using that group's probability. The between-group term measures how the group-specific means vary together, again accounting for the group probabilities.

The Covariance Decomposition

Theorem: If $X$ and $W$ have finite second moments, then $$ \operatorname{Cov}(X,W) = E[\operatorname{Cov}(X,W\mid G)] + \operatorname{Cov}(E[X\mid G],E[W\mid G]). $$

The first term describes association after holding the group fixed. The second describes association that arises because some groups have higher or lower conditional means. Unlike a variance, a covariance can be negative, so either component can be positive, negative, or zero.

Start from the definition.
Write overall covariance as $E[XW]-E[X]E[W]$.

Condition the product.
Use the conditional covariance identity $E[XW\mid G]=\operatorname{Cov}(X,W\mid G)+E[X\mid G]E[W\mid G]$, then take expectations.

Apply the law of total expectation.
Replace $E[X]$ and $E[W]$ by $E[E[X\mid G]]$ and $E[E[W\mid G]]$.

Recognize the two components.
The remaining difference involving the conditional means is their covariance.

Here is the algebra behind the result. The conditional covariance identity follows from its definition, applied within each group:

\begin{aligned} \operatorname{Cov}(X,W) &=E[XW]-E[X]E[W]\\ &=E\!\left[E[XW\mid G]\right] -E\!\left[E[X\mid G]\right]E\!\left[E[W\mid G]\right]\\ &=E[\operatorname{Cov}(X,W\mid G)] +E\!\left[E[X\mid G]E[W\mid G]\right]\\ &\quad -E\!\left[E[X\mid G]\right]E\!\left[E[W\mid G]\right]\\ &=E[\operatorname{Cov}(X,W\mid G)] +\operatorname{Cov}(E[X\mid G],E[W\mid G]). \end{aligned}

The second equality uses the law of total expectation, and the last line uses the definition of covariance for the two random conditional means. This derivation also shows why the components add: conditioning separates $E[XW]$ into a conditional covariance and a product of conditional means.

Calculating the Components from Group Summaries

For a finite set of groups, write $p_g=P(G=g)$, $m_{X,g}=E[X\mid G=g]$, $m_{W,g}=E[W\mid G=g]$, and $c_g=\operatorname{Cov}(X,W\mid G=g)$. Then

E[\operatorname{Cov}(X,W\mid G)]=\sum_g p_g c_g, \qquad \operatorname{Cov}(E[X\mid G],E[W\mid G]) =\sum_g p_g m_{X,g}m_{W,g} -\left(\sum_g p_gm_{X,g}\right)\left(\sum_g p_gm_{W,g}\right).

The second formula is often safer than trying to reason informally about whether the group means “move together.” It calculates the covariance of the two group-mean variables using their probability-weighted means and products.

Worked Example: Opposing Group Means and Within-Group Covariance

A hypothetical portfolio is divided into groups A, B, and C with probabilities $0.2$, $0.5$, and $0.3$. Within A, $(X,W)$ is equally likely to be $(0,9)$ or $(2,7)$. Within B, it is equally likely to be $(3,3)$ or $(5,7)$. Within C, $X$ is equally likely to be 6 or 8, and $W$ independently is equally likely to be 1 or 3. Find the overall covariance and both components.

Method. Calculate each group's conditional means and covariance from its specified conditional distribution, then weight the within-group covariances and calculate the covariance of the group means.

In A, the means are $m_{X,A}=(0+2)/2=1$ and $m_{W,A}=(9+7)/2=8$. The deviations from these means are paired as $(-1,1)$ and $(1,-1)$, so $c_A=[(-1)(1)+(1)(-1)]/2=-1$.

In B, the means are $m_{X,B}=(3+5)/2=4$ and $m_{W,B}=(3+7)/2=5$. The paired deviations are $(-1,-2)$ and $(1,2)$, so $c_B=[(-1)(-2)+(1)(2)]/2=2$.

In C, the means are $m_{X,C}=7$ and $m_{W,C}=2$. Conditional independence gives $c_C=0$. The overall conditional covariance contribution is therefore

$$ 0.2(-1)+0.5(2)+0.3(0)=-0.2+1+0=0.8. $$

The overall means are $E[X]=0.2(1)+0.5(4)+0.3(7)=4.3$ and $E[W]=0.2(8)+0.5(5)+0.3(2)=4.7$. The between-group contribution is

\begin{aligned} &0.2(1-4.3)(8-4.7)+0.5(4-4.3)(5-4.7)+0.3(7-4.3)(2-4.7)\\ &=0.2(-3.3)(3.3)+0.5(-0.3)(0.3)+0.3(2.7)(-2.7)\\ &=-2.178-0.045-2.187=-4.410. \end{aligned}

Thus, $\operatorname{Cov}(X,W)=0.8-4.41=-3.61$. Check: The conditional values of $E[XW\mid G]$ are $7$, $22$, and $14$, so $E[XW]=0.2(7)+0.5(22)+0.3(14)+0.8=16.6$; hence $E[XW]-E[X]E[W]=16.6-(4.3)(4.7)=-3.61$. The check agrees. The common mistake this example avoids is assuming that positive average covariance within groups forces positive overall covariance; here the negative between-group contribution is larger in magnitude.

Reading the Two Contributions

A positive within-group covariance means that, on average across groups, above-group values of $X$ tend to accompany above-group values of $W$, or below-group values tend to accompany below-group values. A negative within-group covariance means the average within-group association is in the opposite direction. The word “average” matters: the expected conditional covariance can conceal differences between individual groups.

The between-group term has a different interpretation. It is positive when groups with high conditional means of $X$ tend also to have high conditional means of $W$. It is negative when groups with higher conditional means of one variable tend to have lower conditional means of the other. The between-group term is not the covariance within any one group.

Worked Example: Finding Conditional Covariances from Joint Probabilities

A hypothetical review classifies applicants into groups R and S with equal probabilities. Let $X$ indicate whether an applicant passes a quantitative assessment and $W$ indicate whether the same applicant passes a communication assessment; each indicator is 1 for a pass and 0 otherwise. In R, the conditional probabilities for $(X,W)=(1,1),(1,0),(0,1),(0,0)$ are $0.5,0.1,0.1,0.3$. In S, they are $0.1,0.3,0.3,0.3$. Find the overall covariance and decompose it.

Method. In each group, calculate the two conditional means and the conditional product expectation. Then use $\operatorname{Cov}(X,W\mid G)=E[XW\mid G]-E[X\mid G]E[W\mid G]$.

In R, $E[X\mid R]=0.5+0.1=0.6$ and $E[W\mid R]=0.5+0.1=0.6$. Since $XW=1$ only in the $(1,1)$ outcome, $E[XW\mid R]=0.5$. Thus $c_R=0.5-(0.6)(0.6)=0.14$.

In S, $E[X\mid S]=0.1+0.3=0.4$ and $E[W\mid S]=0.1+0.3=0.4$. Also $E[XW\mid S]=0.1$, so $c_S=0.1-(0.4)(0.4)=-0.06$. Weighting the conditional covariances gives

E[\operatorname{Cov}(X,W\mid G)] =0.5(0.14)+0.5(-0.06)=0.04.

The conditional means for $X$ are $0.6$ and $0.4$, as are those for $W$. Their overall means are both $0.5$, so the between-group covariance is

$$ 0.5(0.6-0.5)(0.6-0.5)+0.5(0.4-0.5)(0.4-0.5)=0.01. $$

The total covariance is $0.04+0.01=0.05$. Check: The unconditional probability of two passes is $0.5(0.5)+0.5(0.1)=0.3$, and each unconditional pass probability is $0.5$. Therefore $\operatorname{Cov}(X,W)=0.3-(0.5)(0.5)=0.05$, as obtained by decomposition. The between-group contribution is positive because the two group means are both higher in R than in S.

Worked Example: Conditional Association and Group-Level Association

A hypothetical service team has two equally likely workload groups, L and H. Conditional on L, $(X,W)$ is equally likely to be $(1,-0.5)$ or $(3,2.5)$. Conditional on H, it is equally likely to be $(5,5.5)$ or $(7,4.5)$. Find the within-group, between-group, and overall covariance.

Method. Compute the moments from each two-point conditional distribution and then apply the decomposition. The values in this hypothetical model are deliberately chosen to show that a negative covariance in one group need not determine the overall sign.

For L, the conditional means are $m_{X,L}=(1+3)/2=2$ and $m_{W,L}=(-0.5+2.5)/2=1$. The two products $XW$ are $-0.5$ and $7.5$, giving $E[XW\mid L]=3.5$. Thus $c_L=3.5-(2)(1)=1.5$.

For H, the conditional means are $m_{X,H}=(5+7)/2=6$ and $m_{W,H}=(5.5+4.5)/2=5$. The products are $27.5$ and $31.5$, giving $E[XW\mid H]=29.5$. Thus $c_H=29.5-(6)(5)=-0.5$. The expected within-group covariance is $0.5(1.5)+0.5(-0.5)=0.5$.

The overall conditional-mean averages are $E[E[X\mid G]]=0.5(2)+0.5(6)=4$ and $E[E[W\mid G]]=0.5(1)+0.5(5)=3$. The between-group covariance is

$$ 0.5(2-4)(1-3)+0.5(6-4)(5-3) =0.5(4)+0.5(4)=4. $$

Therefore, the overall covariance is $0.5+4=4.5$. Check: The unconditional product expectation is $0.5(3.5)+0.5(29.5)=16.5$, while $E[X]E[W]=(4)(3)=12$. The direct covariance is $16.5-12=4.5$. The negative within-group covariance in H is outweighed by the positive association between the group means.

Common Exam Trap: Omitting the Within-Group Term

The most common error is to calculate only the covariance of the conditional means and report it as the overall covariance. That calculation answers a different question: it measures the between-group component only. For instance, in the joint-probability example, reporting $0.01$ as the overall covariance omits the within-group contribution $0.04$. The correct calculation is $0.04+0.01=0.05$.

Another error is to average the conditional covariances without weighting them by their group probabilities. If group $g$ occurs with probability $p_g$, its contribution is $p_gc_g$, not $c_g$ by itself. Finally, do not assume the two components have the same sign: one can reinforce the other or partly cancel it.

Key takeaway: Overall covariance is the expected within-group covariance plus the covariance of the group-specific means. Calculate both terms, preserve the group probabilities, and interpret their signs separately before combining them.

Check Your Understanding

For each problem, identify the within-group contribution and the between-group contribution before giving the overall covariance.

A grouping variable has two groups with probabilities $0.25$ and $0.75$. Their conditional means $(E[X\mid G],E[W\mid G])$ are $(2,7)$ and $(6,3)$, and their conditional covariances are $1$ and $-2$. Find the overall covariance. (Answer: $-2.75$)
Two groups have probabilities $0.3$ and $0.7$. Their conditional means are $(0,2)$ and $(5,4)$, and conditional covariances are $0$ and $1$. Find both components and the overall covariance. (Answer: Within $=0.7$, between $=2.1$, overall $=2.8$)
Two equally likely groups have conditional joint probabilities for $(X,W)=(1,1),(1,0),(0,1),(0,0)$ as follows: group A has $(0.4,0.2,0.2,0.2)$, and group B has $(0.2,0.2,0.2,0.4)$. Find the overall covariance. (Answer: $0.01$)
Three groups have probabilities $0.2,0.3,0.5$, conditional means $(1,4),(3,2),(5,0)$, and conditional covariances $0.5,1,-0.5$, respectively. Find both components and the overall covariance. (Answer: Within $=0.15$, between $=-2.44$, overall $=-2.29$)
Two equally likely groups have conditional means $(1,2)$ and $(4,5)$. Suppose $X$ and $W$ are conditionally independent in each group. What is their overall covariance? (Answer: $2.25$)

What You'll Learn