Why Every Pair Matters
When a quantity is formed by adding or weighting several random variables, its covariance with another such quantity depends on how each variable in the first sum covaries with each variable in the second. Looking only at matching variables, or assuming that different variables have zero covariance, can give a wrong answer. The systematic approach is to expand the sums and account for every pair.
This is useful in sampling problems, measurement settings, and any model built from several sources of random variation. The formulas below require only finite second moments; the variables need not be independent, and the two weighted sums may involve overlapping collections of variables.
The finite second-moment condition ensures that the means, variances, and covariances used here are finite. Covariance is defined by \(\operatorname{Cov}(U,V)=E[(U-E[U])(V-E[V])]\). In particular, a covariance between \(X_i\) and \(Y_j\) is included whether the variables are the same random variable or different ones.
The Double-Sum Formula
The result follows by centering each sum and distributing the product. All sums here are finite, so linearity of expectation applies directly, with no interchange of infinite sums or limiting operations.
Linearity of expectation gives \(A-E[A]=\sum_{i=1}^{m}a_i(X_i-E[X_i])\) and \(B-E[B]=\sum_{j=1}^{k}b_j(Y_j-E[Y_j])\).
Multiplying these finite sums gives one term for every ordered pair of indices \((i,j)\).
Each expected centered product is a covariance, and the fixed coefficients can be taken outside expectation.
The double sum has \(mk\) terms before any simplification. Even when \(X_i\) and \(Y_j\) come from the same collection, the formula uses an ordered pairing: an index from the first weighted sum is paired with an index from the second. Symmetry of covariance, \(\operatorname{Cov}(U,V)=\operatorname{Cov}(V,U)\), can make some terms equal, but it does not make them disappear.
Worked Example: Two Weighted Sums From a Joint Distribution
Let \(X\) and \(Y\) take values in \(\{-1,1\}\), with joint probabilities \(P(X=1,Y=1)=0.4\), \(P(X=1,Y=-1)=0.2\), \(P(X=-1,Y=1)=0.3\), and \(P(X=-1,Y=-1)=0.1\). Find the covariance of \(A=2X-Y\) and \(B=X+3Y\), showing all four pairwise terms.
First compute the moments needed for the covariances. The marginal probabilities give \(P(X=1)=0.6\) and \(P(Y=1)=0.7\), so
Because \(X^2=Y^2=1\), their variances are
The product \(XY\) equals \(1\) when the signs agree and \(-1\) when they differ. Thus \(E[XY]=0.4+0.1-0.2-0.3=0\), and
Now use the coefficients of \(A\), namely \(2,-1\), and of \(B\), namely \(1,3\). The four pairwise contributions are
The third term is positive because its coefficient product is negative and \(\operatorname{Cov}(Y,X)=-0.08\). Keeping the coefficient signs visible is a useful check on this calculation.
Two Weighted Sums From the Same Collection
A common case uses \(X_1,\ldots,X_n\) in both sums. If \(A=\sum_{i=1}^{n}a_iX_i\) and \(B=\sum_{j=1}^{n}b_jX_j\), then
To obtain the variance expression, set \(b_j=a_j\) in the double sum. The \(n\) diagonal terms, for which \(i=j\), are \(a_i^2\operatorname{Cov}(X_i,X_i)=a_i^2\operatorname{Var}(X_i)\). For each distinct pair \(i<j\), the double sum contains both \(a_i a_j\operatorname{Cov}(X_i,X_j)\) and \(a_j a_i\operatorname{Cov}(X_j,X_i)\). These are equal because the coefficients commute and covariance is symmetric. Their sum is \(2a_i a_j\operatorname{Cov}(X_i,X_j)\), which proves the stated expression. \(\square\)
The doubled-pair form is convenient for a variance because it groups equal terms. For covariance between two different weighted sums, the two directions need not have the same coefficient product; it is safest to retain the full double sum until each term is identified.
Worked Example: Every Pair in a Three-Variable Covariance
Let \(U,V,W\) be mutually independent random variables, each taking \(-1\) and \(1\) with probability \(1/2\). Define \(X_1=U+V\), \(X_2=U-W\), and \(X_3=V+W\). Find \(\operatorname{Cov}(A,B)\) for \(A=2X_1-X_2+X_3\) and \(B=X_1+3X_2-2X_3\).
Each of \(U,V,W\) has mean zero and variance one. Independence therefore makes covariances between distinct source variables zero. Expanding the three variances gives \(\operatorname{Var}(X_1)=\operatorname{Var}(X_2)=\operatorname{Var}(X_3)=2\). The cross-covariances are
For clarity, the coefficient products and resulting contributions for all nine ordered pairs are listed here. The row selects a variable from \(A\); the column selects one from \(B\).
| Pair | Coefficient product | Covariance | Contribution |
|---|---|---|---|
| \((X_1,X_1)\) | \(2(1)=2\) | 2 | 4 |
| \((X_1,X_2)\) | \(2(3)=6\) | 1 | 6 |
| \((X_1,X_3)\) | \(2(-2)=-4\) | 1 | -4 |
| \((X_2,X_1)\) | \((-1)(1)=-1\) | 1 | -1 |
| \((X_2,X_2)\) | \((-1)(3)=-3\) | 2 | -6 |
| \((X_2,X_3)\) | \((-1)(-2)=2\) | -1 | -2 |
| \((X_3,X_1)\) | \(1(1)=1\) | 1 | 1 |
| \((X_3,X_2)\) | \(1(3)=3\) | -1 | -3 |
| \((X_3,X_3)\) | \(1(-2)=-2\) | 2 | -4 |
Adding the contributions gives \(\operatorname{Cov}(A,B)=4+6-4-1-6-2+1-3-4=-9\). As a check, direct substitution gives \(A=U+3V+2W\) and \(B=4U-V-5W\). Independence of \(U,V,W\) then yields \(\operatorname{Cov}(A,B)=1(4)+3(-1)+2(-5)=-9\), agreeing with the nine-term calculation.
Shared Variation and Cancellation
Covariance terms can reinforce or offset the variance contributions from individual variables. This is especially clear when several measurements share a random component. In the next example, the common component creates positive covariances among all four variables, but the negative coefficient in the weighted sum offsets some of those contributions.
Worked Example: A Weighted Sum With a Shared Component
Let \(U,\varepsilon_1,\varepsilon_2,\varepsilon_3,\varepsilon_4\) be mutually independent, mean-zero random variables, with \(\operatorname{Var}(U)=4\) and \(\operatorname{Var}(\varepsilon_i)=i\) for \(i=1,2,3,4\). Set \(X_i=U+\varepsilon_i\) and \(S=X_1+X_2-X_3+2X_4\). Compute \(\operatorname{Var}(S)\) by accounting for every diagonal and off-diagonal term.
Since \(U\) is the only shared component, \(\operatorname{Var}(X_i)=4+i\), giving variances \(5,6,7,8\). For distinct \(i,j\), independence of the errors and of \(U\) from the errors gives \(\operatorname{Cov}(X_i,X_j)=\operatorname{Var}(U)=4\). The coefficients are \(1,1,-1,2\).
The four diagonal contributions are \(1^2(5)=5\), \(1^2(6)=6\), \((-1)^2(7)=7\), and \(2^2(8)=32\), with total \(50\). The six distinct off-diagonal contributions before doubling are
These correspond, in order, to pairs \((1,2),(1,3),(1,4),(2,3),(2,4),(3,4)\). Their sum is \(4\), so the off-diagonal contribution to the variance is \(2(4)=8\). Therefore \(\operatorname{Var}(S)=50+8=58\).
A direct check uses \(S=3U+\varepsilon_1+\varepsilon_2-\varepsilon_3+2\varepsilon_4\). Its variance is \(3^2(4)+1+2+3+2^2(4)=36+22=58\). In particular, the negative coefficient does not make its diagonal variance contribution negative; it is squared there, while its sign matters in the cross-covariance terms.
Common Mistakes and a Practical Check
- Keeping only matching indices. The variance of a sum includes covariances between distinct variables, not just their individual variances. Include every pair before simplifying.
- Assuming zero covariance without a reason. Distinct variables need not be uncorrelated. Independence is one condition that can justify zero covariance when second moments exist, but it is not an automatic property of different variables.
- Dropping a minus sign. A coefficient enters linearly in each covariance term. In a variance formula, a negative coefficient is squared on the diagonal but remains signed in off-diagonal terms.
- Doubling every term in a covariance of two different sums. The factor of two applies when the two symmetric terms are equal, as in the variance of one weighted sum. For two different sums, use the full double sum instead.
A reliable calculation has three checks: write the coefficients of both sums, enumerate all index pairs, and verify signs in the coefficient products. When a model has a simpler representation in terms of independent sources, a direct calculation in those sources provides a useful independent check.
Check Your Understanding
Use the double-sum formula, keeping the coefficient products visible.
- For \(A=3X_1-X_2\) and \(B=2Y_1+Y_2\), how many pairwise covariance terms appear before simplification?
- Write the full variance expansion for \(2X_1-X_2+X_3\), using variances and covariances of the \(X_i\).
- In a variance expansion, why does a negative coefficient contribute positively to its diagonal term but may contribute negatively to a cross term?
- Under what condition may a covariance term between two distinct variables be set to zero? Give one standard sufficient condition.
- Explain why the two terms for a distinct pair combine into a factor of two in a variance formula, but should not generally be combined that way for the covariance of two different sums.