Variance of Differences and Cancellation

Why Differences Behave Differently

When two measurements move together, subtracting one from the other can remove some of their shared fluctuation. When they tend to move in opposite directions, subtraction can instead magnify the fluctuation. The precise amount is determined by covariance: the variance of a difference is not generally just the sum of the two variances.

This matters whenever the quantity of interest is a contrast, such as the change between two measurements on the same unit or the difference between two related outcomes. The marginal variability of each measurement is not enough to determine how variable their difference will be; their joint behavior matters too.

The Variance of a Difference

Write each variable as its mean plus a centered deviation: \(X-E[X]\) and \(Y-E[Y]\). Subtracting \(Y\) from \(X\) subtracts these deviations as well as subtracting the means. Since variance is the expected square of a centered variable, expanding that square produces a cross term. That cross term is exactly where covariance enters.

Proof. Let \(D=X-Y\). Its mean is \(E[D]=E[X]-E[Y]\), so \(D-E[D]=(X-E[X])-(Y-E[Y])\). Both centered variables are square-integrable under the stated conditions, and their product is integrable by the Cauchy–Schwarz inequality. We may therefore expand the square and take expectations:

The last equality follows directly from the definitions of variance and covariance. Thus the covariance term is not an adjustment added by convention; it is the cross term in the squared deviation. \(\square\)

How Covariance Changes the Variability

The two marginal variances contribute positively to the variability of a difference, while covariance is subtracted twice. Therefore, positive covariance reduces the variance relative to the sum of the marginal variances. Negative covariance increases it. If the covariance is zero, the variance of the difference equals that sum, but the formula itself requires no assumption of independence.

When both variances are positive, define the standard deviations by \(\sigma_X=\sqrt{\operatorname{Var}(X)}\) and \(\sigma_Y=\sqrt{\operatorname{Var}(Y)}\), and let \(\rho_{X,Y}\) be their correlation. By definition, \(\operatorname{Cov}(X,Y)=\rho_{X,Y}\sigma_X\sigma_Y\). Consequently,

This expression makes the role of dependence especially clear. With the same marginal standard deviations, a larger correlation means a smaller variance of the difference. If either variable has zero variance, its covariance with the other variable is zero, and the original variance formula applies directly; correlation need not be defined in that case.

A Bound on the Standard Deviation of a Difference

Covariance cannot be arbitrarily large in magnitude. The Cauchy–Schwarz bound for covariance states that \(|\operatorname{Cov}(X,Y)|\leq \sigma_X\sigma_Y\) for variables with finite second moments. This gives a useful range for the variability of their difference.

Proof. If both standard deviations are positive, substitute \(\operatorname{Cov}(X,Y)=\rho_{X,Y}\sigma_X\sigma_Y\) into the variance formula. Since \(-1\leq\rho_{X,Y}\leq1\),

Every term being compared is nonnegative. Taking square roots proves the stated inequalities. If either standard deviation is zero, its covariance with the other variable is zero; the variance of the difference is the square of the nonzero standard deviation, so the same bounds hold. \(\square\)

The bounds describe possible extremes. Perfect positive correlation can make the standard deviation of the difference as small as the absolute difference between the marginal standard deviations; if the standard deviations are equal, that lower bound is zero. Perfect negative correlation can make it as large as their sum.

Common Mistake: Adding Variances Automatically

A frequent error is to write \(\operatorname{Var}(X-Y)=\operatorname{Var}(X)+\operatorname{Var}(Y)\) without checking the covariance. That expression omits the cross term, so it is valid only when \(\operatorname{Cov}(X,Y)=0\). Merely having two different variables, or two measurements with different means, does not make their covariance zero. The correction is to retain the term \(-2\operatorname{Cov}(X,Y)\) unless the covariance has been shown to vanish.

The sign is worth checking before interpreting a result: positive covariance must lower the variance relative to the sum, and negative covariance must raise it. If a calculation gives the opposite conclusion, the sign in the formula has likely been mishandled.