Why Contrast Standard Errors Depend on Covariance

Why a Difference Has Its Own Uncertainty

A regression analysis may estimate two coefficients, and a research question may concern how far apart they are. The estimated difference is easy to calculate: subtract one estimate from the other. Its uncertainty is not obtained by subtracting their standard errors. The reason is that estimates from the same regression are generally related: when one estimate varies, the other may tend to vary with it or in the opposite direction.

The amount and direction of that relationship are summarized by the covariance. It is part of the information needed to find the standard error of a contrast. In particular, positive covariance can make a difference more precise than the two individual standard errors suggest, while negative covariance can make the difference less precise.

Definition: Let \(\hat{\beta}_1\) and \(\hat{\beta}_2\) be two estimated regression coefficients, with standard errors \(SE_1\) and \(SE_2\). Their covariance, written \(\operatorname{Cov}(\hat{\beta}_1,\hat{\beta}_2)\), describes how the two estimates vary together across repeated samples under the model. The estimated difference is \(\hat{\beta}_1-\hat{\beta}_2\). Its standard error is the square root of the estimated variance of that difference.

For a meaningful coefficient difference, the two coefficients must be expressed in compatible units. For example, if both describe changes in the same outcome per one-unit change in predictors measured on the same scale, their difference has the same units as either coefficient. The calculation below concerns uncertainty in that difference, not whether the difference is statistically significant.

Deriving the Variance of a Difference

The variance of a sum or difference depends on both the variances of its components and their covariance. To see the result directly, center the estimates by subtracting their expected values. Write \(U=\hat{\beta}_1-E(\hat{\beta}_1)\) and \(V=\hat{\beta}_2-E(\hat{\beta}_2)\). Then the centered difference is \(U-V\).

\begin{aligned} \operatorname{Var}(\hat{\beta}_1-\hat{\beta}_2) &=E\left[(U-V)^2\right]\\ &=E(U^2)-2E(UV)+E(V^2)\\ &=\operatorname{Var}(\hat{\beta}_1)+\operatorname{Var}(\hat{\beta}_2) -2\operatorname{Cov}(\hat{\beta}_1,\hat{\beta}_2). \end{aligned}

The final equality follows because \(E(U^2)\) and \(E(V^2)\) are the respective variances, while \(E(UV)\) is the covariance of the estimates. In practice, the variance terms are replaced by their estimated values, \(SE_1^2\) and \(SE_2^2\), and the covariance by its estimated value from the fitted regression.

Key formula: If \(C_{12}\) is the estimated covariance between \(\hat{\beta}_1\) and \(\hat{\beta}_2\), then

SE(\hat{\beta}_1-\hat{\beta}_2) =\sqrt{SE_1^2+SE_2^2-2C_{12}}.

The covariance is subtracted twice in the variance formula. It is not itself a standard error and may be positive, zero, or negative.

If both standard errors are positive, covariance can also be expressed using the correlation \(\hat{\rho}_{12}\) between the estimates: \(C_{12}=\hat{\rho}_{12}SE_1SE_2\). Substitution gives \(SE(\hat{\beta}_1-\hat{\beta}_2)=\sqrt{SE_1^2+SE_2^2-2\hat{\rho}_{12}SE_1SE_2}\). Correlation has no units, whereas covariance has units equal to the product of the coefficient units. The two forms give the same result when the covariance and correlation correspond to the same estimates.

Worked Calculations

Worked Example: Positive Covariance Reduces Uncertainty in a Difference

Hypothetical data. A fitted regression estimates two coefficients, each measured in outcome points per unit of predictor. Suppose \(\hat{\beta}_1=2.4\), \(\hat{\beta}_2=1.7\), \(SE_1=0.8\), and \(SE_2=0.6\). The estimated correlation between the coefficient estimates is \(0.75\). Find the estimated difference and its standard error.

First calculate the covariance from the correlation: \(C_{12}=\hat{\rho}_{12}SE_1SE_2=(0.75)(0.8)(0.6)=0.36\), in squared coefficient units. The estimated difference is \(2.4-1.7=0.7\) outcome points per predictor unit.

Now substitute into the variance formula: \(SE(\hat{\beta}_1-\hat{\beta}_2)=\sqrt{(0.8)^2+(0.6)^2-2(0.36)} =\sqrt{0.64+0.36-0.72}=\sqrt{0.28}=0.5292\). Thus the estimated difference is \(0.7\), with standard error about \(0.529\) outcome points per predictor unit.

The covariance is positive: the estimates tend to move in the same direction. Some of their sample-to-sample variation therefore cancels when one is subtracted from the other. Subtracting the individual standard errors would give \(0.8-0.6=0.2\), which is not the standard error of the difference and substantially understates it here.

Worked Example: Negative Covariance Increases Uncertainty

Hypothetical data. In another regression, two compatible coefficient estimates are \(\hat{\beta}_1=1.3\) and \(\hat{\beta}_2=0.6\), with standard errors \(0.5\) and \(0.4\). Suppose their estimated correlation is \(-0.60\). Calculate the difference and its standard error.

The estimated covariance is \(C_{12}=(-0.60)(0.5)(0.4)=-0.12\). The coefficient difference is \(1.3-0.6=0.7\), in the units of the coefficients.

The standard error is \(\sqrt{(0.5)^2+(0.4)^2-2(-0.12)} =\sqrt{0.25+0.16+0.24} =\sqrt{0.65}=0.8062\). So the standard error of the difference is about \(0.806\) coefficient units.

Because the covariance is negative, subtracting the estimates tends to combine their variation rather than cancel it. The result is a standard error larger than either individual standard error. Subtracting \(0.5-0.4=0.1\) would miss this effect entirely.

Worked Example: Reading Covariance From a Matrix

Hypothetical data. Software reports the estimated covariance matrix for two regression coefficient estimates as follows, with both coefficients in the same units:

\widehat{\operatorname{Cov}}(\hat{\beta})= \begin{pmatrix} 0.36 & 0.12\\ 0.12 & 0.25 \end{pmatrix}.

The diagonal entries are the estimated variances, so the individual standard errors are \(\sqrt{0.36}=0.6\) and \(\sqrt{0.25}=0.5\). The off-diagonal entry is the estimated covariance \(C_{12}=0.12\). Suppose the corresponding estimates are \(\hat{\beta}_1=1.8\) and \(\hat{\beta}_2=1.1\).

The estimated difference is \(1.8-1.1=0.7\). Its estimated variance is \(0.36+0.25-2(0.12)=0.37\), so its standard error is \(\sqrt{0.37}=0.6083\), or about \(0.608\) coefficient units.

As a check, the implied correlation is \(0.12/(0.6\cdot0.5)=0.40\). It is positive, consistent with the covariance, and less than one in magnitude. The matrix is positive definite because its diagonal entries are positive and its determinant is \((0.36)(0.25)-(0.12)^2=0.0756>0\). A valid covariance matrix must be positive semidefinite, but it need not be positive definite; a singular covariance matrix can also be valid.

What the Covariance Can and Cannot Be

The covariance is constrained by the sizes of the two standard errors. For two estimates with positive variances, their correlation must lie between \(-1\) and \(1\). Equivalently, the covariance must satisfy \(|C_{12}|\leq SE_1SE_2\). One way to see this is that the two-by-two covariance matrix must be positive semidefinite: its determinant cannot be negative, so \(SE_1^2SE_2^2-C_{12}^2\geq0\). Taking square roots yields the stated bound.

This bound also confirms that the variance formula cannot produce a negative value when the covariance information is valid. The covariance matrix may be singular, including in a case of perfect correlation or perfect negative correlation, so positive definiteness is not required. If either estimate has zero variance, its covariance with the other estimate must be zero.

The direction of covariance gives a useful qualitative guide:

Positive covariance: the subtraction term \(2C_{12}\) reduces the variance of the difference. The stronger the positive association, the more the shared movement can cancel.
Zero covariance: the variance of the difference is the sum of the two variances, so its standard error is \(\sqrt{SE_1^2+SE_2^2}\), not the difference of the standard errors.
Negative covariance: subtracting a negative covariance adds to the variance, making the difference less precise.

These patterns are why two coefficient estimates from one regression should not be treated as if they were independent unless independence is justified. Their covariance is determined by the joint behavior of the estimates, which reflects the fitted model and the observed design. The individual standard errors alone do not contain enough information to recover that joint behavior.

A Reliable Calculation and a Common Pitfall

When calculating the standard error of a coefficient difference, use the following sequence. It keeps the estimate, covariance, and uncertainty distinct.

Define the contrast.
Write the requested difference in a fixed order, such as \(\hat{\beta}_1-\hat{\beta}_2\), and confirm that its coefficient units are compatible.

Locate the covariance information.
Use the covariance directly, or calculate it from the estimate correlation and both standard errors. A covariance matrix places the variances on the diagonal and covariances off the diagonal.

Calculate the variance before taking a square root.
Substitute \(SE_1^2+SE_2^2-2C_{12}\), keep the sign of \(C_{12}\), and then take the nonnegative square root.

Report the estimate and its uncertainty in context.
State the numerical difference and the standard error in the coefficient's units. Do not interpret the standard error as a probability that the true difference lies within a specified distance.

A common mistake is to reason that “the standard error of a difference is the difference of the standard errors.” This treats uncertainty as though standard errors were signed quantities that cancel like coefficient estimates. They are nonnegative measures of spread, and the variance of the contrast depends on a covariance term. The correction is to calculate the contrast variance first, including \(-2C_{12}\), and only then take its square root.

Another mistake is to omit the covariance because it is not printed beside each coefficient's standard error in a compact software table. A table of individual standard errors gives only the diagonal variances. To calculate a contrast standard error, the corresponding covariance must also be available, either directly or through the fitted model's covariance output.

Key takeaway: For a difference of two coefficient estimates, uncertainty depends on both individual variances and their covariance. Positive covariance can reduce the standard error of the difference; negative covariance can increase it. Subtracting standard errors does not account for either effect and is not a valid calculation.

Check Your Understanding

For each question, focus on the distinction between a coefficient estimate, its standard error, and the covariance between estimates.

Two coefficient estimates have standard errors \(0.7\) and \(0.5\), and covariance \(0.10\). Write the formula for the standard error of their difference and calculate it.
If the covariance between two estimates is negative, does the covariance term increase or decrease the variance of their difference? Explain using the formula.
Why is \(|SE_1-SE_2|\) not generally the standard error of \(\hat{\beta}_1-\hat{\beta}_2\)?
A covariance matrix has diagonal entries \(0.04\) and \(0.09\), and off-diagonal entry \(0.08\). Is this a valid two-estimate covariance matrix? Check its determinant and explain.
When the correlation between two estimates is zero, what formula gives the standard error of their difference, and why is it not usually \(SE_1-SE_2\)?