Covariance Between Slope And Intercept Estimates

Why the Intercept and Slope Estimates Are Related

In simple linear regression, the least-squares intercept and slope are estimated from the same observations. Their estimates are therefore generally related: a change in the estimated slope can be accompanied by a compensating change in the intercept. The size and direction of this relationship depend partly on where zero is placed on the predictor scale.

This matters when describing uncertainty in regression coefficients or in quantities calculated from them. A variance calculation for a combination of the intercept and slope must include their covariance; omitting it can substantially overstate or understate uncertainty.

Model and assumptions: Treat the predictor values \(x_1,\ldots,x_n\) as fixed, with at least two distinct values. Suppose \(Y_i=\beta_0+\beta_1x_i+\varepsilon_i\), where the errors have mean zero, common variance \(\sigma^2\), and are uncorrelated with one another. Normal errors are not needed for the covariance formulas below. Write \(\bar{x}=\frac{1}{n}\sum_{i=1}^n x_i\) and \(S_{xx}=\sum_{i=1}^n(x_i-\bar{x})^2\). The condition of at least two distinct predictor values ensures \(S_{xx}>0\), so the slope is identifiable.

The least-squares estimates are denoted by \(\hat{\beta}_0\) and \(\hat{\beta}_1\). Under the model above, they are unbiased, and their variances and covariance are:

\operatorname{Var}(\hat{\beta}_1)=\frac{\sigma^2}{S_{xx}},\qquad \operatorname{Cov}(\hat{\beta}_0,\hat{\beta}_1) =-\frac{\bar{x}\sigma^2}{S_{xx}},\qquad \operatorname{Var}(\hat{\beta}_0) =\sigma^2\left(\frac{1}{n}+\frac{\bar{x}^2}{S_{xx}}\right).

The covariance is negative when \(\bar{x}>0\), positive when \(\bar{x}<0\), and zero when \(\bar{x}=0\). Thus, its sign is not an intrinsic feature of the relationship between the measured quantities: it depends on the predictor's origin. The covariance matrix collects all three quantities in one place.

\operatorname{Cov} \begin{pmatrix} \hat{\beta}_0\\ \hat{\beta}_1 \end{pmatrix} = \sigma^2 \begin{pmatrix} \frac{1}{n}+\frac{\bar{x}^2}{S_{xx}} & -\frac{\bar{x}}{S_{xx}}\\[4pt] -\frac{\bar{x}}{S_{xx}} & \frac{1}{S_{xx}} \end{pmatrix}.

Why the Covariance Formula Holds

Let \(d_i=x_i-\bar{x}\). The least-squares slope can be written as \(\hat{\beta}_1=\sum_i d_iY_i/S_{xx}\). Since \(\sum_i d_i=0\), substituting the regression model removes the constant part and gives the slope error directly in terms of the model errors.

\hat{\beta}_1-\beta_1=\frac{\sum_{i=1}^n d_i\varepsilon_i}{S_{xx}}, \qquad \hat{\beta}_0-\beta_0=\bar{\varepsilon}-\bar{x}(\hat{\beta}_1-\beta_1), \qquad \bar{\varepsilon}=\frac{1}{n}\sum_{i=1}^n\varepsilon_i.

The error average and the slope error have covariance zero. Indeed, using uncorrelated errors with common variance \(\sigma^2\), each matching term contributes \(\sigma^2d_i/(nS_{xx})\), and the sum vanishes because the deviations sum to zero. The slope error has variance \(\sigma^2/S_{xx}\). Taking the covariance of the intercept-error expression with the slope error therefore yields the stated result.

\operatorname{Cov}(\bar{\varepsilon},\hat{\beta}_1-\beta_1) =\frac{\sigma^2}{nS_{xx}}\sum_{i=1}^n d_i=0, \qquad \operatorname{Var}(\hat{\beta}_1)=\frac{\sigma^2}{S_{xx}}, $$ $$ \operatorname{Cov}(\hat{\beta}_0,\hat{\beta}_1) =0-\bar{x}\frac{\sigma^2}{S_{xx}} =-\frac{\bar{x}\sigma^2}{S_{xx}}.

The same decomposition gives the intercept variance: the variance of \(\bar{\varepsilon}\) is \(\sigma^2/n\), and its covariance with the slope error is zero. Adding the variance of the second term in the intercept error gives \(\sigma^2/n+\bar{x}^2\sigma^2/S_{xx}\). These calculations justify every entry in the covariance matrix.

Worked Example: Computing the Covariance Matrix

A hypothetical study records a response at fixed predictor values \(0,2,4,6,8\). Assume the errors are uncorrelated, have mean zero, and have variance \(\sigma^2=10\). Find the covariance between the least-squares intercept and slope, and the full coefficient covariance matrix.

Method. Calculate \(\bar{x}\) and \(S_{xx}\), then substitute into the covariance matrix. This avoids treating the two coefficient estimates as unrelated.

First, \(n=5\), and \(\bar{x}=(0+2+4+6+8)/5=20/5=4\). The deviations from the mean are \(-4,-2,0,2,4\), so \(S_{xx}=(-4)^2+(-2)^2+0^2+2^2+4^2=16+4+0+4+16=40\).

The covariance is \(\operatorname{Cov}(\hat{\beta}_0,\hat{\beta}_1)=-(4)(10)/40=-1\). The slope variance is \(10/40=0.25\). The intercept variance is \(10(1/5+4^2/40)=10(0.2+0.4)=6\).

Therefore, the covariance matrix is \(\begin{pmatrix}6&-1\\-1&0.25\end{pmatrix}\). The negative off-diagonal entries show that the two estimates tend to move in opposite directions under repeated samples with this fixed predictor design.

Check. The determinant is \(6(0.25)-(-1)^2=1.5-1=0.5>0\), consistent with a valid positive-definite covariance matrix. The common mistake of using a positive sign for the covariance would contradict the formula and the positive predictor mean.

Centering the Predictor Changes the Intercept

For a constant \(c\), define a centered predictor \(x_i^*=x_i-c\). The regression line can be written using the same slope and a new intercept: \(\alpha_0=\beta_0+c\beta_1\), with \(\alpha_1=\beta_1\). The least-squares estimates transform in the same way: \(\hat{\alpha}_0=\hat{\beta}_0+c\hat{\beta}_1\) and \(\hat{\alpha}_1=\hat{\beta}_1\).

Applying the covariance rule to this sum shows exactly how the intercept-slope covariance changes. The mean of the centered predictor is \(\bar{x}-c\), so the new covariance is proportional to \(c-\bar{x}\). In particular, centering at the sample predictor mean makes the transformed intercept and slope uncorrelated.

\operatorname{Cov}(\hat{\alpha}_0,\hat{\alpha}_1) =\operatorname{Cov}(\hat{\beta}_0+c\hat{\beta}_1,\hat{\beta}_1) =\frac{(c-\bar{x})\sigma^2}{S_{xx}}.

Key result: If \(c=\bar{x}\), the new intercept is the fitted mean response at the average predictor value, and its covariance with the slope estimate is zero. Its variance is \(\sigma^2/n\). Zero covariance does not in general imply independence; if the regression errors are normally distributed, the coefficient estimates are jointly normal, and zero covariance does imply independence.

Worked Example: Choosing a Center to Change the Covariance

A hypothetical calibration study uses predictor values \(10,20,30,40\), measured in a consistent unit. Assume independent, mean-zero errors with variance \(\sigma^2=25\). Find the covariance between the slope and intercept after centering at \(c=20\), and then find it after centering at the predictor mean.

Method. Use the transformation formula for the covariance. The slope estimate and its variance do not change when the predictor is shifted; only the intercept changes.

The predictor mean is \(\bar{x}=(10+20+30+40)/4=25\). The deviations are \(-15,-5,5,15\), giving \(S_{xx}=225+25+25+225=500\). The original covariance is \(-25(25)/500=-1.25\), and the slope variance is \(25/500=0.05\).

For \(c=20\), the new covariance is \((20-25)(25)/500=-125/500=-0.25\). Equivalently, it is \(-1.25+20(0.05)=-0.25\).

For \(c=\bar{x}=25\), the covariance is \((25-25)(25)/500=0\). The new intercept is the estimated mean response at \(x=25\), and its variance is \(\sigma^2/n=25/4=6.25\).

Check. The two calculations agree with the transformation identity, and centering at the mean gives zero covariance. The mistake to avoid is assuming that any centering value eliminates covariance; it does so here only when \(c=\bar{x}\).

Using the Matrix for Regression Inference

The covariance matrix is useful whenever an inferential quantity is a linear combination of the two coefficients. For constants \(a\) and \(b\), the variance of \(a\hat{\beta}_0+b\hat{\beta}_1\) is obtained by multiplying the covariance matrix on both sides by the coefficient vector. The off-diagonal covariance contributes twice, so it cannot be dropped unless it is zero or the relevant coefficient makes its contribution vanish.

\operatorname{Var}(a\hat{\beta}_0+b\hat{\beta}_1) =a^2\operatorname{Var}(\hat{\beta}_0) +2ab\operatorname{Cov}(\hat{\beta}_0,\hat{\beta}_1) +b^2\operatorname{Var}(\hat{\beta}_1).

For example, the fitted mean response at a specified predictor value \(x_*\) is \(\hat{\beta}_0+x_*\hat{\beta}_1\). Its variance follows by setting \(a=1\) and \(b=x_*\). It simplifies to a useful expression centered on \(\bar{x}\). This is the uncertainty in the estimated mean response, not the uncertainty in a new individual response.

\operatorname{Var}(\hat{\beta}_0+x_*\hat{\beta}_1) =\sigma^2\left(\frac{1}{n}+\frac{(x_*-\bar{x})^2}{S_{xx}}\right).

Worked Example: Variance of a Fitted Mean Using the Matrix

A hypothetical response study uses fixed predictor values \(1,3,5,7\). Assume independent, mean-zero errors with variance \(\sigma^2=4\), and suppose the fitted coefficients are \(\hat{\beta}_0=3.2\) and \(\hat{\beta}_1=1.1\). Find the fitted mean at \(x_*=7\) and the variance of that fitted mean. Also distinguish it from the prediction error variance for a new response at \(x=7\).

Method. First construct the coefficient covariance matrix and then use the vector \((1,7)\) to evaluate the variance of the fitted linear combination. This makes the covariance contribution explicit.

Here \(n=4\), \(\bar{x}=(1+3+5+7)/4=4\), and the deviations \(-3,-1,1,3\) give \(S_{xx}=9+1+1+9=20\). Thus the intercept variance is \(4(1/4+16/20)=4(0.25+0.8)=4.2\). The coefficient covariance is \(-4(4)/20=-0.8\), and the slope variance is \(4/20=0.2\). The matrix is therefore \(\begin{pmatrix}4.2&-0.8\\-0.8&0.2\end{pmatrix}\).

The fitted mean is \(3.2+7(1.1)=3.2+7.7=10.9\). Its variance, using the matrix, is \(4.2+7^2(0.2)+2(1)(7)(-0.8)=4.2+9.8-11.2=2.8\). The simplified formula confirms this: \(4(1/4+(7-4)^2/20)=4(0.25+9/20)=4(0.70)=2.8\).

For a new response at \(x=7\), its error is additionally subject to a fresh model error with variance \(4\), independent of the coefficient estimates. The prediction error variance is therefore \(2.8+4=6.8\), not \(2.8\). The fitted mean variance and new-response prediction variance answer different questions.

Check. The matrix calculation and simplified expression both give \(2.8\) for the fitted mean. Omitting the covariance term would give \(4.2+9.8=14.0\), which is incorrect; here the negative covariance substantially reduces the variance.

Scope and Common Exam Traps

These formulas depend on the stated fixed-predictor model and the common error variance assumption. If errors are correlated or have unequal variances, the displayed covariance matrix generally does not apply; the coefficient covariance must instead reflect the actual error covariance structure. If all predictor values are identical, then \(S_{xx}=0\), and the slope cannot be estimated from the data.

A second distinction is between a coefficient covariance and a correlation. Covariance has units equal to the product of the coefficient units, and its numerical value changes with the scale of either coefficient. Correlation is dimensionless, but it too can change when the predictor origin changes because the intercept itself changes. Neither quantity should be interpreted without knowing how the variables were defined.

Wrong sign: writing \(+\bar{x}\sigma^2/S_{xx}\) instead of \(-\bar{x}\sigma^2/S_{xx}\). The correction follows from \(\hat{\beta}_0=\bar{Y}-\bar{x}\hat{\beta}_1\): when \(\bar{x}>0\), the slope contribution to the intercept has a minus sign.
Ignoring the cross term: the variance of a coefficient combination includes \(2ab\operatorname{Cov}(\hat{\beta}_0,\hat{\beta}_1)\). The correction is to use the full covariance matrix, not just its diagonal entries.
Centering at the wrong value: changing the origin to an arbitrary \(c\) does not generally make the covariance zero. The correction is to center at \(c=\bar{x}\) for this fixed design.
Confusing fitted mean with future response: a future response includes a new error term. Add \(\sigma^2\) to the fitted mean variance when calculating the prediction error variance for an independent future observation.

Key takeaway: Under the standard fixed-predictor regression model, the intercept-slope covariance is \(-\bar{x}\sigma^2/S_{xx}\). Centering at the predictor mean makes it zero, and the full covariance matrix is essential for correct uncertainty calculations involving both coefficients.

Check Your Understanding

For each problem, use the fixed-predictor model with uncorrelated, mean-zero errors of the stated common variance.

A regression uses predictor values \(2,4,6,8\), with error variance \(9\). Find the covariance between the estimated intercept and slope. (Answer: \(-2.25\))
Predictor values are \(1,3,5,7\), the error variance is \(4\), and the predictor is recentered as \(x^*=x-3\). Find the covariance between the new intercept and the slope estimate. (Answer: \(-0.2\))
A design has five predictor values \(5,6,7,8,9\) and error variance \(6\). Find the covariance between the intercept and slope estimates in the original predictor scale. (Answer: \(-4.2\))
For predictor values \(-2,0,2,4\), the error variance is \(5\). Find the variance of the fitted mean at \(x_*=3\), using the coefficient covariance matrix or the simplified formula. (Answer: \(2.25\))
A study has \(n=8\), predictor mean \(12\), \(S_{xx}=80\), and error variance \(16\). Find the centering constant that makes the intercept-slope covariance zero, then find the variance of the fitted mean at the original-scale predictor value \(15\). (Answer: center at \(c=12\); fitted mean variance \(3.8\))

What You'll Learn