Why Coefficient Uncertainty Depends on the Predictors
A regression model can fit observed responses closely while leaving substantial uncertainty about individual coefficients. This is possible because residual error measures how far the responses are from the fitted regression space, whereas coefficient uncertainty also depends on how much independent information the predictor columns provide. When columns are nearly dependent, different coefficient combinations can produce similar fitted values, making some coefficients imprecise.
The key object is the covariance matrix of the least-squares estimator. Its diagonal entries give coefficient variances, and its off-diagonal entries give covariances between coefficient estimates. The matrix makes precise how the design, not merely the residual size, affects coefficient uncertainty.
Deriving the Covariance Matrix
When \(X\) has full column rank, the least-squares estimator minimizes the residual sum of squares and is
Substitute the model equation \(y=X\beta+\varepsilon\) into this estimator. The result separates the fixed true coefficient vector from the random error:
Conditional on \(X\), the first term is fixed. For any fixed matrix \(A\), the covariance of \(A\varepsilon\) is \(A\operatorname{Cov}(\varepsilon\mid X)A^\top\). Taking \(A=(X^\top X)^{-1}X^\top\) gives the result:
The simplification uses the symmetry of \(X^\top X\) and the fact that its inverse exists. This derivation shows why near-dependence matters: if \(X^\top X\) is close to singular, its inverse can have large entries in some directions. Those entries become large coefficient variances or covariances.
Replacing \(\sigma^2\) by \(s^2\) gives the estimated covariance matrix \(s^2(X^\top X)^{-1}\). Under the assumptions above, \(s^2\) is an unbiased estimator of \(\sigma^2\). If the errors are also normally distributed, the coefficient estimator has a multivariate normal distribution conditional on \(X\); normality is needed for exact normal-theory inference, not for the covariance derivation itself.
Two Predictors: The Variance Inflation Appears Explicitly
Consider two centered predictor columns \(x_1\) and \(x_2\), with no intercept included in the following predictor block. Centering makes each column orthogonal to an intercept if the model includes one. Let \(S_{11}=x_1^\top x_1\), \(S_{22}=x_2^\top x_2\), and let \(r\) be their sample correlation. Their cross-product is \(r\sqrt{S_{11}S_{22}}\), so the predictor cross-product matrix is
Its determinant is \(S_{11}S_{22}(1-r^2)\). Inverting the matrix gives
For example, \(\operatorname{Var}(\hat{\beta}_1\mid X)=\sigma^2/[S_{11}(1-r^2)]\). If the other predictor were orthogonal to \(x_1\), the corresponding variance would be \(\sigma^2/S_{11}\). Thus the variance multiplier is \(1/(1-r^2)\), the two-predictor variance inflation factor. As the absolute correlation approaches one, this multiplier increases without bound. At perfect correlation the determinant is zero, the design loses full rank, and the two coefficients cannot be separately identified.
Worked Example: Covariance Matrix for Two Correlated Rating Measures
Method: Invert the two-predictor cross-product matrix, then multiply by the assumed error variance. A common mistake is to use the correlation \(r\) itself as the variance multiplier rather than \(1/(1-r^2)\).
A hypothetical model for annual claim cost uses two centered rating measures. Their sums of squares are \(S_{11}=100\) and \(S_{22}=100\), their correlation is \(r=0.8\), and the error variance is \(\sigma^2=9\) dollars squared. Find the coefficient covariance matrix and standard errors.
First, the cross-product of the predictors is \(0.8\sqrt{100(100)}=0.8(100)=80\). Therefore,
Its determinant is \(100(100)-80(80)=10{,}000-6{,}400=3{,}600\), so
Multiplying every entry by \(\sigma^2=9\) gives
Each coefficient variance is \(0.25\), so each standard error is \(\sqrt{0.25}=0.50\). With orthogonal predictors having the same sums of squares, each variance would instead be \(9/100=0.09\), with standard error \(0.30\). The variance inflation factor here is \(1/(1-0.8^2)=1/0.36=2.7778\), and \(0.09(2.7778)=0.25\), up to rounding. The negative off-diagonal covariance is also part of the result; it indicates that the two coefficient estimates tend to move in opposite directions around their means. The diagonal calculation and the variance-inflation check agree.
Coefficient Uncertainty Is Not Residual Error
The residual sum of squares is \(\mathrm{SSE}=(y-X\hat{\beta})^\top(y-X\hat{\beta})\). It records the size of the part of the response not fitted by the columns of \(X\). Coefficient covariance, by contrast, depends on \((X^\top X)^{-1}\) as well as on the error variance. These are different quantities.
One way to see the distinction is to define the projection matrix \(P_X=X(X^\top X)^{-1}X^\top\) and the residual-maker matrix \(M_X=I-P_X\). The fitted values are \(P_Xy\), the residual vector is \(M_Xy\), and \(\mathrm{SSE}=y^\top M_Xy\). Since \(M_XX=0\), the model equation implies \(\mathrm{SSE}=\varepsilon^\top M_X\varepsilon\). The residual is the component of the error perpendicular to the predictor space; the coefficient estimator depends on how the error is resolved along the predictor columns. Near-collinearity can make that resolution unstable without forcing a large perpendicular residual.
Worked Example: Small Residual Error with Imprecise Coefficients
Method: Use the observed residual sum of squares to estimate the error variance, then use the inverse cross-product matrix for coefficient uncertainty. The mistake to avoid is concluding that a modest SSE guarantees precise coefficients.
Suppose a hypothetical model has \(n=5\) observations, an intercept, and two centered predictors. The predictor sums of squares are both 100 and their correlation is \(0.9\). Suppose the observed residuals are orthogonal to the intercept and both predictors, and their squared length is \(\mathrm{SSE}=40\). These conditions are compatible: the residual subspace has dimension \(5-3=2\). The fitted response can be any vector in the model space; the residual component specified here gives the stated SSE.
There are \(p=3\) fitted coefficients, so the residual degrees of freedom are \(n-p=5-3=2\), and the estimated error variance is \(s^2=40/2=20\). The predictor cross-product matrix and its inverse are
Using \(s^2=20\), the estimated covariance matrix for the two slopes is
Each slope standard error is \(\sqrt{20/19}\), approximately \(1.026\). With the same sums of squares, but orthogonal predictor columns and the same SSE, the slope variances would be \(20/100=0.20\), giving standard errors \(\sqrt{0.20}\), approximately \(0.447\). The residual SSE is 40 in both comparisons; the estimated coefficient uncertainty differs because the predictor cross-products differ. The variance inflation factor for the correlated design is \(1/(1-0.9^2)=1/0.19\), which multiplies the orthogonal variance \(0.20\) to give \(20/19\). This check confirms the matrix calculation.
Reading the Covariances, Not Just the Diagonal
A covariance matrix contains information about joint coefficient variation. A negative covariance does not mean either coefficient has a negative variance; variances are diagonal entries and cannot be negative under a valid covariance matrix. The sign of an off-diagonal entry describes how deviations of two estimates tend to move together. This matters when calculating the uncertainty of a linear combination \(c^\top\hat{\beta}\): its variance is \(c^\top\operatorname{Cov}(\hat{\beta}\mid X)c\), which includes the covariance terms.
For an individual coefficient, the standard error is often the most visible summary of uncertainty. But the covariance terms should not be discarded when the quantity of interest combines coefficients. For example, the variance of \(\hat{\beta}_1+\hat{\beta}_2\) includes twice their covariance, while the variance of \(\hat{\beta}_1-\hat{\beta}_2\) subtracts twice their covariance.
Worked Example: Using the Full Matrix for a Coefficient Combination
Method: Apply the quadratic-form rule \(c^\top\operatorname{Cov}(\hat{\beta})c\). The common mistake is to add the two coefficient variances and omit the covariance contribution.
In a hypothetical model, two centered predictors have \(S_{11}=25\), \(S_{22}=100\), correlation \(r=0.6\), and error variance \(\sigma^2=4\). Find the variances of the sum and difference of the two slope estimates.
The cross-product is \(0.6\sqrt{25(100)}=0.6(50)=30\). The determinant of the cross-product matrix is \(25(100)-30^2=2{,}500-900=1{,}600\), so its inverse is
Multiplication by 4 gives
For the sum, use \(c=(1,1)^\top\). Its variance is \(0.25+0.0625+2(-0.075)=0.1625\), and its standard error is \(\sqrt{0.1625}\), approximately \(0.4031\). For the difference, use \(c=(1,-1)^\top\). Its variance is \(0.25+0.0625-2(-0.075)=0.4625\), and its standard error is \(\sqrt{0.4625}\), approximately \(0.6801\). Both calculations use the same diagonal variances, but the negative covariance makes the sum less variable and the difference more variable. The results are positive, as required for variances, and the covariance has been included with the appropriate sign.
When the Formula Needs Care
The expression \(\sigma^2(X^\top X)^{-1}\) relies on the model assumptions stated at the start. If the errors have unequal variances or are correlated, their conditional covariance is not \(\sigma^2 I\); the general least-squares covariance is instead
Thus the simple formula should not be used automatically when the constant-variance and uncorrelated-error assumptions fail. Also, if the columns are exactly dependent, \((X^\top X)^{-1}\) does not exist. Near-dependence is different: the coefficients remain identified, but the inverse can produce large uncertainty.
A useful interpretation is that collinearity does not mechanically make every coefficient imprecise by the same amount. The design matrix determines which directions in coefficient space are poorly supported. In the two-predictor formula, both individual slope variances are inflated by \(1/(1-r^2)\), while their covariance is negative when the predictor correlation is positive. With more predictors, the diagonal entries of the full inverse account for the dependence among all predictor columns.
Check Your Understanding
For each problem, identify the relevant covariance calculation before checking the final answer.
- Two centered predictors have sums of squares 80 and 45, correlation \(0.5\), and error variance \(\sigma^2=2\). Find the two slope variances and their covariance. (Answer: variances \(1/30\) and \(8/135\); covariance \(-1/45\).)
- A predictor has sum of squares 36, and its squared multiple correlation with the other predictors is \(R_j^2=0.75\). Given \(\sigma^2=9\), calculate its coefficient variance and standard error using the variance-inflation form. (Answer: variance \(1\); standard error \(1\).)
- A two-predictor cross-product matrix is \(\begin{pmatrix}64&56\\56&64\end{pmatrix}\), and \(\sigma^2=4\). Calculate the covariance matrix of the two slope estimates. (Answer: \(\begin{pmatrix}4/15&-7/30\\-7/30&4/15\end{pmatrix}\).)
- A model with an intercept and two predictors has \(n=8\) observations and SSE \(=30\). The predictors have equal sums of squares of 100 and correlation \(0.8\). Estimate the error variance and the two slope variances. (Answer: \(s^2=6\); each slope variance is \(1/6\).)
- Under the classical fixed-design assumptions, two designs produce the same observed SSE, but one has predictor correlation \(0.9\) and the other has orthogonal predictors with equal column sums of squares. Which design gives larger individual slope variances, and by what factor, holding \(\sigma^2\) and the sums of squares fixed? (Answer: the design with correlation \(0.9\), by \(1/(1-0.9^2)=100/19\), approximately \(5.2632\).)