The Geometry of Near Collinearity

Why Near Collinearity Is a Geometric Problem

When two predictors move together across the observations, the data contain abundant information about their shared direction but little information about how their separate effects differ. Regression can still produce fitted values, yet small changes in the data may lead to large changes in the two estimated coefficients. The geometry makes this distinction precise: a long, narrow confidence region for the coefficients can coexist with much shorter uncertainty in directions that the observed predictors actually cover well.

The discussion below uses a two-predictor linear regression with an intercept. The predictors are centered, and each predictor column is scaled to have Euclidean length one. Centering makes the predictor columns orthogonal to the intercept, and scaling makes their relative angle the focus rather than their measurement units. These conventions simplify the formulas; the same geometric idea applies to unscaled predictors, though the numerical covariance matrix then depends on their lengths as well as their angle.

Write the two centered predictor columns as vectors \(x_1\) and \(x_2\), and define \(r=x_1^\mathsf{T}x_2\). Since each vector has length one, \(r\) is their inner product and also their sample correlation under this scaling. Values of \(r\) close to 1 mean the columns point in nearly the same direction; values close to -1 mean they point in nearly opposite directions. Either case means one predictor direction can be approximated by the other, although the formulas below use \(r\) close to 1 for illustration.

Projection Directions and Coefficient Uncertainty

The two predictor columns can be rewritten using orthogonal directions. For \(-1<r<1\), define

Both \(u\) and \(w\) have length one, and \(u^\mathsf{T}w=0\). For example, \((x_1+x_2)^\mathsf{T}(x_1-x_2)=\lVert x_1\rVert^2-\lVert x_2\rVert^2=0\). The model's predictor contribution can now be expressed as

This equation shows why the sum and difference of the coefficients play different roles. The common direction \(u\) is multiplied by \(\sqrt{(1+r)/2}\), which is close to one when \(r\) is near 1. The contrast direction \(w\) is multiplied by \(\sqrt{(1-r)/2}\), which is small. Consequently, a large change in \(\beta_1-\beta_2\) can make only a modest change in the fitted predictor contribution along \(w\).

Deriving the Coefficient Covariance

After the intercept is accounted for, the predictor part of the design matrix has Gram matrix

For ordinary least squares, under the fixed-design and error assumptions above, the covariance matrix of the two slope estimates is \(\sigma^2(X^\mathsf{T}X)^{-1}\). Inverting the matrix gives

The determinant \(1-r^2\) approaches zero as \(|r|\) approaches one, so the coefficient variances grow. The negative covariance when \(r>0\) has a useful interpretation: increasing one estimated slope can be offset by decreasing the other while preserving much of the fitted signal.

Under normal errors with known \(\sigma^2\), the joint confidence region for \((\beta_1,\beta_2)\) is an ellipse: its centered quadratic form is bounded by the appropriate chi-square quantile. Its principal axes point along the coefficient sum and difference directions. The long axis is the difference direction for positive near-collinearity. The ratio of its semiaxis length to the short semiaxis length is \(\sqrt{(1+r)/(1-r)}\). If \(\sigma^2\) is estimated instead, an exact finite-sample region uses the corresponding \(F\)-based scaling; the orientation and relative elongation remain the same. These confidence-region statements require normal errors for their exact finite-sample interpretation.

Why Fitted Values May Be More Stable

Large uncertainty in separate slopes does not automatically imply equally large uncertainty in fitted values. A fitted value depends on a particular combination of the coefficients. If the prediction uses predictor values that follow the same common pattern as the observed columns, the unstable coefficient contrast may cancel. In contrast, predicting for a pattern that separates the two predictors can depend heavily on that uncertain contrast.

For a new predictor vector \(z=(z_1,z_2)^\mathsf{T}\), the estimated mean response from the slopes is \(z^\mathsf{T}\hat\beta\). Its variance, conditional on the fixed design, is \(z^\mathsf{T}\operatorname{Cov}(\hat\beta)z\). This is uncertainty in the estimated mean, not the additional observation error for a future individual response. For the latter, add \(\sigma^2\), provided the future error has the same variance and is independent of the fitted data.

Common Trap: Confusing Coefficient Uncertainty With Prediction Uncertainty

A frequent error is to see large standard errors for individual slopes and conclude that the model's fitted values must all be unreliable. The conclusion does not follow. Each slope standard error describes uncertainty in one coordinate of the coefficient vector. Prediction uncertainty depends on the particular vector \(z\) at which the model is evaluated. Under the assumptions already stated, it is \(z^\mathsf{T}\operatorname{Cov}(\hat\beta)z\), not simply the sum of the two slope variances.

For example, in the \(r=0.90,\ \sigma^2=1\) calculation above, a mistaken approach might add the two slope variances and report \(2/0.19=10.5263\) as the prediction variance for \(z=(1,1)\). That omits the covariance term. The correct calculation includes it:

The omitted covariance is strongly negative, so ignoring it badly overstates uncertainty in the common-pattern prediction. For the contrast \(\hat\beta_1-\hat\beta_2\), the covariance enters with a minus sign and increases uncertainty instead. Always use the full covariance matrix for a linear combination.

When the Geometry Matters

The geometry is useful whenever model interpretation, prediction, or inference depends on separating effects of related predictors. For example, a hypothetical insurer might model claim cost using two exposure measures that tend to rise together. The regression could estimate their combined contribution reasonably while providing weak evidence about which measure has the larger conditional slope. The example is hypothetical: the point is that the design's geometry, not the application label, controls the distinction.

Near collinearity does not by itself bias ordinary least-squares estimates under the stated model assumptions. Instead, it increases sampling variability in certain coefficient directions. Nor does the geometric argument guarantee accurate prediction everywhere: predictions along poorly supported contrast patterns can be highly uncertain, especially outside the patterns represented in the observed design. A careful interpretation therefore asks both what coefficient combination is being discussed and whether the prediction point follows the data's shared pattern.