Why Dependence Among Predictors Matters
A regression model can fit observed responses well while giving uncertain estimates of individual predictor effects. One reason is that predictor columns may carry overlapping information. At one extreme, a predictor column is an exact linear combination of other columns. At the other, a combination of predictors comes close to reproducing another predictor. The first is exact multicollinearity; the second is near multicollinearity.
The distinction matters. Exact dependence means the design matrix cannot distinguish some coefficient values, so the full coefficient vector is not uniquely determined by least squares. Near dependence does not, in exact arithmetic, destroy uniqueness when the design has full rank, but it can make estimates highly variable and computations sensitive to small perturbations. A good diagnosis therefore checks dependence in a way that is not confounded by the units or magnitudes of the columns.
Definitions in the Regression Design
For \(n\) observations and \(p\) predictors, write the multiple linear regression model as \(y=X\beta+\varepsilon\). Here \(y\) is the response vector, \(X\) is the design matrix, and \(\beta\) contains the regression coefficients. When the model includes an intercept, its column of ones is part of \(X\). For the usual least-squares uncertainty calculations, assume the design is fixed and the errors are independent with mean zero and common variance \(\sigma^2\).
The definition includes the intercept column. For instance, if a predictor is a constant plus a multiple of another predictor, the two predictor columns may be dependent once the intercept is included. Exact dependence is a mathematical property. Near dependence is a matter of degree and should be assessed using a scale-aware measure; there is no universal cutoff that separates “near” from “not near.”
A full-column-rank design gives a unique ordinary least-squares coefficient vector. Exact dependence removes that guarantee: different coefficient vectors can yield the same fitted values. With near dependence and full rank, the least-squares coefficients remain unique, but their estimates can be unstable. These are distinct consequences, not interchangeable labels.
Worked Example: A Predictor Defined by Another Predictor
In a hypothetical study, four observations have \(x_1=(0,1,2,3)\) and \(x_2=(1,3,5,7)\). A response is \(y=(4,7,10,13)\). The fitted model includes an intercept and both predictors. Determine whether the design has exact multicollinearity and show what happens to the coefficients.
Method. Check whether a column is an exact combination of the other design columns, then substitute that relation into the regression equation. The common mistake to avoid is checking only whether the two predictor columns are multiples of each other while ignoring the intercept.
Let \(X_0\) denote the intercept column, consisting of four ones. Observation by observation, \(x_2=2x_1+1\), so the design columns satisfy \(X_2=2X_1+X_0\). This is exact dependence, even though \(x_2\) is not a multiple of \(x_1\) by itself.
The model’s fitted value at a predictor value \(x_1\) is \(\beta_0+\beta_1x_1+\beta_2x_2\). Substituting \(x_2=2x_1+1\) gives \((\beta_0+\beta_2)+(\beta_1+2\beta_2)x_1\). Thus the predictions depend on the two combinations \(\beta_0+\beta_2\) and \(\beta_1+2\beta_2\), rather than on all three coefficients separately.
The response values equal \(4+3x_1\): for \(x_1=0,1,2,3\), the values are \(4,7,10,13\). Therefore, every coefficient triple of the form \((\beta_0,\beta_1,\beta_2)=(4-t,3-2t,t)\) gives these fitted values. For example, \(t=0\) gives \((4,3,0)\), while \(t=1\) gives \((3,1,1)\). Both produce \(4+3x_1\) for each observation.
Check. The coefficient triples differ, but their predictor combinations are \(\beta_0+\beta_2=4\) and \(\beta_1+2\beta_2=3\) in both cases. The design is rank deficient, and the least-squares coefficient vector is not unique.
A Scale-Invariant Check for Two Predictors
For two predictor columns, first center each column by subtracting its sample mean. Call the centered columns \(x_{1c}\) and \(x_{2c}\). Define \(S_{11}=\sum_i x_{1c,i}^2\), \(S_{22}=\sum_i x_{2c,i}^2\), and \(S_{12}=\sum_i x_{1c,i}x_{2c,i}\). Suppose both columns have nonzero sums of squares, so their sample correlation \(r=S_{12}/\sqrt{S_{11}S_{22}}\) is defined.
A useful measure of departure from exact proportionality is \(1-r^2\). It is unchanged if either predictor is converted to different units or multiplied by a nonzero constant. Values close to zero indicate that the two centered columns are nearly proportional; \(r=1\) or \(r=-1\) indicates exact proportionality after centering. Centering here accounts for the intercept: proportionality of centered columns means one original predictor is an affine function of the other.
The following identity explains why this measure works. The cross-product matrix for the centered predictors has determinant \(S_{11}S_{22}-S_{12}^2\). Dividing by \(S_{11}S_{22}\) removes the scale of the columns:
By the Cauchy–Schwarz inequality, \(S_{12}^2\leq S_{11}S_{22}\), so \(1-r^2\) cannot be negative. Equality in that inequality occurs exactly when the two nonzero centered columns are proportional. This gives both the scale-invariant diagnostic and its exact-dependence condition.
Worked Example: Near Dependence and Coefficient Uncertainty
A hypothetical model has an intercept and two centered predictors \(x_1=(-2,-1,1,2)\) and \(x_2=x_1+0.1z\), where \(z=(1,-2,2,-1)\). Assume independent errors with mean zero and known variance \(\sigma^2=4\). Calculate the correlation-based measure of near dependence and compare the standard error of the \(x_1\) coefficient with the value it would have if these predictors were orthogonal.
Method. Compute the centered sums of squares and cross-product, then use the two-predictor coefficient-variance formula for a model with an intercept. The common mistake to avoid is treating a high correlation as proof of exact dependence: here the columns are not exactly proportional.
Both vectors have mean zero. Their sums of squares are \(x_1'x_1=10\) and \(z'z=10\), while their cross-product is \(x_1'z=(-2)(1)+(-1)(-2)+(1)(2)+(2)(-1)=0\). Therefore, \(x_2'x_2=(x_1+0.1z)'(x_1+0.1z)=10+0.2(0)+0.01(10)=10.1\), and \(x_1'x_2=10\).
It follows that \(r=10/\sqrt{10(10.1)}=10/\sqrt{101}\), approximately \(0.9950\), and \(1-r^2=1-100/101=1/101\), approximately \(0.009901\). Since \(x_2\) includes a nonzero component in the direction of \(z\), which is orthogonal to \(x_1\), the columns are not exactly proportional. They are nevertheless very close to proportional.
For a fixed design with an intercept and two centered predictors, the predictor block of \(X'X\) is \(\begin{pmatrix}S_{11}&S_{12}\\S_{12}&S_{22}\end{pmatrix}\). Under the stated independent, equal-variance error assumptions, the least-squares coefficient covariance matrix is \(\sigma^2(X'X)^{-1}\). Inverting this two-by-two block gives \(\operatorname{Var}(\hat\beta_1)=\sigma^2/[S_{11}(1-r^2)]\). Here the standard error is therefore \(\sqrt{4/[10(1/101)]}=\sqrt{40.4}\), approximately \(6.3561\).
If the predictors were orthogonal with the same \(S_{11}=10\), then \(r=0\) and the standard error would be \(\sqrt{4/10}\), approximately \(0.6325\). The ratio is \(\sqrt{101}\), approximately \(10.0499\). Thus the coefficient estimate is much less precise under the near-dependent design, even though the coefficient vector is still unique.
Check. The computed \(1-r^2\) is positive, as required for a full-rank two-predictor block, but small. The standard error ratio is also \(\sqrt{101}\), since the variance ratio is \(101\). The results agree.
Why a Small Raw Determinant Can Mislead
The determinant \(S_{11}S_{22}-S_{12}^2\) is not, on its own, a scale-invariant measure of dependence. Multiplying a column by a small constant reduces the determinant even when the columns remain orthogonal. A small raw value can therefore reflect small units or small magnitudes, rather than near multicollinearity.
Worked Example: A Small Determinant Without Dependence
Consider the centered columns \(x_1=(0.001,-0.001)\) and \(x_2=(0.001,0.001)\), with an intercept in the design. A reviewer sees a small determinant and concludes that the predictors are nearly dependent. Check that conclusion using a scale-invariant calculation.
Method. Calculate the two sums of squares and cross-product, then compare the raw determinant with \(1-r^2\). The common mistake to avoid is interpreting the unnormalized determinant without accounting for the scale of the columns.
The sums of squares are \(S_{11}=0.001^2+(-0.001)^2=0.000002\) and \(S_{22}=0.001^2+0.001^2=0.000002\). The cross-product is \(S_{12}=0.001(0.001)+(-0.001)(0.001)=0\). Hence the determinant is \(S_{11}S_{22}-S_{12}^2=(0.000002)^2=4\times10^{-12}\).
Although that raw determinant is small, \(r=0/\sqrt{S_{11}S_{22}}=0\), so \(1-r^2=1\). The columns are orthogonal, not nearly proportional. The small determinant comes from the small scale of the predictor values. This check illustrates why a scale-invariant quantity, rather than the raw determinant, is needed.
Check. A zero cross-product gives zero correlation, and the normalized determinant is \(4\times10^{-12}/(4\times10^{-12})=1\), confirming the calculation.
More Than Two Predictors and Numerical Stability
With several predictors, a pairwise correlation check is not sufficient. A column can be an exact combination of multiple other columns even when no pair of columns has correlation near \(1\) or \(-1\). In a larger design, the relevant question is whether a column is well approximated by a combination of the other columns, not only whether it resembles one predictor at a time.
Near dependence also affects numerical computation. Least squares requires solving equations built from the design matrix. When columns are almost dependent, small rounding errors or small changes in the observed data can cause noticeable changes in the estimated coefficients. Rescaling columns may improve numerical handling when units differ, but rescaling does not remove the underlying overlap in information. Nor does a unique computed answer, by itself, show that the coefficient estimates are stable.
In applications, interpret coefficient uncertainty alongside the fitted values and the purpose of the model. A model can predict well over the observed combinations of predictors while individual coefficients remain imprecise. That does not make the coefficient estimates interchangeable with predictions: it means the design contains limited information for separating the individual effects.
Common Exam Traps
- Ignoring the intercept. A predictor can be an affine function of another predictor, making the full design dependent even when the raw predictor columns are not multiples.
- Calling high correlation exact dependence. A correlation close to \(1\) or \(-1\) signals near proportionality; exact dependence requires equality, subject to nonzero column variation.
- Using a raw determinant as a dependence score. Its magnitude changes with predictor units. Normalize it, or use the equivalent \(1-r^2\) in the two-predictor setting.
- Checking only pairs in a larger model. A predictor can be an exact or near combination of several columns while its pairwise correlations are not extreme.
- Assuming a good fit guarantees precise coefficients. Fitted values and individual coefficient estimates answer different questions; overlapping predictors can make the latter uncertain.
Check Your Understanding
For each design, distinguish exact from near dependence and use a scale-aware calculation where the information permits it.
- A model with an intercept has predictor values \(x_1=(-2,-1,0,1,2)\) and \(x_2=3x_1-4\). State whether there is exact dependence, give a dependence relation among design columns, and state the rank of the design. (Answer: Exact dependence; \(X_2=3X_1-4X_0\); rank \(2\).)
- Two centered predictors have \(S_{11}=16\), \(S_{22}=25\), and \(S_{12}=-18\). Calculate \(r\), \(1-r^2\), and the determinant. (Answer: \(r=-0.9000\), \(1-r^2=0.1900\), determinant \(76\); the predictors are not exactly proportional but show substantial pairwise association.)
- In a model with an intercept, one predictor column is \(x_2=0.02x_1\), where \(x_1\) is nonconstant. Does the small multiplier make the design merely near-dependent, and what is the centered correlation? (Answer: It is exact dependence; \(r=1\). A nonzero change of scale does not turn exact dependence into near dependence.)
- Three centered columns satisfy \(x_3=x_1+x_2\), where \(x_1=(-1,-1,1,1)\) and \(x_2=(-1,1,-1,1)\). Calculate the correlations of \(x_3\) with each of \(x_1\) and \(x_2\), then decide whether pairwise correlations detect the exact dependence. (Answer: Both correlations are \(1/\sqrt{2}\), approximately \(0.7071\); the three columns are exactly dependent despite neither pairwise correlation being near \(1\) in magnitude.)
- For a centered two-predictor model, \(r=0.98\). By what factor is the standard error of either predictor coefficient multiplied relative to an orthogonal design with the same predictor sums of squares, under independent errors with common variance? (Answer: \(1/\sqrt{1-0.98^2}=1/\sqrt{0.0396}\), approximately \(5.025\).)