A Worked Collinearity Diagnosis Across Multiple Measures

Diagnosing Collinearity Requires More Than One Number

A regression can fit its observed responses well while giving imprecise estimates of individual predictor effects. This can happen when predictors contain much of the same information: the model may estimate their combined contribution reliably but have difficulty separating their conditional contributions. A useful diagnosis therefore considers multiple measures, rather than treating a high correlation or a single threshold as a verdict.

This tutorial works through predictor correlations, variance inflation factors (VIFs), condition indices, and variance-decomposition proportions. These measures describe the predictor design, not the response fit itself. Their interpretation belongs alongside, not in place of, the regression model and its inferential assumptions.

Model and assumptions: Consider the multiple linear regression model \(Y=\beta_0+\beta_1X_1+\cdots+\beta_pX_p+\varepsilon\). For the standard coefficient variance and exact \(t\)- and \(F\)-inference formulas used below, assume the design matrix is fixed, the errors have mean zero and common variance \(\sigma^2\), and the errors are independent and normally distributed. Collinearity diagnostics can be calculated without normal errors. Exact collinearity makes some coefficients unidentifiable; near collinearity leaves coefficients identifiable but can make them imprecise.

First Measure: Predictor Correlations and VIFs

A predictor correlation matrix can quickly reveal pairs that move together. It does not, by itself, give a complete diagnosis: one predictor can be closely approximated by a combination of several others even if no individual pairwise correlation is especially large. VIFs address this by measuring how well each predictor is linearly explained by all the other predictors.

Definition: For predictor \(X_j\), regress it on the other \(p-1\) predictors, including an intercept, and let the coefficient of determination be \(R_j^2\). Its variance inflation factor is \(\operatorname{VIF}_j=1/(1-R_j^2)\). The tolerance is \(1/\operatorname{VIF}_j=1-R_j^2\). With centered, standardized predictors and correlation matrix \(R\), the VIFs are the diagonal elements of \(R^{-1}\).

The VIF is a factor for variance, not standard error. Relative to the variance that would result if \(X_j\) were orthogonal to the other predictors with the same sum of squares, its coefficient variance is multiplied by \(\operatorname{VIF}_j\); its standard error is multiplied by \(\sqrt{\operatorname{VIF}_j}\). A VIF of 9, for example, corresponds to three times the standard error, not nine times the standard error.

Worked Example: Pairwise Correlation and VIF

A hypothetical pricing study has three centered, standardized rating measures. The predictor correlation matrix is

R= \begin{pmatrix} 1&0.95&0\\ 0.95&1&0\\ 0&0&1 \end{pmatrix}.

Method. Regress each predictor on the other two, or invert the correlation matrix. The zero correlations show that \(X_3\) is orthogonal to the correlated pair. For \(X_1\), the regression on \(X_2\) and \(X_3\) has \(R_1^2=0.95^2+0^2=0.9025\). Thus

\operatorname{VIF}_1=\frac{1}{1-0.9025} =\frac{1}{0.0975} =10.2564.

By symmetry, \(\operatorname{VIF}_2=10.2564\). Since \(X_3\) is uncorrelated with both other predictors, \(R_3^2=0\) and \(\operatorname{VIF}_3=1\). The tolerance for each of \(X_1\) and \(X_2\) is \(0.0975\); for \(X_3\), it is 1. The corresponding standard-error inflation for the first two coefficients is \(\sqrt{10.2564}=3.2026\), to four decimal places.

Check. Inverting the \(X_1,X_2\) block gives diagonal entries \(1/(1-0.95^2)=10.2564\), agreeing with the separate regressions. The large VIFs identify a pair whose conditional effects may be uncertain. Common mistake avoided: using \(1/(1-0.95)=20\) treats the correlation itself as \(R_j^2\). The correct calculation squares the correlation in this two-predictor setting, giving \(1/(1-0.95^2)=10.2564\).

Why VIF Inflates Coefficient Variance

The reason for the VIF formula can be seen by isolating the part of a predictor that is not explained by the others. Let \(Z_j\) be the residual from regressing centered \(X_j\) on the other centered predictors. After accounting for those other predictors, the information that identifies \(\beta_j\) comes from \(Z_j\), not from the portion of \(X_j\) they can already predict.

\operatorname{Var}(\hat{\beta}_j\mid X) =\frac{\sigma^2}{\sum_{i=1}^n Z_{ij}^2} =\frac{\sigma^2}{S_{jj}(1-R_j^2)} =\frac{\sigma^2}{S_{jj}}\operatorname{VIF}_j.

Here \(S_{jj}=\sum_i(X_{ij}-\bar X_j)^2\). The middle equality follows because the residual sum of squares from regressing \(X_j\) on the other predictors is \(S_{jj}(1-R_j^2)\), by the definition of \(R_j^2\). If the predictors are orthogonal, \(R_j^2=0\), so the variance is \(\sigma^2/S_{jj}\). As \(R_j^2\) approaches 1, less independent variation remains to estimate \(\beta_j\), and the variance increases. At \(R_j^2=1\), the denominator is zero: this is exact, not merely near, collinearity, and the coefficient is not separately identifiable.

Condition Indices and Variance Decomposition

VIFs focus on each predictor in turn. Eigenvalue diagnostics instead look for directions in the predictor space that have little variation. To make the comparison meaningful, center and scale the predictor columns before calculating eigenvalues; exclude the intercept. Let \(\lambda_{\max}\) be the largest eigenvalue of the scaled cross-product matrix (or the predictor correlation matrix). For each eigenvalue \(\lambda_k\), the condition index is

\kappa_k=\sqrt{\frac{\lambda_{\max}}{\lambda_k}}.

A small eigenvalue produces a large condition index and signals a direction in which a linear combination of predictors varies little. Common rules of thumb treat indices above about 10 as worth investigating and those above about 30 as potentially serious. These are screening conventions, not universal decision boundaries. Changing the scaling can change condition indices, which is why the scaling convention must be stated.

The condition index identifies a weak direction, but not which coefficients are implicated. Variance-decomposition proportions provide that additional information. If \(q_{jk}\) is the \(j\)th element of the normalized eigenvector for \(\lambda_k\), the contribution of that direction to the variance of coefficient \(j\), apart from a common factor, is \(q_{jk}^2/\lambda_k\). Divide this quantity by its sum across all eigen-directions to obtain the proportion for coefficient \(j\). Large proportions for two or more coefficients on the same high-condition-index direction are evidence that those coefficient variances are being inflated together. This evidence should be interpreted with the correlations and VIFs, not in isolation.

Worked Example: Identifying the Coefficients Behind a Condition Index

A hypothetical operations model uses three centered, standardized predictors with correlation matrix

R= \begin{pmatrix} 1&0.90&0\\ 0.90&1&0\\ 0&0&1 \end{pmatrix}.

Method. The two eigenvalues for the first two predictors are \(1+0.90=1.90\) and \(1-0.90=0.10\); the third eigenvalue is 1. The largest is 1.90, so the condition indices are \(\sqrt{1.90/1.90}=1\), \(\sqrt{1.90/1}=1.3784\), and \(\sqrt{1.90/0.10}=\sqrt{19}=4.3589\). The largest index is associated with the eigenvector \((1,-1,0)/\sqrt{2}\), the direction in which \(X_1\) and \(X_2\) nearly cancel.

For either of the first two coefficients, the low-eigenvalue direction contributes \((1/2)/0.10=5\) to the unnormalized variance decomposition. The high-eigenvalue direction contributes \((1/2)/1.90=0.2632\). Normalizing gives \(5/(5+0.2632)=0.95\) for the low-eigenvalue direction and \(0.05\) for the other. For the third coefficient, the eigenvector direction associated with eigenvalue 1 contributes all of its variance; the other two directions contribute zero.

Check. The two coefficients with 95% of their decomposed variance on the same direction are \(\beta_1\) and \(\beta_2\). The high condition index pinpoints the weak direction, and the variance proportions identify the affected coefficients. In this example the index is a useful signal, but its value alone is not a threshold-based declaration of severe collinearity.

Coefficient Uncertainty Is Not the Same as Poor Fit

Collinearity concerns how well the design separates conditional predictor effects. Overall fit concerns how much response variation the fitted model explains. A model can have a high \(R^2\) and a significant overall \(F\)-test while one or more individual coefficient tests are weak. When correlated predictors have similar estimated effects, their combined contribution can be clear even though the data do not precisely distinguish their separate contributions.

The following hypothetical regression output is constructed to show this distinction. The sample has \(n=40\) observations and three centered predictors. Each predictor has sum of squares 39; \(X_1\) and \(X_2\) have correlation 0.95, and \(X_3\) is orthogonal to both. The fitted slopes are \(2,2,0\), and the residual sum of squares is 152.1. Assume the usual independent, normal, constant-variance error model for the exact tests.

Worked Example: Strong Overall Fit but Uncertain Separate Effects

Method. First calculate the explained sum of squares from the centered predictor cross-product matrix and fitted slopes. The \(X_1,X_2\) cross-product is \(0.95(39)=37.05\). Therefore

\operatorname{SSR} =39(2^2)+39(2^2)+2(2)(2)(37.05) =156+156+296.4 =608.4.

The total sum of squares is \(608.4+152.1=760.5\), so \(R^2=608.4/760.5=0.8000\). There are \(40-3-1=36\) residual degrees of freedom, and \(\operatorname{MSE}=152.1/36=4.225\). The overall test statistic for the three slopes is

F=\frac{\operatorname{SSR}/3}{\operatorname{MSE}} =\frac{608.4/3}{4.225} =\frac{202.8}{4.225} =48.0.

Now calculate the standard errors. The first two predictors have \(\operatorname{VIF}=1/(1-0.95^2)=400/39\), and each has \(S_{jj}=39\). Thus

\operatorname{SE}(\hat{\beta}_1) =\operatorname{SE}(\hat{\beta}_2) =\sqrt{\frac{4.225(400/39)}{39}} =\sqrt{\frac{10}{9}} =1.0541.

Each corresponding \(t\)-statistic is \(2/1.0541=1.8974\). With 36 degrees of freedom, this is below the usual two-sided 5% critical value of approximately 2.028. The estimated slope for \(X_3\) is zero, so its \(t\)-statistic is zero. In contrast, the overall \(F\)-statistic is 48.0, indicating strong evidence that the slopes are not all zero under the stated model.

Check. The results agree: the model explains 80% of the response variation and has a large overall \(F\)-statistic, but the separate tests for the two highly correlated slopes are not individually significant at 5%. This is not a contradiction. Common mistake avoided: concluding that the model must fit poorly because individual slopes are not significant. Here the weakness is precision in separating the two effects, not lack of overall fit.

A Practical Diagnostic Sequence

A diagnosis is clearest when each measure answers a distinct question. The following sequence prevents a single statistic from carrying more meaning than it can support.

Check the predictor structure.
Review units, coding, and the correlation matrix. State whether predictors have been centered and scaled before using eigenvalue diagnostics.

Calculate VIFs.
Identify which predictors are explained by the others, and interpret VIF as a variance multiplier. Do not confuse it with the standard-error multiplier, which is its square root.

Examine condition indices and variance proportions.
Use a high condition index to locate a weak design direction, then check whether several coefficient variance proportions are large on that same direction.

Connect the diagnosis to inference and fit.
Discuss coefficient standard errors and conditional tests separately from \(R^2\), residual error, and the overall \(F\)-test. A design diagnosis alone does not establish poor predictions or poor response fit.

No diagnostic has a universal cutoff that replaces judgment. A large VIF or condition index is a reason to investigate the design and the question being asked. It does not, by itself, establish that a predictor should be removed, that a model is unusable, or that predictions must be inaccurate. In particular, a coefficient can be difficult to interpret as a separate conditional effect while the model’s fitted values remain stable over the range of observed predictor combinations.

Key takeaway: Correlations describe pairwise relationships, VIFs quantify how each predictor is explained by the others, and condition diagnostics locate weak combinations of predictors. Use variance-decomposition proportions to link those combinations to coefficient uncertainty. Then assess overall fit separately.

Check Your Understanding

For each problem, distinguish what the diagnostic measures establish from what they do not establish.

Two standardized predictors have correlation 0.80, and a third is orthogonal to both. Find the VIF for each of the correlated predictors and state the standard-error inflation factor. (Answer: VIF \(=1/(1-0.80^2)=2.7778\) for each; standard-error factor \(=\sqrt{2.7778}=1.6667\).)
A scaled predictor correlation matrix has eigenvalues 1.90, 1.00, and 0.10. Calculate the condition index for the smallest eigenvalue. (Answer: \(\sqrt{1.90/0.10}=\sqrt{19}=4.3589\).)
A model has \(n=50\), four predictors, and \(R^2=0.75\). Calculate adjusted \(R^2\), using \(1-(1-R^2)(n-1)/(n-p-1)\). (Answer: \(1-(0.25)(49/45)=0.7278\).)
A regression has a high \(R^2\), a large overall \(F\)-statistic, and two individually weak slope tests. The two predictors also have high VIFs. Give the most direct interpretation. (Answer: The model may explain response variation well, while collinearity makes the two separate conditional slope estimates imprecise; the weak individual tests do not by themselves imply poor overall fit.)
Two predictors have a low pairwise correlation, but the VIF for one is large when it is regressed on all the others. What can explain this, and what additional diagnostic can help locate the shared dependency? (Answer: A combination of several other predictors may explain it despite low pairwise correlations; scaled-design eigenvalues and condition indices, interpreted with variance-decomposition proportions, can help locate the dependency and identify affected coefficients.)

What You'll Learn