A Worked VIF Diagnosis With Correlated Predictors

Diagnosing Shared Information in a Predictor Set

When several predictors carry similar information, a regression can have difficulty separating their individual contributions. A variance inflation factor (VIF) diagnosis makes this issue specific to each predictor: it asks how well that predictor can be explained by all the other predictors in the model. The result helps describe coefficient uncertainty, but it does not by itself determine whether a predictor belongs in the model.

The calculation uses an auxiliary regression: temporarily make one predictor the response and regress it on the remaining predictors. The fit is not a second model for the scientific outcome. It is a way to measure how much of that predictor's variation is shared with the rest of the predictor set.

Definition: In a multiple linear regression with predictors $X_1,\ldots,X_p$, the auxiliary regression for $X_j$ regresses $X_j$ on the other $p-1$ predictors, with an intercept. Let $R_j^2$ be its coefficient of determination. The variance inflation factor for $X_j$ is $$ \mathrm{VIF}_j=\frac{1}{1-R_j^2}. $$ The VIF is at least 1. A value near 1 means that the other predictors explain little of $X_j$'s variation; larger values indicate more shared information.

Model and Conditions for Interpretation

To connect a VIF to coefficient uncertainty, consider the multiple linear regression model

Y_i=\beta_0+\beta_1X_{i1}+\cdots+\beta_pX_{ip}+\varepsilon_i.

For the usual least-squares variance interpretation, assume the predictor values are treated as fixed, the predictor columns together with the intercept have full rank, and the errors have mean zero, common variance $\sigma^2$, and are uncorrelated across observations. Normal errors are additionally needed for exact small-sample $t$-inference, but not for the VIF calculation itself. VIFs depend on the predictor columns, not on the observed values of $Y$.

The full-rank condition matters: if one predictor is an exact linear combination of the others, its auxiliary $R_j^2$ is 1, so the VIF formula has a zero denominator. In that case, the individual least-squares coefficients cannot all be uniquely estimated. A large but finite VIF instead describes substantial, but not exact, redundancy.

Under these conditions, the VIF compares the variance of a least-squares coefficient with the variance it would have if that predictor were orthogonal to the other predictors, holding its own variation and the error variance fixed. Thus the corresponding standard-error multiplier is $\sqrt{\mathrm{VIF}_j}$. This is a comparison, not a statement that every fitted model will have that standard error in particular units.

Worked Example: Auxiliary Regressions for Three Correlated Predictors

Suppose a hypothetical dataset contains eight plots and three centered predictor scores: $X_1$ for soil moisture, $X_2$ for shade, and $X_3$ for nutrient level. The values are constructed for teaching, not taken from a real study. The columns are centered, have equal sums of squares, and have sample correlations $r_{12}=0.8$, $r_{13}=0.3$, and $r_{23}=0.3$. The table shows the centered scores rounded to four decimal places; the calculations use the exact construction underlying the table.

Plot	$X_1$	$X_2$	$X_3$
1	1	1.4	1.3487
2	-1	-0.2	-1.1487
3	1	0.2	-0.7487
4	-1	-1.4	0.5487
5	1	1.4	1.3487
6	-1	-0.2	-1.1487
7	1	0.2	-0.7487
8	-1	-1.4	0.5487

For a centered response $X_1$ regressed on centered $X_2$ and $X_3$, the auxiliary-regression coefficients can be calculated from the predictor correlations. The two-predictor normal equations give

\hat{\gamma}_{2}=\frac{r_{12}-r_{13}r_{23}}{1-r_{23}^2} =\frac{0.8-(0.3)(0.3)}{1-0.3^2} =\frac{0.71}{0.91}=0.7802, $$ $$ \hat{\gamma}_{3}=\frac{r_{13}-r_{12}r_{23}}{1-r_{23}^2} =\frac{0.3-(0.8)(0.3)}{0.91} =\frac{0.06}{0.91}=0.06593.

Because the response and predictors are centered, the auxiliary regression's fitted values are $0.7802X_2+0.06593X_3$. Its coefficient of determination is the sum of each coefficient multiplied by the response's correlation with that predictor:

R_1^2=(0.7802)(0.8)+(0.06593)(0.3) \approx 0.64396, \qquad \mathrm{VIF}_1=\frac{1}{1-0.64396}\approx 2.809.

Repeating the calculation for $X_2$ gives $R_2^2\approx0.64396$ and $\mathrm{VIF}_2\approx2.809$, by the symmetry of the correlations. For $X_3$, the auxiliary regression is on $X_1$ and $X_2$. Its coefficients are

\hat{\gamma}_{1} =\frac{0.3-(0.3)(0.8)}{1-0.8^2} =\frac{0.06}{0.36} =0.1667, \qquad \hat{\gamma}_{2}=0.1667.

Therefore $R_3^2=(0.1667)(0.3)+(0.1667)(0.3)=0.1000$, and $\mathrm{VIF}_3=1/(1-0.1000)=1.111$. In this predictor set, $X_1$ and $X_2$ share substantial information, while the other two predictors together explain only 10% of the variation in $X_3$.

From Auxiliary Fit to Coefficient Uncertainty

The link between $R_j^2$ and coefficient uncertainty can be seen by asking how much unique variation remains in $X_j$ after accounting for the other predictors. In an auxiliary regression, the residual sum of squares is the fraction $1-R_j^2$ of the original centered sum of squares. As that unique fraction shrinks, the fitted regression has less information for distinguishing the contribution of $X_j$ from the contributions of its correlated predictors.

More precisely, under the stated regression assumptions, the coefficient variance is proportional to the inverse of that remaining fraction. Relative to the orthogonal-predictor comparison, the variance multiplier is $1/(1-R_j^2)$, which is exactly the VIF. Taking square roots converts a variance multiplier to a standard-error multiplier.

Worked Example: Translating the VIFs Into Standard-Error Multipliers

Use the VIFs from the three-predictor example to describe how collinearity changes coefficient precision, relative to the orthogonal-predictor comparison with the same predictor variation and error variance.

For $X_1$ and $X_2$, the standard-error multiplier is

\sqrt{2.809}\approx1.676.

Thus, under the model assumptions, each coefficient's standard error is about 1.676 times its orthogonal-predictor comparison value: about 67.6% larger. For $X_3$, the multiplier is

\sqrt{1.111}\approx1.054.

Its standard error is about 5.4% larger than that comparison value. These percentages describe the effect of predictor geometry on coefficient precision; they do not give the standard errors themselves, which also depend on the residual variation and the scale and spread of the predictors.

Why Pairwise Correlations Are Not the Whole Diagnosis

A correlation between two predictors is a useful first screen, but a VIF uses all the other predictors at once. A predictor may be well explained by a combination of several other predictors even if no single pairwise correlation looks especially large. Conversely, one striking pairwise correlation does not by itself tell you the VIF for every predictor in a larger model. The auxiliary regression makes the set being assessed explicit.

In the worked data, the pairwise correlation $r_{12}=0.8$ already signals considerable overlap between $X_1$ and $X_2$. The auxiliary results show that this overlap remains substantial after including the third predictor: $R_1^2$ and $R_2^2$ are both about 0.644. They also show that $X_3$ has a different diagnosis: $X_1$ and $X_2$ together explain just 10% of its variation. A VIF should therefore be reported predictor by predictor, not as a single label for the entire regression.

Worked Example: A Two-Predictor Comparison

In another hypothetical model, suppose two centered, equally scaled predictors have correlation $r=0.6$. Find the VIF for each predictor and explain the result.

Regressing either predictor on the other gives $R^2=r^2=0.6^2=0.36$. Therefore,

\mathrm{VIF}=\frac{1}{1-0.36} =\frac{1}{0.64} =1.5625.

The standard-error multiplier relative to the orthogonal-predictor comparison is $\sqrt{1.5625}=1.25$. Under the regression assumptions, each coefficient's standard error is therefore 25% larger than its comparison value. This is a consequence of the predictors sharing information; it does not establish that either predictor should be removed.

How to Report a VIF Diagnosis

A useful diagnosis connects the calculations to the specific predictor set and avoids treating a VIF as a verdict. State which predictors were included in each auxiliary regression, report $R_j^2$ and $\mathrm{VIF}_j$, and identify which predictors are most explainable by the others. Then describe the implication for coefficient uncertainty in measured terms, such as the standard-error multiplier $\sqrt{\mathrm{VIF}_j}$.

There is no single VIF threshold that automatically makes a model unacceptable. The practical importance depends on the analysis goal, the amount of data, the uncertainty that is tolerable, and whether the individual coefficients need separate interpretations. A VIF diagnoses overlap among predictors; deciding how to respond requires considering the purpose of the model.

Common Mistake

Wrong reasoning: “The largest VIF is 2.809, so the model is invalid and one of those predictors must be deleted.”
Correction: A VIF quantifies how much the predictor relationships inflate coefficient variance relative to an orthogonal-predictor comparison. It does not test whether the model is valid or choose which predictors to keep. In this example, it identifies greater uncertainty for the $X_1$ and $X_2$ coefficients than for the $X_3$ coefficient; the modeling decision must also reflect the question and intended interpretation.

Key takeaway: Fit one auxiliary regression for each predictor, using all the remaining predictors. Its $R_j^2$ gives $\mathrm{VIF}_j=1/(1-R_j^2)$; the square root of the VIF is the corresponding standard-error multiplier under the regression assumptions. Read the values as a predictor-specific diagnosis of shared information and coefficient uncertainty, not as automatic model-selection rules.

Check Your Understanding

Use the calculations and definitions in this tutorial to answer the following questions.

In the three-predictor example, what is the response in the auxiliary regression used to calculate $\mathrm{VIF}_2$, and which predictors are its explanatory variables?
If an auxiliary regression has $R_j^2=0.50$, calculate $\mathrm{VIF}_j$ and the corresponding standard-error multiplier.
In the worked dataset, which predictor has the smallest VIF? What does its auxiliary $R^2$ say about its variation?
Why can pairwise correlations alone fail to provide a complete VIF diagnosis when there are several predictors?
Explain why a high VIF does not, by itself, show that a predictor should be removed.