Why Measure Redundancy With VIF?
Two predictors can carry overlapping information. In a regression model, that overlap can make it harder to distinguish their separate contributions: the fitted predictions may remain useful, while individual coefficient estimates become less precise. A variance inflation factor (VIF) measures this effect for one predictor at a time by relating it to the other predictors in the model.
VIF is a diagnostic for linear redundancy among predictors. It does not measure a predictor’s association with the response, prove that a model is poor, or prescribe which variable to remove. Its interpretation depends on the model’s purpose and the consequences of losing information.
The auxiliary regression uses one predictor as its response and the remaining predictors as explanatory variables. It is not the regression of the original outcome on the predictors. A value \(R_j^2=0\) gives \(\operatorname{VIF}_j=1\): the other predictors do not linearly explain \(x_j\) in this regression. As \(R_j^2\) approaches 1, the VIF grows. If the auxiliary regression fits \(x_j\) exactly, its \(R_j^2\) is 1 and the corresponding coefficient cannot be separately estimated in the full model.
Where the Formula Comes From
The variance interpretation follows from ordinary least squares. Consider a multiple linear regression with an intercept:
For the derivation, treat the predictor values as fixed and assume the errors have mean zero, common variance \(\sigma^2\), and are uncorrelated. Also assume the predictors are not in exact linear dependence, so the coefficients can be estimated. These assumptions make the usual conditional OLS variance formula applicable.
Center \(x_j\) and the other predictors around their sample means. Regress the centered \(x_j\) values on the centered values of the other predictors, and call the residual for observation \(i\) \(r_{ij}\). Let \(SST_j=\sum_i(x_{ij}-\bar{x}_j)^2\) be the total centered sum of squares for \(x_j\), and let \(RSS_j=\sum_i r_{ij}^2\) be its auxiliary-regression residual sum of squares. With an intercept, the auxiliary \(R^2\) satisfies
To see how this residual variation determines the coefficient variance, remove from \(x_j\) the part that the other predictors explain. The remaining vector \(r_j\) is orthogonal to those other predictors. Consequently, the coefficient on \(x_j\) in the full regression is the slope obtained by relating this residualized predictor to the response after the other predictors have been accounted for. The random error in that slope’s numerator has variance \(\sigma^2\sum_i r_{ij}^2\), while its denominator is \((\sum_i r_{ij}^2)^2\). Therefore,
If \(x_j\) were orthogonal to all the other predictors while keeping the same \(SST_j\), the coefficient variance would be \(\sigma^2/SST_j\). Dividing the actual variance by this orthogonal-predictor benchmark gives
This is the variance inflation interpretation: under the stated OLS assumptions, the VIF is the factor by which the coefficient variance is multiplied relative to that benchmark. The corresponding standard error is multiplied by \(\sqrt{\operatorname{VIF}_j}\), not by the VIF itself.
Calculating and Interpreting VIF
The calculation has two stages: fit the auxiliary regression for each predictor, then transform its \(R^2\). For example, \(R_j^2=0.75\) gives \(\operatorname{VIF}_j=1/(1-0.75)=4\). This means four times the benchmark variance and twice the benchmark standard error, not four times the standard error.
When calculating VIFs in practice, use the same set of observations and the same predictors as in the model being diagnosed. If different auxiliary regressions use different rows because of missing values, their results no longer describe redundancy on a common analysis sample. For categorical predictors, the relevant design columns depend on the model’s encoding and contrasts; an exact linear dependence among columns prevents unique coefficient estimation.
Worked Example: Turning an Auxiliary Regression Into a VIF
A hypothetical pricing analysis uses a multiple linear regression with an intercept. For one predictor \(x_1\), its centered total sum of squares is \(SST_1=250\). Regressing \(x_1\) on all the other predictors gives a residual sum of squares \(RSS_1=40\). Calculate its auxiliary \(R^2\), VIF, variance ratio, and standard-error multiplier. Method: use the auxiliary-regression sums of squares. The mistake to avoid is using the original outcome model’s \(R^2\).
First calculate the auxiliary coefficient of determination:
Then calculate the VIF:
The direct variance check gives \(SST_1/RSS_1=250/40=6.25\). Thus the coefficient variance is 6.25 times the orthogonal-predictor benchmark, and its standard error is multiplied by
The calculation is internally consistent: \(R_1^2=0.84\), \(1-R_1^2=0.16\), and \(1/0.16=6.25\). This indicates substantial linear redundancy with the other predictors, but does not by itself establish that \(x_1\) should be removed.
Worked Example: Comparing VIFs Across Predictors
A hypothetical claims analysis includes three numeric predictors, \(x_1,x_2,x_3\). Separate auxiliary regressions, each using the same observations, give \(R_1^2=0.75\), \(R_2^2=0.60\), and \(R_3^2=0.20\). Calculate the three VIFs, identify which predictor has the greatest linear redundancy with the others, and state the associated standard-error multiplier. Method: apply the VIF formula to each auxiliary \(R^2\). The mistake to avoid is treating the \(R^2\) values as VIFs themselves.
For \(x_1\):
For \(x_2\):
For \(x_3\):
Among these predictors, \(x_1\) has the greatest linear redundancy with the other two: its auxiliary \(R^2\) and VIF are largest. Its standard error is inflated by a factor of 2 relative to the stated benchmark. The corresponding factors are about 1.581 for \(x_2\) and 1.118 for \(x_3\). The values do not establish that \(x_1\) has the weakest relationship with the outcome; VIF uses no outcome in its calculation.
Common Mistake: Using the Outcome Model’s \(R^2\)
The original model’s \(R^2\) describes how much variation in the outcome is explained by the predictors together. VIF instead uses the \(R^2\) from a separate regression of one predictor on the other predictors. Confusing these two values can produce a plausible-looking but irrelevant calculation.
Worked Example: Separating Outcome Fit From Predictor Redundancy
In a hypothetical portfolio model, the outcome regression has \(R^2=0.90\). For predictor \(x_1\), an auxiliary regression on the other predictors has \(R_1^2=0.75\). The centered sum of squares for \(x_1\) is 400, and its auxiliary residual sum of squares is 100. Calculate the VIF and standard-error multiplier, and check them using the sums of squares. Method: use the predictor-specific auxiliary regression. The common mistake is to substitute the outcome model’s \(R^2=0.90\), yielding \(1/(1-0.90)=10\), which is not the VIF.
The correct VIF uses \(R_1^2=0.75\):
The standard-error multiplier is \(\sqrt{4}=2\). The sums-of-squares check gives
Both checks give a VIF of 4. The outcome model’s \(R^2=0.90\) is not part of this calculation. It describes overall outcome fit, whereas \(R_1^2=0.75\) describes how well the other predictors explain \(x_1\).
What a Large VIF Does—and Does Not—Mean
A large VIF is evidence that a predictor is close to a linear combination of the other predictors in the model. This can make its estimated coefficient less precise and more sensitive to changes in the data or model specification. When predictors overlap, coefficient estimates can also be difficult to interpret as separate effects.
A VIF is not a universal decision rule. There is no single cutoff that, for every dataset and purpose, determines that a predictor must be deleted. The consequences of a given value depend on the sample size, the desired precision, the role of the predictor, and whether the goal is explanation or prediction. High VIF may be important when the analysis depends on precise, separately interpreted coefficients; it may be less consequential for a prediction task if predictive performance remains adequate.
Also, VIF is a linear diagnostic. A low VIF does not rule out every kind of relationship among predictors; it indicates only that the auxiliary linear regression does not explain much of the predictor’s variation. Conversely, an exact linear dependence leads to \(R_j^2=1\), so \(1-R_j^2=0\) and the coefficient variance formula has a zero denominator. In that case the coefficient is not separately identifiable in the specified model; reporting an ordinary finite VIF would be inappropriate.
For a generalized linear model, the same auxiliary regressions can be used as a descriptive screen of linear dependence among the design columns. However, the OLS variance derivation above is not an exact coefficient-variance formula for every generalized linear model. Interpret a VIF there as a predictor-redundancy diagnostic, not as a guaranteed multiplier for the fitted model’s standard error.
Check Your Understanding
For each problem, use the specified auxiliary regression—not the original outcome model—to calculate or interpret VIF.
- A predictor’s auxiliary regression has \(R^2=0.20\). Calculate its VIF and the standard-error multiplier relative to the orthogonal-predictor benchmark. (Answer: VIF \(=1.25\); standard-error multiplier \(=\sqrt{1.25}\approx1.1180\).)
- For a centered predictor, \(SST=160\) and its auxiliary-regression residual sum of squares is 40. Find the auxiliary \(R^2\), VIF, and standard-error multiplier. (Answer: \(R^2=0.75\); VIF \(=4\); standard-error multiplier \(=2\).)
- Three predictors have auxiliary \(R^2\) values of 0.64, 0.80, and 0.10. Calculate their VIFs and identify the predictor with the greatest linear redundancy. (Answer: VIFs \(=2.7778,\ 5,\ 1.1111\), respectively; the second predictor has the greatest linear redundancy.)
- An outcome regression has \(R^2=0.96\), while one predictor’s auxiliary regression has \(R^2=0.50\). What is its VIF, and why is using 0.96 incorrect? (Answer: VIF \(=2\); 0.96 measures outcome fit, while VIF uses the predictor’s auxiliary \(R^2=0.50\).)
- A predictor is exactly the sum of two other predictors in the model. What does its auxiliary \(R^2\) imply, and can the ordinary full-model coefficient for that predictor be uniquely estimated? (Answer: auxiliary \(R^2=1\), so the VIF is infinite or undefined; the coefficient is not uniquely estimable because of exact linear dependence.)