Why Pairwise Correlations Can Miss Redundancy
A correlation screen compares predictors two at a time. That is useful, but a predictor need not closely track any one other predictor to be predictable from several of them together. If a predictor is nearly a weighted sum of other predictors, it can be redundant in a multiple regression even when no single pairwise correlation looks especially large.
An auxiliary regression makes this possibility visible. To assess one predictor, treat it as the response and regress it on the other predictors in the model. The resulting \(R^2\) summarizes how much of its variation is accounted for by those predictors collectively. This is a diagnostic about the predictor data, not a regression of the study outcome.
With an intercept, the auxiliary-regression coefficient of determination is
Here \(\mathrm{RSS}_j\) is the sum of squared differences between the observed values of \(X_j\) and their auxiliary-regression fitted values; \(\mathrm{TSS}_j\) is the total sum of squared deviations of \(X_j\) from its sample mean \(\bar{x}_j\). Thus \(R_j^2\) is near 1 when little variation in \(X_j\) remains unexplained by the other predictors. This is the same definition of \(R^2\) used in ordinary least-squares regression.
A Combination Can Matter More Than Any Pair
The key distinction is between predicting a variable from one other variable and predicting it from a set of variables. Pairwise correlations describe the first kind of relationship. An auxiliary regression measures the second, allowing the fitted combination to assign a different coefficient to each predictor.
Worked Example: Four Modest Pairwise Correlations, One Strong Combination
Consider eight hypothetical observations on five centered predictors \(A,B,C,D,E\). The values are constructed so that \(A,B,C,D\) are mutually uncorrelated and \(E=A+B+C+D+0.2Q\), where \(Q\) is another centered column orthogonal to all four. The table gives the observed predictor values; \(Q\) is included to make the small part of \(E\) not explained by \(A,B,C,D\) explicit.
| Observation | A | B | C | D | Q | E |
|---|---|---|---|---|---|---|
| 1 | 1 | 1 | 1 | 1 | 1 | 4.2 |
| 2 | 1 | 1 | -1 | -1 | -1 | -0.2 |
| 3 | 1 | -1 | 1 | -1 | -1 | -0.2 |
| 4 | 1 | -1 | -1 | 1 | 1 | 0.2 |
| 5 | -1 | 1 | 1 | 1 | -1 | 1.8 |
| 6 | -1 | 1 | -1 | -1 | 1 | -1.8 |
| 7 | -1 | -1 | 1 | -1 | 1 | -1.8 |
| 8 | -1 | -1 | -1 | 1 | -1 | -2.2 |
Every column has mean 0. Each of \(A,B,C,D,Q\) has sum of squares 8, and distinct columns among them have cross-products summing to 0. Therefore, the pairwise correlations among \(A,B,C,D\) are all 0. For example, the correlation of \(A\) and \(E\) is
The same calculation applies to \(B,E\), \(C,E\), and \(D,E\), so none of those four pairwise correlations exceeds about 0.50 in magnitude. But in the auxiliary regression of \(E\) on \(A,B,C,D\), the fitted values are exactly \(A+B+C+D\). The residuals are \(0.2Q\), so \(\mathrm{RSS}=0.2^2(8)=0.32\), while \(\mathrm{TSS}=8(4+0.04)=32.32\). Hence
The four predictors together explain about 99.01% of the observed variation in \(E\), despite each having only a modest pairwise correlation with it. This is the kind of redundancy a pairwise screen can understate.
Why the Pattern Is Possible
A short calculation shows why adding several predictors can make a substantial difference. Suppose \(Z_1,\ldots,Z_k\) are centered, mutually uncorrelated predictors, each with the same sum of squares \(S\). Suppose a target predictor \(X\) is their sum plus a centered residual \(e\) that is uncorrelated with each \(Z_m\), with sum of squares \(cS\), where \(c>0\). Then \(X=\sum_{m=1}^{k}Z_m+e\).
For any one component \(Z_m\), its cross-product with \(X\) is \(S\), because its cross-products with the other components and with \(e\) are zero. The sum of squares of \(X\) is \(kS+cS=(k+c)S\). Therefore, the correlation of \(Z_m\) with \(X\) is
The combination \(\sum_{m=1}^{k}Z_m\), however, leaves only \(e\) as its auxiliary-regression residual. Its total sum of squares is \((k+c)S\), and its residual sum of squares is \(cS\). Thus
For example, with \(k=9\) and \(c=0.1\), each pairwise correlation is about \(1/\sqrt{9.1}=0.3315\), while the auxiliary \(R^2\) is \(9/9.1\approx0.9890\). This derivation does not require a special statistical theorem: it follows directly from the sums of squares and cross-products. It demonstrates that many modest individual associations can combine into a very strong linear prediction.
Using Auxiliary Regressions as a Diagnostic
For each predictor under review, fit an auxiliary regression using the same observations and the other predictors in the intended model. Include an intercept, and calculate or read the auxiliary regression’s \(R^2\). A high value means that the predictor is close to a linear combination of the others in this sample. A low value means that the others do not explain much of its variation through that linear combination; it does not guarantee that every possible modeling concern is absent.
Use it as the response in its auxiliary regression; do not use the study outcome as the response for this diagnostic.
Use the other predictors from the model and include an intercept. Keep the observations consistent so the diagnostic compares the intended columns of predictor data.
Examine how much of the predictor’s variation the combination explains, and consider the model’s purpose and the consequences of the observed redundancy. Do not treat one numerical cutoff as a universal decision rule.
Worked Example: Reading an Auxiliary Regression Summary
In a hypothetical model of clinic appointment duration, an analyst checks whether a predictor measuring room-preparation time is redundant with three other predictors. The auxiliary regression uses 60 appointments, includes an intercept, and has total sum of squares \(160\) minutes-squared and residual sum of squares \(12.8\) minutes-squared. Its pairwise correlations with the three other predictors are \(0.18\), \(-0.27\), and \(0.22\).
The auxiliary coefficient of determination is
Thus the three predictors collectively explain 92% of the observed variation in room-preparation time in these 60 appointments, leaving 8% unexplained by this fitted linear combination. Each listed pairwise correlation is modest, yet the multiple \(R^2\) is high. The analyst should therefore investigate whether the predictors carry overlapping information and how that affects the intended model. The calculation alone does not establish that any predictor must be removed, and it says nothing about whether room-preparation time causes appointment duration.
The auxiliary regression is a descriptive calculation on the predictor values. It does not require the study outcome to satisfy a regression error model, and normality of an auxiliary regression’s errors is not needed to compute its \(R^2\). Its interpretation does depend on what has been included: changing the set of other predictors, the observations, or the form of the fitted relationship can change the result. In particular, \(R^2\) here addresses linear prediction using the specified predictors; it is not a guarantee about every nonlinear relationship or every data collection problem.
Why the Diagnosis Matters
Redundancy among predictors makes it harder to distinguish their separate contributions in a multiple regression. When one predictor can be closely reconstructed from others, there may be many combinations of coefficient values that produce similar fitted values over the observed data. An auxiliary regression flags this structure using the predictors themselves, before interpreting separate regression coefficients.
A diagnostic is a prompt to understand the model, not an automatic instruction to delete a variable. A predictor may be important to the research question even when it overlaps with others. Decisions should reflect the purpose of the model, the meaning of the variables, and whether the available data can support the comparisons the coefficients are meant to describe. Pairwise correlations remain a useful first screen, but they do not replace checking predictors against combinations of other predictors.
Common Mistake
- Wrong reasoning: “No correlation between any two predictors is large, so the predictor set has no redundancy.”
- Correction: A predictor may be well explained by several others together. Fit its auxiliary regression and examine the resulting multiple \(R^2\); pairwise correlations alone cannot rule out that pattern.
- Another mistake: Treating a high auxiliary \(R^2\) as proof that a predictor is useless or that the main model is invalid. It is evidence of overlap in the observed predictor data, not a decision rule or a causal conclusion.
Check Your Understanding
Answer these questions using the definitions and calculations in this tutorial.
- In an auxiliary regression for \(X_3\), what is the response, and which predictors are explanatory variables?
- An auxiliary regression has \(\mathrm{TSS}=250\) and \(\mathrm{RSS}=25\). Calculate its \(R^2\) and state what the result means.
- Why can several small pairwise correlations coexist with a high auxiliary \(R^2\)?
- Does a high auxiliary \(R^2\) show that the predictor has no scientific value? Explain.
- Why is the study outcome not the response in an auxiliary regression used to diagnose predictor redundancy?