Why Relationships Among Predictors Matter
A regression model may include several predictors to describe how an outcome varies with multiple characteristics. But the amount of information in the data about each predictor depends not only on how the outcome varies: it also depends on how the predictors vary in relation to one another. If two predictors move together in nearly the same pattern, the data may provide few observations that distinguish their separate contributions.
This issue is called collinearity. It is a feature of the predictor data, not a measure of how well predictions match outcomes. It also does not, by itself, show that a regression model is biased or that an association is causal. The key question here is narrower: do the observed predictor values contain enough independent variation to help estimate separate predictor effects?
The Regression Model and Collinearity
For \(n\) observations and \(p\) predictors, write the multiple linear regression model as
Here \(Y_i\) is the outcome for observation \(i\), \(x_{ij}\) is the value of predictor \(j\) for that observation, \(\beta_0\) is the intercept, \(\beta_j\) is the coefficient for predictor \(j\), and \(\varepsilon_i\) is the model error. The model describes the outcome as a linear function of the predictors plus an error term. In the usual regression interpretation, \(\beta_j\) describes the change in the model’s mean outcome associated with a one-unit increase in predictor \(j\), holding the other predictors fixed.
The definition concerns the values of the predictors across observations. It does not require any particular pattern in the outcome. A pair of predictors can be strongly associated whether the outcome is high, low, or unrelated to them. Nor is collinearity the same as a nonlinear relationship: this topic concerns linear dependence or near-dependence among predictor values.
Exact Redundancy and Approximate Association
With two predictors, exact collinearity occurs when one predictor can be written as a straight-line function of the other for every observation. For example, if \(x_{i2}=2x_{i1}\) for all \(i\), the observed values of the second predictor contain no variation beyond a rescaled version of the first. With several predictors, exact redundancy can also involve a linear combination of multiple other predictors.
Approximate collinearity means the observations lie near, but not exactly on, a linear relationship. In that case, the predictors do have some separate variation, but it may be limited. The distinction is about whether the relationship is exact, not about whether the relationship seems “large enough” by a universal cutoff. How consequential an approximate relationship is depends on the data and the estimation question.
Worked Example: Recognizing Exact Redundancy
Suppose a hypothetical data set records two predictors, \(x_1\) and \(x_2\), for three observations.
| Observation | \(x_1\) | \(x_2\) |
|---|---|---|
| 1 | 1 | 2 |
| 2 | 2 | 4 |
| 3 | 3 | 6 |
For every row, \(x_2=2x_1\): \(2=2(1)\), \(4=2(2)\), and \(6=2(3)\). This is an exact linear relationship, so the predictors are exactly collinear in these observations. Knowing \(x_1\) determines \(x_2\) without error. The table therefore provides no cases in which one of these predictors changes independently of the other.
This conclusion comes entirely from the predictor columns. No outcome values are needed to identify the exact relationship. The implications for estimating coefficients in such a model are a separate question; the important point here is that the observed data do not vary the two predictors independently.
Worked Example: Recognizing Approximate Association
Now consider a different hypothetical data set. The values of \(x_2\) are close to twice the values of \(x_1\), but not exactly twice them.
| Observation | \(x_1\) | \(x_2\) | \(x_2-2x_1\) |
|---|---|---|---|
| 1 | 1 | 2.1 | 0.1 |
| 2 | 2 | 3.9 | -0.1 |
| 3 | 3 | 6.2 | 0.2 |
| 4 | 4 | 7.8 | -0.2 |
The departures from the proposed line \(x_2=2x_1\) are calculated row by row. For observation 1, \(2.1-2(1)=0.1\); for observation 2, \(3.9-2(2)=-0.1\); for observation 3, \(6.2-2(3)=0.2\); and for observation 4, \(7.8-2(4)=-0.2\). Since none of these departures is zero, the relationship is not exact. Since each departure is small relative to the predictor values, the points are close to the line, illustrating approximate collinearity.
The table also shows why approximate collinearity is about information, not simply the fact that two columns are associated. There is some variation away from the line, but little compared with the shared pattern. Those small departures are the main opportunities in these data to distinguish the predictors’ separate contributions.
Why Independent Predictor Variation Helps
The “holding other predictors fixed” part of a coefficient interpretation matters. To learn about the association of \(x_1\) with the outcome separately from \(x_2\), it helps to observe cases where \(x_1\) differs while \(x_2\) is similar. More generally, estimation of separate effects is better informed when the data include predictor combinations that allow their contributions to be distinguished.
A simple contrast illustrates the logic. If two observations have the same value of \(x_2\), but \(x_1\) differs by \(\Delta x_1\), then the model’s difference in mean outcome between them is
The \(\beta_2 x_2\) contribution is the same in both observations, so it cancels in the contrast. This is why comparisons that vary \(x_1\) while keeping \(x_2\) fixed can inform the separate \(x_1\) effect. If the data instead show \(x_1\) and \(x_2\) nearly always changing together, such comparisons may be scarce. Then the observed outcome differences may not clearly indicate which predictor is responsible for which part of the pattern.
Worked Example: Predictor Combinations and Separate Comparisons
Suppose four hypothetical observations have predictor values and outcomes as shown below. The outcomes are generated exactly by the illustrative model \(Y_i=0+2x_{i1}+3x_{i2}\), with no error departures.
| Observation | \(x_1\) | \(x_2\) | \(Y\) |
|---|---|---|---|
| 1 | 1 | 1 | 5 |
| 2 | 1 | 2 | 8 |
| 3 | 2 | 1 | 7 |
| 4 | 2 | 2 | 10 |
Check the model row by row. For observation 1, \(0+2(1)+3(1)=5\). For observation 2, \(0+2(1)+3(2)=8\). For observation 3, \(0+2(2)+3(1)=7\). For observation 4, \(0+2(2)+3(2)=10\). Thus each listed outcome is consistent with the stated model.
This set of predictor combinations includes comparisons that vary one predictor while holding the other fixed. Comparing observations 1 and 3 keeps \(x_2=1\) and increases \(x_1\) from 1 to 2; the outcome increases from 5 to 7, a difference of 2. Comparing observations 1 and 2 keeps \(x_1=1\) and increases \(x_2\) from 1 to 2; the outcome increases from 5 to 8, a difference of 3. These contrasts match the model’s respective coefficients.
This example is a calculation illustrating why predictor combinations matter, not a claim that every data set will follow this model exactly. The four rows provide separate changes in each predictor. If observations instead varied only along a pattern where both predictors always increased together, it would be harder to use observed changes to distinguish their separate effects.
What Collinearity Does—and Does Not—Tell You
Collinearity describes the information structure of the predictors. It does not say that a predictor is irrelevant, that the outcome model is incorrect, or that the overall predictions must be poor. A model can predict outcomes reasonably well while the data provide limited information about how to divide an observed pattern between closely related predictors. Conversely, finding some association between predictors alone does not establish that coefficient estimates will be unusable.
The central distinction is between predicting from a shared pattern and learning separate effects. If two predictors carry similar information in the observed data, their shared pattern may still help describe or predict outcomes. But the data may offer fewer informative comparisons for deciding how much of that pattern belongs to each predictor separately. The degree of concern depends on how much independent variation is present and on what conclusion the analysis needs to support.
Collinearity is also not a causal conclusion. A regression coefficient’s “holding other predictors fixed” interpretation describes a model comparison; it does not establish that changing a predictor would cause the outcome to change. Causal interpretation requires appropriate study design and assumptions beyond the predictor relationships discussed here.
Common Mistake: Treating Correlated Predictors as Identical
A common mistake is to reason, “These predictors are associated, so the regression cannot use either one,” or, in the opposite direction, “They are not perfectly related, so there is no collinearity concern.” Both conclusions are too strong. Exact redundancy means one predictor adds no independent variation beyond the exact linear relationship in the observed data. Approximate collinearity leaves some distinct variation, but that variation may be limited. The appropriate question is not whether the predictors have any relationship at all; it is how much independent information the data provide for estimating their separate effects.
Check Your Understanding
Answer each question using the definitions and reasoning in this tutorial.
- In a data set, \(x_2=5x_1\) for every observation. Is this exact or approximate collinearity? What feature of the data supports your answer?
- A predictor \(x_2\) is usually close to \(3x_1\), but several values depart from that line. Explain why the relationship is not exact and why it could still limit information about separate effects.
- Why does comparing observations with the same \(x_2\) but different \(x_1\) help explain the coefficient for \(x_1\)?
- Does collinearity, by itself, show that a regression model predicts poorly? Explain the distinction between shared predictor information and prediction performance.
- Two predictors are associated in the observed data. What additional question should you ask before deciding how consequential that association is for estimating their separate effects?