Why Collinearity Makes Attribution Uncertain
When two predictors contain much of the same information, a model may predict well without assigning that information consistently to either one. Small changes in a training sample can make a fitted model favor one predictor over its close substitute. This is especially visible in lasso selection and in variable-importance rankings: the names at the top of a list can change even when the model’s predictions barely change.
The useful question is not simply, “Which variable was selected?” It is whether the pattern persists across resamples, whether prediction performance is stable, and whether the predictors are sufficiently distinct to support separate attribution. The analysis below focuses on these questions. It does not establish that a selected variable causes the outcome.
Measure Selection Stability Across Resamples
A resampling analysis fits the same modeling procedure repeatedly to different training samples or partitions. For lasso, the procedure should specify how predictors are preprocessed, how the penalty is chosen, and which performance metric is recorded. Use the same resampling partitions when comparing candidate models, and carry out preprocessing and tuning using only the training data available in each fit. These controls help distinguish selection changes caused by sampling from changes caused by a different modeling procedure.
A coefficient can be numerically nonzero but too small to count as a meaningful selection under the analysis rule. Choose a numerical tolerance \(\tau\) before counting, and define predictor \(j\) as selected in resample \(b\) when \(|\hat{\beta}_{jb}|>\tau\). For \(B\) fitted resamples, the selection frequency is:
Here, \(I(\cdot)\) equals 1 when its condition is true and 0 otherwise. A frequency near 1 indicates that the predictor is selected consistently under this resampling and fitting procedure; a frequency near 0 indicates infrequent selection. Neither value, by itself, establishes importance for every prediction, causality, or a probability that the predictor is “truly relevant.” If tuning the penalty separately in each resample is part of the intended modeling procedure, the frequencies also reflect that tuning variability.
Worked Example: Selection Frequencies and Stable Validation Error
A hypothetical insurer fits lasso models for claim cost in 10 training resamples. Two strongly correlated, standardized predictors are \(X_A\) and \(X_B\). The fitting protocol is held fixed, and the numerical selection tolerance is \(\tau=0.01\). The table gives the fitted coefficients and validation RMSE for each resample.
| Resample | \(\hat{\beta}_A\) | \(\hat{\beta}_B\) | Validation RMSE |
|---|---|---|---|
| 1 | 0.42 | 0.00 | 0.98 |
| 2 | 0.38 | 0.00 | 1.02 |
| 3 | 0.31 | 0.08 | 0.99 |
| 4 | 0.006 | 0.41 | 1.01 |
| 5 | 0.27 | 0.00 | 1.00 |
| 6 | 0.00 | 0.36 | 0.97 |
| 7 | 0.35 | 0.00 | 1.03 |
| 8 | 0.18 | 0.20 | 1.01 |
| 9 | 0.00 | 0.33 | 0.98 |
| 10 | 0.29 | 0.00 | 1.01 |
Method. Apply the stated tolerance to each coefficient, count selections, then summarize the validation errors.
For \(X_A\), the coefficients exceeding 0.01 occur in resamples 1, 2, 3, 5, 7, 8, and 10. Therefore, \(\widehat{f}_A=7/10=0.70\). For \(X_B\), selections occur in resamples 3, 4, 6, 8, and 9, so \(\widehat{f}_B=5/10=0.50\). Both are selected in resamples 3 and 8; some resamples favor one predictor over the other.
The validation RMSE values sum to \(10.00\), so their mean is \(10.00/10=1.00\). Their range is \(1.03-0.97=0.06\). The individual coefficients and selection statuses vary, while the validation errors remain in a narrow range. This is evidence of unstable attribution alongside relatively stable performance in these resamples—not proof that the two predictors are interchangeable in every future sample.
Common mistake avoided. Counting every exactly nonzero coefficient would include the 0.006 for \(X_A\) in resample 4 and report \(8/10=0.80\). That contradicts the stated rule: \(0.006\) is not greater than \(\tau=0.01\). The consistent calculation is \(7/10=0.70\).
Why Similar Predictors Can Produce Similar Fits
The behavior has a simple mathematical explanation. If two predictors are identical, a linear predictor using coefficients \(\beta_A\) and \(\beta_B\) depends on their sum, not on how that sum is divided between them. In a lasso fit, allocations with the same positive sum also have the same L1 penalty. Thus, multiple coefficient allocations can yield identical fitted values and identical objective values.
For example, the allocations \((1.8,0)\), \((0,1.8)\), and \((0.9,0.9)\) all have coefficient sum 1.8 and L1 penalty 1.8. They produce the same linear predictor when the two columns are identical. With strongly correlated—but not identical—predictors, the equality is only approximate. Small changes in the observations can then favor a different allocation, while the resulting predictions remain close.
This reasoning does not say that lasso always selects exactly one member of a correlated pair, or that every pair of high-correlation predictors will have unstable selections. The observed pattern depends on the data, the penalty, the preprocessing, and the resampling design. The explanation identifies why instability is plausible; the resampling results show whether it occurs in the analysis at hand.
Worked Example: Same Predictions, Different Coefficient Allocations
Consider four hypothetical observations with an intercept and two identical predictors \(X_A=X_B=x\), where \(x=(0,1,2,3)\), and responses \(y=(1,3,5,7)\). Fit lasso using the objective
with the intercept unpenalized. Suppose four resampled fits return coefficient allocations \((1.8,0)\), \((0,1.8)\), \((0.9,0.9)\), and \((0.4,1.4)\). All have nonnegative coefficients with sum 1.8.
Method. First find the fitted values from the common coefficient sum. Since the mean of \(x\) is \(1.5\) and the mean of \(y\) is \(4\), the intercept that minimizes squared error for a slope sum of 1.8 is \(4-(1.8)(1.5)=1.3\). Each allocation therefore gives predictions \(1.3, 3.1, 4.9, 6.7\).
The residuals, defined as observed minus predicted, are \(-0.3,-0.1,0.1,0.3\). Their squared sum is \(0.09+0.01+0.01+0.09=0.20\). With \(n=4\), the squared-error term is \(0.20/(2\cdot4)=0.025\). Every allocation has L1 penalty \(|\beta_A|+|\beta_B|=1.8\), contributing \(0.25(1.8)=0.45\). Thus, each allocation has objective value \(0.025+0.45=0.475\).
At tolerance \(\tau=0.01\), \(X_A\) is selected in three of four allocations and \(X_B\) is also selected in three of four: both frequencies are \(3/4=0.75\). Yet the predictions are identical for all four fits. The training RMSE is \(\sqrt{0.20/4}=\sqrt{0.05}\approx0.2236\), also identical across allocations.
Check. Each coefficient pair sums to 1.8, and each has L1 penalty 1.8; the fitted values and objective therefore agree. In real resampling, highly correlated columns need not be identical, so agreement in performance may be close rather than exact. The example isolates the attribution issue without claiming that all resampled fits will behave this way.
Common mistake avoided. It would be wrong to conclude that \(X_A\) is more useful because one fit assigns it coefficient 1.8, or that the fit with split coefficients is better. Here those allocations have exactly the same objective and predictions.
Assess Importance Without Overstating Individual Rankings
Variable importance is not one universal quantity. Its meaning depends on the model, the importance method, and the performance measure. For permutation importance, one common definition is the increase in a chosen validation loss after a predictor is permuted among validation observations. For mean squared error, an increase means worse predictions after permutation. When predictors are correlated, permuting one while leaving its substitute intact may have little effect: the remaining predictor can still carry much of the shared signal.
This can make the individual importance of both variables appear modest, even when the pair contains useful information. Permuting both as a group can help assess the contribution of their shared information, though group importance is a different question from individual attribution. Individual and group scores should be calculated using the same validation data, fitted model, and defined loss within each comparison. Across resamples, report the scores or their distributions rather than relying only on a single rank.
Worked Example: Individual and Group Importance Across Resamples
A hypothetical risk model is evaluated using five resamples. Importance is defined as the increase in validation MSE after permutation; larger increases indicate a greater deterioration in that fit’s validation predictions. Each column is computed within the same resample and on the same validation observations. The group column permutes \(X_A\) and \(X_B\) together.
| Resample | Permute \(X_A\) only | Permute \(X_B\) only | Permute pair together |
|---|---|---|---|
| 1 | 0.06 | 0.02 | 0.10 |
| 2 | 0.03 | 0.07 | 0.10 |
| 3 | 0.05 | 0.04 | 0.11 |
| 4 | 0.02 | 0.08 | 0.10 |
| 5 | 0.04 | 0.03 | 0.09 |
Method. Compare the individual scores within each row, then calculate column means. \(X_A\) ranks first in resamples 1, 3, and 5, so it ranks first in \(3/5=0.60\) of the resamples. \(X_B\) ranks first in resamples 2 and 4, or \(2/5=0.40\).
For \(X_A\), the mean individual importance is \((0.06+0.03+0.05+0.02+0.04)/5=0.20/5=0.040\). For \(X_B\), it is \((0.02+0.07+0.04+0.08+0.03)/5=0.24/5=0.048\). The group scores sum to \(0.50\), giving a mean of \(0.50/5=0.100\). Thus, \(X_A\) ranks first more often, while \(X_B\) has the higher mean individual score. The group score is larger than either individual mean in these resamples.
These results suggest that the individual ranking is not settled by a single summary and that the pair together carries information that either single-predictor permutation can leave partly available through its correlated substitute. The group score does not equal the sum of individual scores by definition, and it should not be interpreted as a causal effect. Importance values are conditional on this model, loss, validation procedure, and permutation method.
Common mistake avoided. Reporting only that \(X_A\) ranks first more often and calling it unequivocally more important would ignore both \(X_B\)’s higher mean score and the larger group score. Report the definition, the resampling pattern, and the group comparison together.
A Practical Interpretation Sequence
A useful report connects selection, importance, and prediction performance rather than treating them as interchangeable evidence. The following sequence keeps the interpretation tied to the analysis.
State the resampling design, coefficient tolerance, preprocessing and penalty-tuning rule, importance method, and performance metric.
Identify which predictors are strongly related, and note that overlap can make individual attribution harder to separate.
Report selection frequencies and how individual and, if appropriate, group importance varies across resamples. A ranking from one fit is not a stability analysis.
Show whether validation performance changes alongside the attribution. Similar performance does not prove identical models, but it can show that different selections had similar predictive results in the evaluated samples.
Explain that unstable attribution among correlated predictors does not identify which predictor is causal or uniquely necessary. Avoid choosing an operational variable solely because it won one resample.
Resampling estimates variability under the particular data-generating and resampling conditions represented in the analysis. Repeated folds or bootstrap samples may overlap, so selection frequencies should generally be presented as descriptive stability summaries, not as independent-trial probabilities or confidence statements without further justification. If model selection or tuning itself changes across fits, say so; that is part of the procedure whose stability is being evaluated.
Check Your Understanding
For each problem, distinguish what the resampling result says about attribution from what it says about prediction.
- With \(\tau=0.02\), a predictor has coefficients \(0.31, 0.00, 0.019, 0.08, 0.00\) across five fits. Calculate its selection frequency. (Answer: \(2/5=0.4000\).)
- Two correlated predictors have selection frequencies 0.30 and 0.80 across the same resamples, while their models have similar mean validation RMSE. What is the most defensible interpretation? (Answer: Selection is more stable for the second predictor under this procedure, but the similar RMSE does not establish that it is uniquely important or causal.)
- Across four resamples, individual permutation-importance increases are 0.03, 0.06, 0.02, 0.05 for \(X_A\) and 0.07, 0.02, 0.04, 0.03 for \(X_B\). Which ranks first how often, and what are their mean scores? (Answer: \(X_A\) ranks first twice and \(X_B\) ranks first twice; their means are 0.040 and 0.040, respectively.)
- In six resamples, two correlated predictors are both selected in two fits, only \(X_A\) in two fits, and only \(X_B\) in two fits. What are their individual selection frequencies, and how many fits select at least one? (Answer: Each frequency is \(4/6=0.6667\); at least one is selected in \(6/6=1.0000\) fits.)
- A correlated pair has modest individual permutation-importance scores but a much larger score when permuted together. What does this pattern suggest, and what does it not establish? (Answer: The pair may carry shared predictive information that an individual permutation leaves partly available through the other predictor; it does not establish a causal effect or prove that either variable is uniquely necessary.)