Lasso Choices Among Correlated Predictors

Why Correlated Predictors Create a Selection Choice

Suppose a model contains several measurements that carry much of the same information. In a hypothetical insurance analysis, these might be two related measures of vehicle use. If both help predict a response, a lasso fit may retain one and set the other’s coefficient to zero. Which one is retained can depend on small differences in their relationship with the response, even when either predictor gives nearly the same fitted values.

This behavior follows from the lasso penalty, not from a rule that the selected predictor is the uniquely important one. The penalty charges for the sum of the absolute coefficient values. When predictors are strongly correlated, multiple coefficient patterns can produce similar fitted values; the lasso may prefer a sparse pattern among them. The selected variable can therefore be a representative of shared predictive information rather than a uniquely superior feature.

Definition: Predictors are correlated when their observed values tend to vary together. In this tutorial, a predictor is selected when its fitted lasso coefficient is nonzero, and it is suppressed when its coefficient is zero. A correlated set may contain predictors that carry overlapping information about the response.

To make the mechanism precise, consider centered response values \(y\) and centered predictors \(x_1,\ldots,x_p\), with each predictor scaled so that \(x_j^\mathsf{T}x_j/n=1\). The intercept is not penalized; centering allows us to focus on the slopes. For a fixed tuning parameter \(\lambda \geq 0\), use the objective

\frac{1}{2n}\left\|y-X\beta\right\|_2^2+\lambda\sum_{j=1}^{p}|\beta_j|.

Here \(X\) is the matrix of predictor columns and \(\beta\) is the coefficient vector. This scaling fixes the meaning of \(\lambda\) for the calculations below. In applied work, the objective’s scaling and the predictor scaling must be checked before comparing numerical \(\lambda\) values across fits.

The Residual-Correlation Condition

The lasso’s Karush–Kuhn–Tucker (KKT) conditions connect selection to the correlations between predictors and the residuals. If \(\hat{\beta}\) is a lasso solution, define its residual vector as \(r=y-X\hat{\beta}\). For an active predictor, the residual correlation is \(\lambda\) times the sign of its coefficient. For an inactive predictor, the absolute residual correlation cannot exceed \(\lambda\):

\frac{x_j^\mathsf{T}r}{n}= \lambda\,\operatorname{sign}(\hat{\beta}_j) \quad\text{if }\hat{\beta}_j\ne 0, \qquad \left|\frac{x_j^\mathsf{T}r}{n}\right|\leq\lambda \quad\text{if }\hat{\beta}_j=0.

These conditions are necessary for an optimum, and they are sufficient for this convex lasso objective. In words, an omitted predictor must not be correlated with the current residual strongly enough to justify paying the penalty to add it.

Key result: For a proposed fit with only predictor \(x_1\) active and a positive coefficient, the one-predictor fit is valid if the active predictor satisfies its KKT equality and every inactive predictor satisfies the absolute residual-correlation bound. Strong correlation between \(x_1\) and \(x_2\) can make \(x_2\)’s residual correlation small after \(x_1\) has been fitted.

A short derivation explains the threshold. With only one standardized predictor, the objective as a function of its coefficient \(b\) is

\frac{1}{2n}\|y-x_1b\|_2^2+\lambda|b| =\text{constant}+\frac{1}{2}b^2-c_1b+\lambda|b|, \qquad c_1=\frac{x_1^\mathsf{T}y}{n}.

For a positive coefficient, differentiating the smooth part and the penalty gives \(b-c_1+\lambda=0\), so \(b=c_1-\lambda\), provided this value is positive. The residual is \(r=y-x_1b\). If another standardized predictor \(x_2\) has correlation \(\rho=x_1^\mathsf{T}x_2/n\) with \(x_1\), then

\frac{x_2^\mathsf{T}r}{n} =\frac{x_2^\mathsf{T}y}{n} -\frac{x_2^\mathsf{T}x_1}{n}b =c_2-\rho b.

Thus \(x_2\) can remain inactive when \(|c_2-\rho b|\leq\lambda\). If \(\rho\) is close to one, fitting \(x_1\) also removes much of the response association that \(x_2\) shared with it. This is the central reason lasso may select one member of a correlated set and suppress another.

Worked Example: Checking Whether One Correlated Predictor Is Enough

Suppose a hypothetical study has two centered, standardized predictors with correlation \(\rho=0.90\). Their response correlations are \(c_1=0.80\) and \(c_2=0.73\), where \(c_j=x_j^\mathsf{T}y/n\). Fit the lasso with \(\lambda=0.20\). Check whether the solution with only \(x_1\) active is valid.

Method. Compute the positive one-predictor lasso coefficient, then check the KKT bound for the omitted predictor.

Step 1: Fit \(x_1\). Since \(c_1>\lambda\), the positive one-predictor coefficient is

\hat{\beta}_1=c_1-\lambda=0.80-0.20=0.60.

The active-predictor condition holds: its residual correlation is \(c_1-\hat{\beta}_1=0.80-0.60=0.20=\lambda\).

Step 2: Check \(x_2\). After fitting \(x_1\), the residual correlation for \(x_2\) is

c_2-\rho\hat{\beta}_1 =0.73-(0.90)(0.60) =0.73-0.54 =0.19.

Since \(|0.19|\leq0.20\), the inactive-predictor condition holds. The one-predictor fit is therefore a lasso solution: \(x_1\) is selected and \(x_2\) is suppressed.

Check. The active residual correlation equals \(\lambda\), and the omitted predictor’s absolute residual correlation is below \(\lambda\). The KKT conditions are satisfied. The common mistake is to compare \(c_2=0.73\) directly with \(\lambda\); the correct check uses \(x_2\)’s correlation with the residual after fitting \(x_1\).

When Both Correlated Predictors Are Selected

Correlation does not force the lasso to select only one predictor. If the response has enough association with both predictors after accounting for the other, both coefficients can be nonzero. For two standardized predictors with correlation \(\rho\), suppose both coefficients are positive. Their KKT equalities are

c_1=\hat{\beta}_1+\rho\hat{\beta}_2+\lambda, \qquad c_2=\rho\hat{\beta}_1+\hat{\beta}_2+\lambda.

The terms involving \(\rho\) account for the shared variation. Solving these equations, rather than shrinking each predictor separately as if the other were absent, gives the joint active coefficients. The solution is only valid if both coefficients are positive and any other inactive predictors satisfy their KKT bounds.

Worked Example: Solving for Two Active Correlated Predictors

Suppose two centered, standardized predictors have \(\rho=0.90\), and each has response correlation \(c_1=c_2=0.80\). Use \(\lambda=0.20\). Find the lasso coefficients when both predictors are active and verify the conditions.

Method. Solve the two positive-coefficient KKT equations simultaneously.

By symmetry, the equations give equal coefficients, say \(\hat{\beta}_1=\hat{\beta}_2=b\). Substituting into either equation yields

0.80=b+0.90b+0.20, \qquad 1.90b=0.60, \qquad b=\frac{0.60}{1.90}=\frac{6}{19}\approx0.3158.

Thus \(\hat{\beta}_1=\hat{\beta}_2=6/19\). Check the first active-predictor equality:

\hat{\beta}_1+\rho\hat{\beta}_2+\lambda =\frac{6}{19}+\frac{9}{10}\frac{6}{19}+\frac{1}{5} =\frac{6}{19}+\frac{27}{95}+\frac{19}{95} =\frac{30+27+19}{95} =\frac{76}{95} =0.80.

The second equality is identical by symmetry. Both coefficients are positive, so the assumed signs are consistent. There are no other predictors in this example, so there are no additional inactive-predictor checks.

Check. The solution satisfies both KKT equalities and has positive coefficients. Treating each predictor as if it were fitted alone would give \(0.80-0.20=0.60\) for each, which ignores their shared variation and is not the joint lasso solution.

Small Changes Can Switch the Selected Representative

When two predictors are highly correlated, small changes to the data can change which one satisfies the selection conditions. This can happen even if the resulting fitted values change little. The distinction matters: a changing coefficient pattern is evidence that the allocation of shared information is sensitive, not necessarily that the model’s predictions are equally sensitive.

For standardized predictors, the squared size of the difference between two fitted vectors can be computed directly. If one fit is \(b x_1\) and another is \(b x_2\), then

\frac{1}{n}\|b x_1-b x_2\|_2^2 =b^2\left(\frac{x_1^\mathsf{T}x_1}{n} +\frac{x_2^\mathsf{T}x_2}{n} -2\frac{x_1^\mathsf{T}x_2}{n}\right) =2b^2(1-\rho).

This follows by expanding the squared norm and using \(x_1^\mathsf{T}x_1/n=x_2^\mathsf{T}x_2/n=1\). The root mean squared difference between the two fitted vectors is therefore \(|b|\sqrt{2(1-\rho)}\). It becomes small as \(\rho\) approaches one, provided the coefficient size remains bounded.

Worked Example: A Small Change Switches the Selected Predictor

Two standardized predictors have correlation \(\rho=0.99\), and \(\lambda=0.20\). In one data version, their response correlations are \(c_1=0.800\) and \(c_2=0.793\). In a slightly perturbed version, these correlations are \(c_1=0.793\) and \(c_2=0.800\). Determine the selected predictor in each version and compare the fitted vectors.

Method. Apply the one-active-predictor KKT check in each version, then calculate the root mean squared difference between the fitted vectors.

First version. Fitting \(x_1\) alone gives

\hat{\beta}_1=0.800-0.200=0.600.

The omitted predictor’s residual correlation is \(0.793-(0.99)(0.600)=0.793-0.594=0.199\). Its absolute value is no greater than \(0.200\), so \(x_1\) can be selected while \(x_2\) is suppressed.

Perturbed version. By the same calculation, fitting \(x_2\) gives \(\hat{\beta}_2=0.800-0.200=0.600\). The omitted predictor’s residual correlation is \(0.793-(0.99)(0.600)=0.199\), which again satisfies the KKT bound. Thus the selected representative switches from \(x_1\) to \(x_2\).

The difference between the fitted vectors is \(0.600x_1-0.600x_2\). Its root mean squared size is

0.600\sqrt{2(1-0.99)} =0.600\sqrt{0.02} \approx0.0849.

The calculation is in the standardized response units used for the model. The coefficients have switched predictors, but the fitted vectors are close because the predictors are highly correlated.

Check. In each version, the active residual correlation equals \(0.200\), and the inactive one is \(0.199\). The root mean squared difference is approximately \(0.0849\). The conclusion is conditional: highly correlated, similarly scaled predictors with similar predictive information can yield similar fits when the representative changes; high correlation alone does not guarantee similar predictions in every setting.

Interpretation and Exam Traps

A selected predictor in a correlated group should be interpreted cautiously. The model has identified a useful predictive direction under the chosen data, scaling, and \(\lambda\); it has not established that the selected variable is the sole source of information. A suppressed predictor may still be informative on its own, but add too little conditional information to pass the lasso’s residual-correlation threshold once a correlated predictor is included.

This distinction also limits how coefficients should be communicated. If small data changes can switch which member of a correlated group is selected, avoid presenting the selected member as a uniquely important driver or a causal explanation. The lasso fit is a predictive model, and selection is conditional on the model specification and observed data.

Checking raw correlations instead of residual correlations. An inactive predictor’s KKT check is against the current residual, not the original response.
Assuming correlation means only one variable can be active. Both predictors can be selected when their joint KKT equations yield valid nonzero coefficients.
Calling the selected predictor uniquely important. A correlated alternative may carry nearly the same predictive information.
Concluding that coefficient changes imply equally large prediction changes. Compare fitted values as well as coefficients; for standardized predictors, their difference depends on \(1-\rho\).
Ignoring scaling or the objective convention. The numerical value of \(\lambda\) depends on the loss scaling and predictor scaling. Apply KKT calculations using a consistent convention.

Key takeaway: Lasso can select one predictor from a correlated set because fitting that predictor may reduce the other predictors’ residual correlations below \(\lambda\). Small data changes can switch the chosen representative, while fitted values remain close when the predictors are highly correlated. This tutorial is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Check Your Understanding

For each question, use the centered, standardized-predictor lasso objective defined above. Treat stated correlations as exact.

Two predictors have \(\rho=0.80\), \(c_1=0.65\), and \(c_2=0.59\), with \(\lambda=0.15\). If only \(x_1\) is proposed as active, calculate its coefficient and the residual correlation of \(x_2\). Does that one-active-predictor proposal satisfy the KKT bound? (Answer: \(\hat{\beta}_1=0.50\), the residual correlation is \(0.59-(0.80)(0.50)=0.19\), so the proposal fails because \(0.19>0.15\).)
Two predictors have \(\rho=0.95\), \(c_1=0.70\), and \(c_2=0.65\), with \(\lambda=0.10\). Check the proposed fit with only \(x_1\) active. (Answer: \(\hat{\beta}_1=0.60\); the inactive residual correlation is \(0.65-(0.95)(0.60)=0.08\), so the proposal satisfies the KKT bound.)
Two predictors have \(\rho=0.80\), \(c_1=c_2=0.60\), and \(\lambda=0.20\). Assuming both coefficients are positive, solve for both coefficients and check their signs. (Answer: \(\hat{\beta}_1=\hat{\beta}_2=(0.60-0.20)/(1+0.80)=2/9\approx0.2222\); both are positive and satisfy the active KKT equations.)
Two standardized predictors have \(\rho=0.98\), and the lasso coefficient is \(0.40\) on whichever one is selected. Find the root mean squared difference between the fitted vectors if the selected representative switches from one predictor to the other. (Answer: \(0.40\sqrt{2(1-0.98)}=0.08\) response units.)
With \(\rho=0.98\) and \(\lambda=0.10\), one data version has \(c_1=0.50,c_2=0.484\), and a perturbed version swaps these response correlations. Check the selected representative in each version and state the coefficient. (Answer: The first version selects \(x_1\) with coefficient \(0.40\), since the inactive residual correlation is \(0.484-(0.98)(0.40)=0.092\leq0.10\). The swapped version selects \(x_2\) with coefficient \(0.40\) by the same check.)

What You'll Learn