Lasso Selection Among Correlated Predictors

Why Correlated Predictors Create a Selection Problem

Lasso is designed to produce sparse models: some fitted coefficients are exactly zero. When predictors are strongly correlated, however, several variables can contain almost the same information about the response. Lasso may select one of them, but the data may provide little basis for treating that variable as uniquely important. Small changes in the data can change which predictor is selected or how the coefficients are divided, even when the fitted predictions barely change.

To isolate this issue, consider a regression with centered response and centered, scaled predictors. Let \(X_j\) denote predictor \(j\)'s column, with squared length \(X_j^\mathsf{T}X_j=1\), and let \(y\) be the response vector. For a fixed penalty parameter \(\lambda>0\), use the lasso objective

\frac{1}{2}\left\|y-\sum_{j=1}^{p}X_j\beta_j\right\|^2 +\lambda\sum_{j=1}^{p}|\beta_j|.

The intercept, if included, is not penalized; centering makes it unnecessary to display here. Scaling matters because the penalty acts on coefficient magnitudes, so coefficients are comparable only when predictors are on a common scale. The examples use columns of squared length one to make the arithmetic clear. They illustrate the optimization, not a claim that a particular data-generating model or sampling design must hold.

Definition: Two predictors are correlated when their observed values vary together. In these examples, each predictor column has squared length one, so their sample correlation is \(X_1^\mathsf{T}X_2\). Correlation near \(1\) means the columns point in nearly the same direction: they provide similar, though not identical, ways to fit the response.

The lasso optimality condition gives a useful check. For a nonzero coefficient, the predictor's inner product with the residual equals \(\lambda\) times the coefficient's sign. For a zero coefficient, the absolute inner product with the residual must be no greater than \(\lambda\). Here the residual is \(r=y-\sum_jX_j\beta_j\). These conditions help establish whether a proposed fit with only one selected predictor is valid.

Identical Predictors: The Allocation Is Not Identified

If two predictors are identical, their separate contributions to the fitted values cannot be distinguished. The lasso penalty does not resolve this ambiguity: among coefficients with the same sign, it can assign the shared contribution in many different ways without changing either the fit or the penalty.

Worked Example: Two Identical Predictors

Suppose a hypothetical centered data set has two identical predictor columns, \(X_1=X_2=X\), with \(X^\mathsf{T}X=1\). Suppose also that \(X^\mathsf{T}y=2\), and set \(\lambda=0.5\). The components of \(y\) perpendicular to \(X\) affect the objective only by a constant, so the part that depends on the coefficients is

\frac{1}{2}(2-\beta_1-\beta_2)^2 +0.5\left(|\beta_1|+|\beta_2|\right).

Write \(t=\beta_1+\beta_2\). For any fixed \(t\), the triangle inequality gives

|\beta_1|+|\beta_2|\geq |t|.

Equality holds whenever the two coefficients have the same sign, including when one is zero. The best total coefficient therefore minimizes \(\frac{1}{2}(2-t)^2+0.5|t|\). Since the minimizing \(t\) is positive, differentiating \(\frac{1}{2}(2-t)^2+0.5t\) gives \(t-2+0.5=0\), so \(t=1.5\).

Every nonnegative pair summing to \(1.5\) gives the same fitted values and penalty. For example, \((\beta_1,\beta_2)=(1.5,0)\), \((0,1.5)\), and \((0.75,0.75)\) all minimize the objective. The fit identifies a combined contribution of \(1.5X\), but it cannot identify how much belongs to either duplicate predictor.

With identical columns, the ambiguity is exact. With predictors that are merely very highly correlated, the ambiguity is approximate: reallocating coefficients changes the fitted values, but often only slightly. The lasso penalty can still favor a sparse allocation, which may make a single selected predictor look more definitive than the data warrant.

Near-Duplicates: Small Changes Can Switch the Selected Variable

For two scaled predictors with correlation \(r=X_1^\mathsf{T}X_2\), suppose a proposed lasso fit uses \(X_1\) alone and has a positive coefficient. Its coefficient is \(X_1^\mathsf{T}y-\lambda\), since \(X_1^\mathsf{T}X_1=1\). To check whether \(X_2\) can remain excluded, calculate its inner product with the residual and check that its absolute value is at most \(\lambda\). A small change in \(X_2^\mathsf{T}y\), or in the corresponding value for \(X_1\), can change whether this condition is met.

Worked Example: A Small Change Switches the Selected Predictor

Consider two hypothetical, scaled predictor columns with \(X_1^\mathsf{T}X_2=0.99\), and let \(\lambda=0.5\). First suppose their inner products with the centered response are \(X_1^\mathsf{T}y=2\) and \(X_2^\mathsf{T}y=1.98\). A fit using \(X_1\) alone has

\beta_1=2-0.5=1.5,\qquad \beta_2=0.

The residual is \(r=y-1.5X_1\). The inner product for the excluded predictor is

X_2^\mathsf{T}r = X_2^\mathsf{T}y-1.5X_2^\mathsf{T}X_1 =1.98-(1.5)(0.99) =0.495.

Because \(0.495<0.5=\lambda\), the condition for keeping \(\beta_2=0\) is satisfied. The active predictor also satisfies its condition: \(X_1^\mathsf{T}r=2-1.5=0.5=\lambda\). Thus selecting \(X_1\) alone is consistent with the lasso optimality conditions.

Now consider a slightly changed response association: \(X_1^\mathsf{T}y=1.98\) and \(X_2^\mathsf{T}y=2\), with the same predictor columns and penalty. The corresponding one-predictor fit uses \(X_2\):

\beta_2=2-0.5=1.5,\qquad \beta_1=0.

For the excluded predictor, \(X_1^\mathsf{T}r=1.98-(1.5)(0.99)=0.495\), again no greater than \(\lambda\). A change of only \(0.02\) in which predictor has the larger response inner product has switched the selected variable. Both predictors have nearly the same relationship with the response, and the selected name depends on a small difference.

This example does not say that every correlated pair will produce a single-variable fit. Depending on the response associations and penalty, lasso may retain both predictors. It shows instead why selection among similar predictors can be sensitive: the conditions that exclude one predictor can be close to their boundary.

Unstable Coefficients Can Accompany Similar Predictions

The coefficient pattern and the fitted values answer different questions. Coefficients describe how the fitted signal is allocated among the predictor columns. Fitted values describe the combined signal. When columns are close to one another, a sizable coefficient change in opposite directions can leave their combined contribution nearly unchanged.

Worked Example: Coefficients Shift More Than the Fit

Use two hypothetical scaled predictors with correlation \(0.99\) and set \(\lambda=0.5\). In one data set, suppose \(X_1^\mathsf{T}y=X_2^\mathsf{T}y=2\). If both coefficients are positive, the active-predictor conditions require \(X_j^\mathsf{T}r=0.5\) for each predictor. The resulting equations are

\beta_1+0.99\beta_2=1.5,\qquad 0.99\beta_1+\beta_2=1.5.

Subtracting the equations gives \(0.01(\beta_1-\beta_2)=0\), so \(\beta_1=\beta_2\). Substitution gives \(1.99\beta_1=1.5\), hence \(\beta_1=\beta_2=1.5/1.99\approx0.7538\). These positive coefficients satisfy the active conditions.

In a slightly changed data set, let \(X_1^\mathsf{T}y=2.02\) and \(X_2^\mathsf{T}y=1.98\). A fit using only \(X_1\) has \(\beta_1=2.02-0.5=1.52\). The inner product for the excluded predictor is

X_2^\mathsf{T}r =1.98-(0.99)(1.52) =0.4752,

which is less than \(0.5\). So this one-predictor fit satisfies the lasso conditions. Comparing the two fits, the coefficients have changed from about \((0.7538,0.7538)\) to \((1.52,0)\), even though the response inner products changed by only \(0.02\).

The fitted-value difference is \(1.52X_1-0.7538(X_1+X_2)\), or approximately \(0.7662X_1-0.7538X_2\). Its squared length is about \(0.0117\), so its length is about \(0.1082\). The coefficient allocation changed substantially, but the fitted vectors remain close because \(X_1\) and \(X_2\) point in nearly the same direction.

The examples distinguish three related phenomena: exact non-uniqueness with duplicate columns, a switch in which near-duplicate is selected, and a large coefficient reallocation with a comparatively small change in predictions. In repeated samples or small data perturbations, any of these patterns can make a selected-variable list or coefficient table look unstable.

How to Read a Lasso Selection Among Correlated Predictors

A selected predictor is part of a particular penalized fit; it is not automatically the uniquely relevant variable in the subject matter. When predictors are correlated, selection can depend on small differences in their observed associations with the response and on the penalty. A lasso fit can be useful for obtaining a sparse predictive representation, but the selected member of a group may be a fragile representative of shared information.

A common mistake is to conclude, “Lasso selected \(X_1\) and set \(X_2\) to zero, so \(X_1\) matters and \(X_2\) does not.” The zero says that \(X_2\) has no separate coefficient in this fitted penalized solution. It does not show that \(X_2\) contains no information, especially when \(X_2\) closely tracks \(X_1\). A more careful account notes the correlation, reports the selection as conditional on the fitted procedure and data, and avoids treating a selection as causal evidence.

Coefficient instability also does not necessarily imply equally unstable prediction. If two predictors move together, reallocating weight between them can change their individual coefficients while changing their combined fitted contribution little. Conversely, similar-looking coefficient patterns should not be assumed to give identical predictions: the columns are highly correlated, not necessarily identical. The distinction should be checked rather than presumed.

Key takeaway: Among highly correlated predictors, lasso can select one variable arbitrarily, switch selections after small data changes, or produce unstable coefficient allocations. Interpret the selected variables and coefficients in light of the predictors' shared information; do not equate a zero coefficient with proof that a predictor is irrelevant.

Check Your Understanding

Use the definitions and calculations above to answer the following.

If \(X_1=X_2\), why can several coefficient pairs produce the same fitted values?
For a predictor with coefficient zero, what condition must its inner product with the residual satisfy under the lasso optimality conditions?
In the near-duplicate example, what changed between the two response associations, and which variable was selected in each fit?
How can coefficient estimates change substantially while fitted values change only slightly?
Why is it incorrect to interpret a lasso zero among correlated predictors as proof that the predictor contains no information about the response?