Why Correlated Predictors Require Careful Interpretation
Lasso can set some regression coefficients exactly to zero, which makes it useful for selecting predictors. But when two predictors carry much the same information, the selection may not have a simple interpretation: lasso can choose one and exclude the other, choose both, or—in the case of exact redundancy—have more than one optimal coefficient vector. These outcomes depend on the relationship between the predictors, their association with the response, and the penalty strength.
This distinction matters in risk modeling. For example, a hypothetical insurer might use two highly related measures of policyholder activity. If one measure is selected and the other is not, that alone does not establish that the selected measure is uniquely important. Small changes in the data can change which variable is selected even when the fitted predictions change little. This tutorial is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
Model, Scaling, and Lasso Conditions
Consider a linear regression with response vector \(y\), predictor matrix \(X\), an unpenalized intercept, and coefficient vector \(\beta\). Assume the response and predictor columns have been centered, so the intercept can be handled separately. For comparisons of penalty size across predictors, take the columns to be standardized so that \(x_j^\top x_j/n=1\). The lasso estimate minimizes squared error plus an absolute-value penalty:
The optimization is defined whether or not the errors are normally distributed. A linear model such as \(y=X\beta^*+\varepsilon\), with \(E[\varepsilon\mid X]=0\), can provide a statistical interpretation, but it is not needed to solve the stated optimization problem. Standardizing is important: if predictors use very different units, the same coefficient penalty can treat them differently simply because of their scales.
For two standardized columns, their sample correlation is \(x_1^\top x_2/n\). Values near 1 or -1 indicate that the columns are nearly redundant; a value of 1 or -1 means one column is an exact multiple of the other. The lasso objective depends on the fitted values \(X\beta\), but its penalty depends on the individual coefficients. That difference is central to understanding correlated predictors.
How Lasso Chooses Within a Correlated Pair
For centered, standardized predictors, define \(G=X^\top X/n\) and \(z=X^\top y/n\). Apart from a term that does not depend on \(\beta\), the lasso objective can be written as
For two predictors, \(G_{11}=G_{22}=1\) and \(G_{12}=G_{21}=\rho\), where \(\rho\) is their correlation. The KKT conditions give a convenient check. If \(\beta_j\ne 0\), the derivative of the squared-error part in coordinate \(j\), plus \(\lambda\operatorname{sign}(\beta_j)\), must be zero. If \(\beta_j=0\), the absolute value of the squared-error gradient in that coordinate must be at most \(\lambda\). Thus, a proposed solution can be checked both for its active coefficients and for whether any excluded coefficient should enter.
Worked Example: Exact Duplicates and a Continuum of Lasso Solutions
Suppose two standardized predictor columns are identical. After reducing the objective to the total coefficient \(t=\beta_1+\beta_2\), assume the lasso objective is
Find the minimizing total and determine whether the coefficient allocation is unique. The method is to minimize the penalty over all allocations with a fixed total, then minimize over every real value of that total.
Step 1: Minimize the penalty for a fixed total. For any real \(t=\beta_1+\beta_2\), the triangle inequality gives
This bound is attainable: for \(t\geq0\), take nonnegative coefficients summing to \(t\); for \(t\leq0\), take nonpositive coefficients summing to \(t\). Therefore the objective minimized over allocations with total \(t\) is
Step 2: Check both signs of \(t\). For \(t\geq0\), \(q(t)=\frac12((5-t)^2+9)+t\), so \(q'(t)=t-4\). Its minimum on this half-line occurs at \(t=4\), with \(q(4)=\frac12(1+9)+4=9\). For \(t<0\), \(q(t)=\frac12((5-t)^2+9)-t\), so \(q'(t)=t-6<0\). It decreases as \(t\) increases toward zero, and its limiting value at zero is \(q(0)=\frac12(25+9)=17\). Thus no negative total is a global minimizer, and the minimizing total is \(t=4\).
Step 3: Recover the allocations. Every nonnegative pair with \(\beta_1+\beta_2=4\) attains the minimum. For example, \((4,0)\), \((2,2)\), and \((0,4)\) all have the same fitted values and penalty. The coefficient vector is therefore not unique, even though the minimizing total and fitted contribution are fixed.
Check and common mistake. At \((2,2)\), the residual term is \(\frac12(1+9)=5\) and the penalty is \(4\), totaling \(9\). A common error is to use the penalty \(t\) for every real \(t\); when \(t<0\), the minimum penalty is \(|t|=-t\). Checking the negative half-line as above is necessary to establish the global result.
Near-Redundancy Can Make Selection Unstable
When predictors are highly correlated but not identical, the coefficient solution may be unique, yet which member of the pair is selected can be sensitive to small data changes. The next example checks this with the KKT conditions. With \(\rho=0.98\), the Gram matrix is positive definite because its eigenvalues are \(1+0.98\) and \(1-0.98\), both positive. The squared-error part is therefore strictly convex, so a KKT solution is the unique minimizer for each set of response associations.
Worked Example: A Small Change Switches the Selected Predictor
Two standardized predictors have correlation \(\rho=0.98\), and \(\lambda=1\). First let \(z=(3.0,2.9)\), where \(z_j=x_j^\top y/n\). Find the lasso coefficients. Then consider the nearby values \(z=(2.9,3.0)\).
First response association. Try \(\beta=(2,0)\). For the active first coefficient, the KKT equation is \(G_{11}\beta_1-z_1+\lambda=0\), since \(\beta_1>0\). Substitution gives \(2-3.0+1=0\). For the inactive second coefficient, the residual score is \(z_2-G_{21}\beta_1=2.9-(0.98)(2)=0.94\). Since \(|0.94|\leq1\), the inactive-coordinate condition holds. Thus \((2,0)\) is the minimizer. The reduced objective is \(\frac12(2^2)-3.0(2)+1(2)=2-6+2=-2\).
Swapped response association. Now let \(z=(2.9,3.0)\). By the same calculation, \(\beta=(0,2)\) satisfies the active equation \(2-3.0+1=0\). Its inactive score is \(2.9-(0.98)(2)=0.94\), again within the required bound. The reduced objective is \(\frac12(2^2)-3.0(2)+1(2)=-2\). The selected predictor switches, although each entry of \(z\) has changed by only \(0.1\).
Check and common mistake. The inactive coefficient is valid only after checking its score against \(\lambda\); simply setting it to zero because the predictors are correlated is not a solution. Here the KKT checks verify both answers. The example shows sensitivity of the selected identity, not a large change in the achieved objective.
Correlation Does Not Always Mean Lasso Picks Only One
A highly correlated pair need not produce a one-variable solution. If both variables have similar associations with the response, lasso may assign nonzero coefficients to both. The KKT conditions determine whether that allocation is optimal; a general rule that lasso always keeps exactly one member of a correlated group is incorrect.
Worked Example: Both Correlated Predictors Remain Active
Let two standardized predictors have correlation \(\rho=0.5\), let \(z=(2,2)\), and let \(\lambda=1\). Find the lasso coefficients by testing a symmetric solution, and verify that a one-variable solution does not satisfy the KKT conditions.
Step 1: Solve the active equations. By symmetry, try \(\beta_1=\beta_2=b>0\). The first active-coordinate equation is \(b+0.5b-2+1=0\), so \(1.5b=1\) and \(b=2/3\). The second equation is identical. Hence the candidate is \((2/3,2/3)\).
Step 2: Verify optimality. Both coefficients are positive, and both active KKT equations hold:
The same equality holds for the second coordinate. The Gram matrix has eigenvalues \(1.5\) and \(0.5\), so the objective is strictly convex; this KKT solution is the unique minimizer.
Step 3: Compare with excluding one predictor. The candidate \((1,0)\), obtained by fitting only the first predictor, has inactive score \(z_2-G_{21}(1)=2-0.5=1.5\), which exceeds \(\lambda=1\). It fails the inactive-coordinate condition. The reduced objective at the two-variable solution is
At \((1,0)\), it is \(\frac12-2+1=-\frac12\), which is larger. The common mistake is to assume a correlated pair must be reduced to one variable; the KKT check shows that both coefficients are supported here.
What to Conclude From a Selected Set
The examples illustrate three distinct situations. With near-duplicate but distinct predictors, the minimizer can be unique while the selected identity changes under small perturbations. With less extreme correlation and similar response associations, both predictors can remain active. With exactly redundant predictors, multiple coefficient allocations can have the same fitted values and penalty. These are different mechanisms and should not be conflated.
For interpretation, consider the fitted contribution of a group of correlated predictors as well as the individual coefficients. A selected variable is selected under a particular data set, scaling convention, and penalty value; that fact alone does not prove that it is the only useful variable in its group. Conversely, correlation does not guarantee that lasso will exclude all but one member. If the goal is explanation, report the correlation and selection sensitivity rather than treating a single selected coefficient as a definitive ranking.
Check Your Understanding
For each problem, use standardized centered predictors and the lasso objective with penalty \(\lambda\). Where response associations \(z_j\) and correlations are given, use the reduced objective or KKT conditions as appropriate.
- Two predictor columns are identical. The reduced objective for their total \(t\) is \(\frac12(3-t)^2+\frac12(|\beta_1|+|\beta_2|)\), where \(t=\beta_1+\beta_2\). Find the minimizing total and describe the minimizing allocations. (Answer: \(t=2.5\); every nonnegative pair summing to \(2.5\) is optimal.)
- Two predictors have correlation \(0.9\), \(z=(2.4,2.2)\), and \(\lambda=0.5\). Check whether \((1.9,0)\) satisfies the KKT conditions. (Answer: Yes; active equation holds and the inactive score is \(2.2-0.9(1.9)=0.49\leq0.5\).)
- Two standardized predictors have correlation \(0.4\), \(z=(1.8,1.8)\), and \(\lambda=0.6\). By symmetry, find the solution with both coefficients positive. (Answer: \((6/7,6/7)\).)
- Two predictor columns are identical, and their common response association is \(-2\), with \(\lambda=0.5\). What is the minimizing total coefficient, and what sign must nonzero allocations have to attain the minimum penalty for that total? (Answer: Total \(-1.5\); nonpositive allocations summing to \(-1.5\) attain the minimum penalty.)
- Two standardized predictors have correlation \(0.6\), \(z=(1.7,2.1)\), and \(\lambda=0.8\). Solve the active KKT equations and check whether both coefficients are positive. (Answer: \((0.1875,1.1875)\); both are positive.)