Why Collinearity Makes Lasso Coefficients Hard to Interpret
In genomic data, two measured features can carry almost the same information: for example, two genes may have nearly identical expression profiles across the samples. A lasso fit may then predict the response well while assigning markedly different coefficients to those features. With perfectly redundant predictors, the coefficient vector can even be nonunique: several vectors attain exactly the same objective value. With near-redundancy, the solution may be unique but highly sensitive to small changes in the data.
These are distinct situations. Nonuniqueness means that more than one coefficient vector is an exact minimizer for the same data and penalty. Instability means that modest changes in the data can lead to substantially different estimated coefficients. Neither automatically implies that fitted values are similarly unreliable. The distinction matters when interpreting selected genes, comparing coefficient lists, or judging a model by its predictions.
Objective and Conditions
Assume the response and predictor columns have been centered, and predictors have been scaled as appropriate for the penalty. We use the lasso objective
where \(y\) is the response vector, \(X\) is the design matrix, \(\beta\) is the coefficient vector, and \(\|\beta\|_1=\sum_j|\beta_j|\). The fitted response is \(X\beta\), and the residual is \(y-X\beta\). If the model includes an intercept, it is unpenalized; the same arguments apply by including that intercept in the fitted values. We focus on \(\lambda>0\), the usual penalized case.
The lasso KKT condition says that a minimizer satisfies
Thus an active coefficient has residual correlation of magnitude \(\lambda\), with sign matching the coefficient; an inactive coefficient has residual correlation no larger than \(\lambda\) in magnitude. These conditions characterize an optimum for this convex objective. They do not, by themselves, guarantee that the coefficient vector is unique.
Worked Example: Two Identical Predictors Give Many Optima
Consider a hypothetical centered data set with \(n=2\), predictors \(x_1=x_2=x=(-1,1)^\mathsf{T}\), and response \(y=2x\). Since \(x^\mathsf{T}x/n=1\), the objective at coefficients \((b_1,b_2)\) depends on their sum \(t=b_1+b_2\):
Set \(\lambda=0.5\). For any fixed \(t\geq 0\), the smallest possible value of \(|b_1|+|b_2|\) is \(t\), attained whenever \(b_1,b_2\geq 0\) and \(b_1+b_2=t\). The objective to minimize over \(t\geq0\) is therefore \(\frac12(2-t)^2+0.5t\). Its derivative is \(t-2+0.5\), which is zero at \(t=1.5\); the second derivative is \(1\), so this is the minimum.
For example, \((b_1,b_2)=(1.5,0)\) and \((0.75,0.75)\) both have \(t=1.5\). Their predictions are both \(1.5x\), and their objective values are both \[ \frac12(2-1.5)^2+0.5(1.5)=0.125+0.75=0.875. \] In fact, every nonnegative split of 1.5 between the two coefficients is optimal. The data determine the combined contribution, but not how that contribution should be divided between identical predictors.
Why the Fitted Values Are Still Unique
For the squared-error lasso, different minimizers can have different coefficients, but their fitted values must agree. The reason is that squared error is strictly convex as a function of the fitted response \(X\beta\), even when it is not strictly convex as a function of \(\beta\). The lasso penalty is convex, so it cannot cancel the strict improvement in squared error that would result from averaging two different fitted responses.
To justify the result, suppose instead that the two fitted responses differ. Write \(f_1=X\hat{\beta}^{(1)}\) and \(f_2=X\hat{\beta}^{(2)}\), with \(f_1\ne f_2\). Their midpoint corresponds to the coefficient vector \(\bar{\beta}=(\hat{\beta}^{(1)}+\hat{\beta}^{(2)})/2\), whose fitted response is \((f_1+f_2)/2\). The squared-error identity gives
The final term is strictly positive because \(f_1\ne f_2\). Also, by convexity of the \(\ell_1\) norm, \(\|\bar{\beta}\|_1\leq(\|\hat{\beta}^{(1)}\|_1+\|\hat{\beta}^{(2)}\|_1)/2\). Consequently, \(Q(\bar{\beta})\) is strictly less than the average of the two minimum objective values. That contradicts the assumption that both are minimizers. Therefore their fitted responses must be equal.
This result does not say that coefficients are always unique when fitted values are unique. If \(X\) has a nontrivial null space, then distinct coefficient vectors can produce the same fitted response. The penalty determines which such vectors minimize the objective, and in cases such as duplicated columns it can leave multiple optimal representations.
Worked Example: Near-Collinearity Amplifies a Small Perturbation
Consider a hypothetical design with two centered, unit-scaled predictor columns whose Gram matrix is \(G=X^\mathsf{T}X/n=\begin{pmatrix}1&0.99\\0.99&1\end{pmatrix}\). The predictors are not identical, so \(G\) is invertible, but their correlation of 0.99 makes the direction \((1,-1)\) weakly represented: its eigenvalue is \(1-0.99=0.01\). Let \(\lambda=0.1\), and define \(z=X^\mathsf{T}y/n\), the vector of predictor-response inner products.
First take \(z=(2.09,2.09)^\mathsf{T}\). The positive-coefficient KKT equations are \(G\hat{\beta}=z-\lambda(1,1)^\mathsf{T}\). Since \(G(1,1)^\mathsf{T}=(1.99,1.99)^\mathsf{T}\), the solution is \(\hat{\beta}=(1,1)^\mathsf{T}\), with both coefficients positive as assumed.
Now perturb the response so that \(z\) changes by only \((0.004,-0.004)^\mathsf{T}\). The new score vector is \((2.094,2.086)^\mathsf{T}\). The corresponding coefficient change solves \(G\Delta\beta=(0.004,-0.004)^\mathsf{T}\). Since \(G(1,-1)^\mathsf{T}=0.01(1,-1)^\mathsf{T}\), it follows that \[ \Delta\beta=(0.4,-0.4)^\mathsf{T},\qquad \hat{\beta}_{\mathrm{new}}=(1.4,0.6)^\mathsf{T}. \] Both coefficients remain positive, so the active-set equations used in the calculation are consistent.
The coefficients changed by 0.4 in opposite directions even though the score perturbation was only 0.004 in each coordinate. Yet the fitted-value change is \(X\Delta\beta=0.4(x_1-x_2)\). Its squared size per observation is \[ \frac{\|X\Delta\beta\|_2^2}{n} =\Delta\beta^\mathsf{T}G\Delta\beta =0.16\{2(1-0.99)\}=0.0032, \] so its root-mean-square size is \(\sqrt{0.0032}\approx0.0566\). The coefficients moved substantially relative to the small perturbation, while the change in fitted values remained comparatively small.
Exact Redundancy and Practical Interpretation
With exact duplicate predictors, the objective depends on their combined contribution, while the \(\ell_1\) penalty often permits that contribution to be allocated in many ways. A solver may return one allocation, such as assigning the whole coefficient to the first feature, but that output is not evidence that the first feature has a uniquely supported role. Different algorithms, small changes in data, or numerical details may produce a different allocation with the same predictions and objective.
Near-collinearity is different: the optimum can be unique, but the problem is poorly conditioned in directions where predictor columns nearly cancel. Changes in the response or in a subset of samples can then produce large opposing coefficient changes. The fitted values may move much less because the corresponding change in \(X\beta\) is small. This is why a stable prediction does not, by itself, establish stable feature selection or stable attribution to individual genes.
A sufficient condition for a unique coefficient vector is that \(X\) have full column rank: then squared error is strictly convex in \(\beta\), and adding a convex penalty preserves strict convexity. This condition is not necessary, and failure of full rank does not imply that every particular lasso fit is nonunique. In high-dimensional settings with more predictors than samples, full column rank is impossible, but the lasso can still have a unique solution for particular data. The key question is whether the objective and data identify one coefficient vector, not just whether the design is rank deficient.
Diagnostics, Alternatives, and Limitations
Useful checks focus separately on coefficients and predictions. Inspect pairs or groups of highly correlated predictors; compare coefficient vectors across modest data perturbations or resampled data; and check whether selected features exchange places while predictions remain similar. For a given fit, verify that the KKT residual correlations satisfy the active and inactive conditions. If two reported solutions have similar objective values and predictions but sharply different coefficients, that is evidence of weak coefficient identification rather than necessarily a failure of optimization.
Ridge regression is a useful contrast: adding a positive squared-\(\ell_2\) penalty makes the objective strictly convex in the coefficient vector, even when \(X\) is rank deficient, and therefore gives a unique coefficient solution. Its coefficients generally do not become exactly zero. Lasso instead encourages sparsity, but collinear features can compete to represent the same signal. Neither method removes the underlying lack of information in redundant measurements; they impose different rules for choosing a representation.
A common mistake is to interpret a lasso coefficient of zero as proof that the corresponding gene is unrelated to the response, or to interpret a selected member of a correlated pair as uniquely important. The correction is that the fitted model may identify a shared predictive direction without identifying which correlated feature deserves the individual coefficient. Report coefficient stability and predictive performance as separate properties, and avoid causal claims based on penalized coefficients alone.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- If two columns of \(X\) are identical, what feature of the lasso objective can allow their combined coefficient contribution to be split in multiple ways?
- Can two exact lasso minimizers have different fitted responses? Give the central reason for your answer.
- In the near-collinearity example, why did a small score perturbation cause opposite-signed coefficient changes?
- Does a rank-deficient design guarantee a nonunique lasso coefficient vector for every response and penalty? Explain.
Solutions. (1) Identical columns make the prediction depend on the sum of their coefficients. For nonnegative coefficients with a fixed sum, their combined \(\ell_1\) penalty is the same, so multiple allocations can minimize the objective. (2) No. If two minimizers had different fitted responses, their midpoint would have strictly smaller squared error, while convexity of the penalty would prevent an offsetting increase; this contradicts minimality. (3) The predictor difference direction had Gram-matrix eigenvalue \(0.01\), so the inverse problem magnified a perturbation in that direction by a factor of \(1/0.01=100\). (4) No. Rank deficiency permits distinct coefficient vectors to have the same fitted response, but the penalty and data may still select a unique minimizer for a particular fit.