Why Lasso Uniqueness Needs Care
A lasso fit can give a definite prediction for every observation even when it does not determine a single coefficient vector. This distinction matters when predictors are linearly dependent: several allocations of coefficient weight can produce the same fitted values and the same absolute-value penalty. The coefficients may then be nonunique, although the fitted response is fixed.
We use the lasso objective below, with no intercept. If a model includes an unpenalized intercept, it can be handled separately or by centering the response and predictor columns first. Assume \(\lambda>0\), let \(X\) be the \(n\times p\) design matrix, let \(y\) be the response vector, and let \(\beta\) be the coefficient vector. The scaling of the squared-error term matters: the results and examples here use one-half times the residual sum of squares.
The penalty parameter \(\lambda\) is fixed when asking whether the minimizer is unique. A zero coefficient is allowed, and the absolute-value penalty is not differentiable at zero; the optimality condition therefore uses subgradients. These conditions do not require \(X\) to have full column rank.
Fitted Values Are Unique
For a fixed response \(y\), the lasso objective is convex, but it need not be strictly convex as a function of \(\beta\): if \(X\) has a nonzero vector \(d\) with \(Xd=0\), moving along \(d\) does not change the squared-error term. The fitted values, however, are unique among all minimizers.
To see why the theorem holds, suppose two minimizers gave different fitted values, denoted \(z_1=X\hat\beta^{(1)}\) and \(z_2=X\hat\beta^{(2)}\). The squared-error term is strictly convex in the fitted vector \(z\), so its value at the midpoint \((z_1+z_2)/2\) would be strictly less than the average of its values at \(z_1\) and \(z_2\). The midpoint corresponds to the coefficient vector \((\hat\beta^{(1)}+\hat\beta^{(2)})/2\). The \(L_1\) penalty is convex, so its value there is no greater than the average of the two penalties. Together these facts would make the objective at the midpoint strictly less than its minimum, a contradiction. Thus the fitted vectors must agree.
This argument establishes uniqueness of \(X\hat\beta\), not uniqueness of \(\hat\beta\). Distinct vectors in the same fitted-value set can have the same \(L_1\) norm, leaving multiple minimizers.
The Equicorrelation Set and a Sufficient Condition
The Karush-Kuhn-Tucker (KKT) conditions provide a useful way to distinguish these cases. For a minimizing vector \(\hat\beta\), write \(\hat r=y-X\hat\beta\), and let \(x_j\) denote column \(j\) of \(X\). For a nonzero coefficient, the penalty has derivative \(\operatorname{sign}(\hat\beta_j)\); at zero, its subgradient can be any number between \(-1\) and \(1\). Consequently,
Equivalently, every column satisfies \(|x_j^\mathsf{T}\hat r|\leq\lambda\), and equality holds for any column whose coefficient is nonzero. This follows by setting the subgradient of the objective to zero: \(0\in-X^\mathsf{T}\hat r+\lambda\,\partial\|\hat\beta\|_1\). The residual is the same for every minimizer because the fitted values are unique. Thus the following set is common to all minimizing solutions.
If the columns indexed by \(E\) are linearly independent, the coefficient vector must be unique. This is a sufficient condition, not a necessary one. To justify it, suppose two minimizers have the same fitted values. Both have zero coefficients outside \(E\), so their coefficient difference is supported on \(E\). The fitted-value equality gives \(X_E(\hat\beta_E^{(1)}-\hat\beta_E^{(2)})=0\). Linear independence of the columns in \(X_E\) implies \(\hat\beta_E^{(1)}-\hat\beta_E^{(2)}=0\), so the full coefficient vectors agree.
It is not enough that the nonzero columns in one proposed solution are independent. A different minimizer may use additional columns from \(E\) to represent the same fitted vector with the same penalty. Independence of the active columns in one solution only describes that solution’s support; it does not rule out another optimal representation.
Worked Example: Identical Predictors Give Multiple Coefficient Vectors
Let \(x=(1,1)^\mathsf{T}\), let both predictor columns equal \(x\), take \(y=(3,3)^\mathsf{T}\), and set \(\lambda=1\). Find the lasso minimizers and check whether their coefficients are unique.
Method. Since the two columns are identical, the fitted vector depends only on \(t=\beta_1+\beta_2\): \(X\beta=tx\). Also, \(\|x\|_2^2=2\), so the squared-error term is \(\frac12\|(3-t)x\|_2^2=(3-t)^2\). For any coefficients, \(|\beta_1|+|\beta_2|\geq|\beta_1+\beta_2|=|t|\). Therefore, for fixed \(t\), the smallest possible objective is \((3-t)^2+|t|\), achieved by nonnegative coefficients when \(t\geq0\).
For \(t\geq0\), differentiate \((3-t)^2+t\): its derivative is \(2(t-3)+1\). Setting this equal to zero gives \(t=2.5\). The second derivative is \(2>0\), so this is the minimum on that range. For \(t<0\), the objective is \((3-t)^2-t\), whose derivative \(2(t-3)-1\) is negative throughout \(t<0\); its smallest value on that range is approached at \(t=0\), where the objective is \(9\). At \(t=2.5\), the objective is \((0.5)^2+2.5=2.75\), so \(t=2.5\) is the global minimum.
Any nonnegative pair summing to \(2.5\) attains that minimum. For example, \((2.5,0)^\mathsf{T}\) and \((1.25,1.25)^\mathsf{T}\) are distinct minimizers. Both give fitted values \((2.5,2.5)^\mathsf{T}\), residual \((0.5,0.5)^\mathsf{T}\), and penalty \(2.5\). The residual’s inner product with either column is \(1\), equal to \(\lambda\), so both columns belong to \(E\); they are linearly dependent.
Check and common mistake. The objective at either displayed solution is \(\frac12(0.5^2+0.5^2)+2.5=2.75\), as calculated. It would be incorrect to claim that the first solution proves uniqueness because its sole nonzero column is linearly independent. The second solution is another minimizer, so active-column independence in one solution is not enough.
Using the Equicorrelation Condition
The next example shows how the condition can establish uniqueness even when the full design matrix has more columns than rows. First check the proposed coefficients with the KKT conditions; then use the equicorrelation set to rule out any other coefficient vector.
Worked Example: A Unique Coefficient Vector With a Rank-Deficient Design
Let \(x_1=(1,0)^\mathsf{T}\), \(x_2=(0,1)^\mathsf{T}\), and \(x_3=(1,1)^\mathsf{T}/\sqrt{2}\). Take \(y=(3,0)^\mathsf{T}\) and \(\lambda=1\). Determine whether \(\hat\beta=(2,0,0)^\mathsf{T}\) is the unique lasso minimizer.
Method. The proposed fit is \(2x_1=(2,0)^\mathsf{T}\), so the residual is \(\hat r=y-X\hat\beta=(1,0)^\mathsf{T}\). The three inner products with this residual are \(x_1^\mathsf{T}\hat r=1\), \(x_2^\mathsf{T}\hat r=0\), and \(x_3^\mathsf{T}\hat r=1/\sqrt{2}\). The first equals \(\lambda\) with the positive sign of \(\hat\beta_1\); the other two have absolute value less than \(\lambda\), as required for zero coefficients. The KKT conditions therefore verify that the proposed vector is a minimizer.
The equicorrelation set is \(E=\{1\}\), whose single column is linearly independent. The sufficient condition proves that the coefficient vector is unique. The full design matrix is rank deficient because it has three columns in two dimensions, but that fact alone does not force lasso coefficients to be nonunique: the relevant condition here concerns the columns in \(E\). As a direct check, the objective at the solution is \(\frac12(1^2+0^2)+2=2.5\).
Common mistake avoided. Counting columns and rows, or checking only whether the full design matrix is rank deficient, does not settle uniqueness. The KKT check identifies which columns can participate in any optimum; independence of those equicorrelation columns is what supplies the uniqueness argument here.
A Sufficient Condition Is Not a Necessary One
If the columns in \(E\) are dependent, the sufficient condition does not answer whether coefficients are unique. It does not prove nonuniqueness; it only means that this particular independence argument cannot establish uniqueness. The next example shows why this distinction matters.
Worked Example: A Dependent Equicorrelation Set Can Still Have a Unique Minimizer
Let \(x_1=x_2=(1,0)^\mathsf{T}\), \(y=(1,0)^\mathsf{T}\), and \(\lambda=1\). Determine whether the zero vector is the unique lasso minimizer.
Method. Write \(t=\beta_1+\beta_2\), so \(X\beta=(t,0)^\mathsf{T}\). For any coefficients, \(|\beta_1|+|\beta_2|\geq|t|\). The objective is therefore at least \(\frac12(1-t)^2+|t|\), with equality attainable for each \(t\) by a suitable allocation of coefficients.
At \(t=0\), the zero vector has objective \(\frac12\). For \(t\geq0\), the difference between the lower bound and \(\frac12\) is \(\frac12(1-t)^2+t-\frac12=\frac12t^2\), which is positive whenever \(t\ne0\). For \(t<0\), the difference is \(\frac12(1-t)^2-t-\frac12=\frac12t^2-2t\), also positive. Thus any minimizer must have \(t=0\). If \(t=0\), the objective is \(\frac12+|\beta_1|+|\beta_2|\), which equals \(\frac12\) only when both coefficients are zero. The zero vector is the unique minimizer.
At that solution, the residual is \(y\), and both columns have inner product \(1=\lambda\) with the residual. Thus \(E=\{1,2\}\) is dependent, even though the coefficient vector is unique. The example confirms that independence of \(X_E\) is sufficient but not necessary.
Check and common mistake. The zero vector satisfies the KKT inequalities, and the direct objective comparison shows that no nonzero vector ties it. It would be incorrect to conclude that dependent equicorrelation columns automatically produce multiple minimizers.
Common Traps and Practical Interpretation
When assessing a lasso result, separate three questions: whether the fit is unique, whether the coefficient vector is unique, and whether a particular sufficient condition for coefficient uniqueness applies. They are not interchangeable.
- Confusing fitted values with coefficients. All minimizers give the same \(X\hat\beta\), but may allocate that fit differently across dependent predictors.
- Using independence of one active set as a uniqueness proof. The support of one solution may omit other columns that can appear in another optimum. Check independence of the full equicorrelation set, or use a separate argument that rules out other optimal representations.
- Treating a failed sufficient condition as proof of nonuniqueness. A dependent equicorrelation set makes the condition inconclusive; the zero-vector example shows that a unique solution can still occur.
- Changing objective scaling without changing the KKT threshold. For the stated objective, the threshold is \(\lambda\). If the squared-error term is divided by \(n\), the corresponding gradient and threshold relation must be adjusted consistently.
For interpretation, a unique fitted vector ensures that predictions on the observed design are settled, while nonunique coefficients mean the separate effects assigned to collinear predictors are not identified by this optimization problem alone. When reporting a lasso solution, avoid describing a coefficient allocation as the only possible one unless uniqueness has been established.
Check Your Understanding
For each problem, identify the relevant fitted values, residual correlations, or coefficient allocations before deciding what is unique.
- Two predictor columns are both \(x=(1,1)^\mathsf{T}\), the response is \(y=(2,2)^\mathsf{T}\), and \(\lambda=1\). Find the optimal coefficient sum and give two distinct minimizing coefficient vectors. (Answer: \(\beta_1+\beta_2=1.5\); for example, \((1.5,0)^\mathsf{T}\) and \((0.75,0.75)^\mathsf{T}\), both with objective \(1.75\).)
- For a lasso minimizer, suppose \(X_E\) has linearly independent columns and all coefficients outside \(E\) are zero in every minimizer. What does this imply about the coefficient vector, and why? (Answer: It is unique, because equal fitted values imply \(X_E\) times the coefficient difference is zero, and independence forces that difference to be zero.)
- Let \(x_1=x_2=(1,0)^\mathsf{T}\), \(y=(1,0)^\mathsf{T}\), and \(\lambda=2\). Is the zero vector the unique minimizer? (Answer: Yes. With \(t=\beta_1+\beta_2\), the objective is at least \(\frac12(1-t)^2+2|t|\); its difference from the zero-vector objective \(\frac12\) is \(\frac12t^2+t>0\) for \(t>0\), and \(\frac12t^2-3t>0\) for \(t<0\).)
- A proposed solution has one nonzero coefficient, and that predictor column is nonzero. Is that fact alone enough to prove coefficient uniqueness? State the additional set and condition that give a standard sufficient test. (Answer: No. Check the equicorrelation set \(E=\{j:|x_j^\mathsf{T}\hat r|=\lambda\}\); linear independence of all columns in \(X_E\) is a sufficient condition.)
- Suppose two distinct lasso minimizers are reported for the same data and penalty. What must be true of their fitted vectors and residuals, and what does that imply about the residual inner products used to form \(E\)? (Answer: Their fitted vectors and residuals must agree; consequently their residual inner products and the equicorrelation set \(E\) agree as well.)