Fitted-Value Uniqueness Under Lasso

Why Prediction and Coefficient Uniqueness Are Different

A lasso fit reports coefficients, but the model’s predictions on the training data depend on those coefficients only through their linear combination. When predictors are linearly dependent, different coefficient vectors can sometimes produce exactly the same linear combination. The distinction matters: one can have a well-defined fitted vector even when there is no single coefficient vector to report as the unique answer.

This tutorial focuses on that distinction for the squared-error lasso. It is aligned with the SOA Exam PA (Predictive Analytics) syllabus. Throughout, assume there are $n$ observations and $p$ predictors, and use centered data with no intercept in the displayed examples. An unpenalized intercept can be handled separately; it does not change the argument about fitted values.

Definition: For response vector $y$, design matrix $X$, and coefficient vector $\beta$, the lasso minimizes $$ Q(\beta)=\frac{1}{2n}\|y-X\beta\|_2^2+\lambda\|\beta\|_1,\qquad \lambda>0. $$ The fitted vector is $\eta=X\beta$, the vector of predictions on the $n$ observations. A coefficient vector is a lasso solution if it attains the minimum value of $Q$.

The penalty is convex, but the squared-error term is not strictly convex in $\beta$ unless the columns of $X$ are linearly independent. It is, however, strictly convex in the fitted vector $X\beta$. That difference gives the central result.

Theorem: For squared-error lasso, all minimizing coefficient vectors produce the same fitted vector $X\beta$. The minimizing coefficient vector itself need not be unique. If $\lambda>0$, any two minimizing vectors that produce the same fitted vector must also have the same $L_1$ norm.

Why the Fitted Vector Is Unique

Suppose, to the contrary, that two minimizing coefficient vectors $\beta^{(1)}$ and $\beta^{(2)}$ give different fitted vectors $\eta_1=X\beta^{(1)}$ and $\eta_2=X\beta^{(2)}$. The midpoint coefficient vector $(\beta^{(1)}+\beta^{(2)})/2$ has fitted vector $(\eta_1+\eta_2)/2$. The squared-error loss at this midpoint is strictly less than the average of the losses at the two endpoints, while convexity ensures that its penalty is no greater than the average penalty. The midpoint would therefore have a lower objective than a minimizer, a contradiction.

\frac{\|y-\eta_1\|_2^2+\|y-\eta_2\|_2^2}{2} -\left\|y-\frac{\eta_1+\eta_2}{2}\right\|_2^2 =\frac{\|\eta_1-\eta_2\|_2^2}{4}>0

After multiplying this strict loss reduction by $1/(2n)$, it is $\|\eta_1-\eta_2\|_2^2/(8n)$. For the penalty, convexity of the $L_1$ norm gives $\|(\beta^{(1)}+\beta^{(2)})/2\|_1\leq(\|\beta^{(1)}\|_1+\|\beta^{(2)}\|_1)/2$. Thus the midpoint objective is strictly below the average of the two minimizing objectives, which is the minimum itself. That contradiction proves fitted-value uniqueness.

For $\lambda>0$, if two solutions have the same fitted vector, they have the same squared-error loss. Since their total objectives are also equal, their $L_1$ penalties must be equal. This does not mean that every pair of coefficient vectors producing the same fitted vector are both solutions: their penalty values may differ.

Checking Solutions with the KKT Conditions

The Karush–Kuhn–Tucker (KKT) conditions provide a practical way to verify a proposed lasso solution. Define the residual $r=y-X\beta$. At a minimizer, for each predictor $j$, $x_j^\mathsf{T}r/n=\lambda\operatorname{sign}(\beta_j)$ when $\beta_j\ne 0$, and $|x_j^\mathsf{T}r/n|\leq\lambda$ when $\beta_j=0$. Here $x_j$ is column $j$ of $X$. These conditions use the objective scaling $1/(2n)$ specified above.

Worked Example: Two Different Solutions with the Same Predictions

A hypothetical model has $n=2$ observations and three predictor columns $x_1=(1,1)^\mathsf{T}$, $x_2=(1,-1)^\mathsf{T}$, and $x_3=(1,0)^\mathsf{T}$. The response is $y=(2.5,0)^\mathsf{T}$, and $\lambda=0.25$. Verify that $\beta^{(A)}=(1,1,0)^\mathsf{T}$ and $\beta^{(B)}=(0,0,2)^\mathsf{T}$ are both lasso solutions, then describe a continuum of solutions. The method is to compare the predictions and objectives, then check the KKT conditions.

Predictions. Since $x_1+x_2=(2,0)^\mathsf{T}=2x_3$, both vectors give $X\beta=(2,0)^\mathsf{T}$. The residual for either is $r=y-X\beta=(0.5,0)^\mathsf{T}$.

Objective value. The squared residual norm is $0.5^2+0^2=0.25$, so the loss is $0.25/(2n)=0.25/4=0.0625$. Each coefficient vector has $L_1$ norm $2$, so its penalty is $0.25(2)=0.5$. Each objective is $0.0625+0.5=0.5625$.

KKT check. The three residual scores are $x_1^\mathsf{T}r/n=0.5/2=0.25$, $x_2^\mathsf{T}r/n=0.5/2=0.25$, and $x_3^\mathsf{T}r/n=0.5/2=0.25$. For $\beta^{(A)}$, the first two positive coefficients have score $+\lambda$, and the zero third coefficient has absolute score $\lambda$, which satisfies the required inequality. For $\beta^{(B)}$, the third positive coefficient has score $+\lambda$, and both zero coefficients satisfy the same inequality. The objective is convex, so satisfying these KKT conditions verifies global optimality.

In fact, for any $t$ between $0$ and $2$, the vector $\beta(t)=(1-t/2,1-t/2,t)^\mathsf{T}$ has the same fit: its first two terms sum to $(1-t/2)(x_1+x_2)$, and its third term is $t(x_1+x_2)/2$. Together these equal $x_1+x_2$. Its coefficients are nonnegative and its $L_1$ norm is $1-t/2+1-t/2+t=2$. Its residual scores therefore satisfy the KKT conditions for every such $t$. This is a continuum of minimizing coefficient vectors with the same fitted vector.

Check and trap avoided. At $t=0$ and $t=2$, the stated endpoint vectors are recovered, and both have objective $0.5625$. Do not infer that a unique fit forces a unique coefficient vector; here the predictor columns obey $x_3=(x_1+x_2)/2$, allowing different coefficient allocations.

Equal Predictions Do Not Guarantee Equal Objectives

Linear dependence is necessary for two different coefficient vectors to give the same fitted vector: if $X\beta=X\gamma$, then $X(\beta-\gamma)=0$. But this equality alone does not make both vectors lasso solutions. With $\lambda>0$, their $L_1$ norms also matter. If their predictions agree but one vector has a larger $L_1$ norm, it has a larger objective and cannot be a minimizer whenever the lower-penalty representation is available.

Worked Example: Same Fit, Different Penalty

Use the predictor columns and response from the preceding example. Compare $\beta=(1,1,0)^\mathsf{T}$ with $\gamma=(1.5,1.5,-1)^\mathsf{T}$. The task is to decide whether equal predictions are enough to make both vectors lasso solutions. The method is to calculate the fitted vector and objective for each, using $\lambda=0.25$.

Fitted vectors. The first vector gives $x_1+x_2=(2,0)^\mathsf{T}$. For the second, $1.5x_1+1.5x_2-x_3=1.5(x_1+x_2)-x_3=(3,0)^\mathsf{T}-(1,0)^\mathsf{T}=(2,0)^\mathsf{T}$. Thus the predictions and residual $(0.5,0)^\mathsf{T}$ agree.

Penalties and objectives. The first vector has $L_1$ norm $2$, giving penalty $0.25(2)=0.5$. Its loss is $0.0625$, so its objective is $0.5625$. The second vector has $L_1$ norm $1.5+1.5+1=4$, giving penalty $0.25(4)=1$. Its loss is still $0.0625$, so its objective is $1.0625$.

Check and trap avoided. Both predictions are $(2,0)^\mathsf{T}$, but the objectives differ by $0.5$. Since the first vector is a minimizer by the preceding KKT check, the second is not. The common mistake is to treat equal fitted values as sufficient for equal lasso status; the correction is to compare the penalties as well.

When Coefficients Are Unique

The fitted-value theorem does not say coefficients are always nonunique. A simple sufficient condition for coefficient uniqueness is that $X$ has full column rank. Then the squared-error term is strictly convex as a function of $\beta$, and adding a convex penalty preserves strict convexity. A strictly convex objective has at most one minimizer.

When $X$ is rank deficient, coefficient uniqueness is not guaranteed, but it is still possible. The key question is whether distinct coefficient vectors can satisfy both requirements: they must produce the same fitted vector, and they must have the same minimum objective. Predictor dependence creates the possibility; it does not by itself prove that multiple solutions exist.

Worked Example: A Unique Coefficient Vector in a Full-Rank Design

Let $n=2$, $x_1=(1,1)^\mathsf{T}$, $x_2=(1,-1)^\mathsf{T}$, $y=(2,0)^\mathsf{T}$, and $\lambda=0.25$. Find the lasso solution and determine whether the coefficient vector is unique. The method is to use the orthogonal columns to identify a candidate, then check the objective and KKT conditions.

The columns are orthogonal and each has squared norm $2$, so $x_j^\mathsf{T}x_j/n=1$. The response scores are $x_1^\mathsf{T}y/n=2/2=1$ and $x_2^\mathsf{T}y/n=2/2=1$. For this orthonormal-scaled design, each coordinate minimizes a quadratic with unit curvature and an $L_1$ penalty; the positive-coordinate minimizer is the score minus $\lambda$. Thus $\hat{\beta}=(1-0.25,1-0.25)^\mathsf{T}=(0.75,0.75)^\mathsf{T}$.

The fitted vector is $0.75x_1+0.75x_2=(1.5,0)^\mathsf{T}$, so the residual is $(0.5,0)^\mathsf{T}$. Its squared norm is $0.25$, giving loss $0.25/4=0.0625$. The penalty is $0.25(0.75+0.75)=0.375$, so the objective is $0.4375$. Each residual score is $0.5/2=0.25=\lambda$, as required for the two positive coefficients.

The design matrix has full column rank because its columns are nonzero and orthogonal. The objective is strictly convex in $\beta$, so this KKT solution is the unique coefficient vector as well as the unique fitted vector. The check is consistent: the coefficients are positive, their residual scores equal $+\lambda$, and the objective components sum to $0.0625+0.375=0.4375$.

Exam Checks and Common Misreadings

A reliable response separates three questions: Are the predictions the same? Are the penalty values the same? Do the vectors satisfy the optimality conditions? The first question concerns the null space of $X$; the second concerns the $L_1$ penalty; the third determines whether a candidate actually minimizes the objective.

Do not confuse uniqueness of predictions with uniqueness of coefficients. The theorem guarantees one fitted vector, not necessarily one coefficient vector.
Do not assume every equal-fit pair is optimal. Different $L_1$ norms give different objectives when $\lambda>0$.
Use the stated scaling consistently. With loss $\|y-X\beta\|_2^2/(2n)$, KKT residual scores are divided by $n$. Omitting that divisor changes the apparent value of $\lambda$.
Do not infer coefficient uniqueness just from a numerical optimizer returning one vector. In a rank-deficient design, a software routine may return one solution among several.

Key takeaway: Squared-error lasso has a unique fitted vector because the loss is strictly convex in predictions. Different coefficient vectors can both minimize the objective only when they give that same fit and attain the same minimum penalty; linear dependence makes this possible but does not guarantee it.

Check Your Understanding

For each problem, distinguish equality of predictions from lasso optimality.

Suppose $x_2=2x_1$. Two coefficient vectors are $(3,0)$ and $(1,1)$. Do they give the same fitted vector? Compare their $L_1$ norms. (Answer: Yes; both give $3x_1$. Their norms are 3 and 2, so for $\lambda>0$ they do not have equal penalties.)
Suppose $x_3=(x_1+x_2)/2$. Give a coefficient vector other than $(1,1,0)$ that produces $x_1+x_2$, and state its $L_1$ norm. (Answer: $(0,0,2)$; its fitted vector is $2x_3=x_1+x_2$, and its $L_1$ norm is 2.)
Two lasso minimizers have fitted vectors $\eta_1$ and $\eta_2$. What must be true of these vectors, and what argument establishes it? (Answer: $\eta_1=\eta_2$; otherwise their midpoint gives a strictly smaller squared-error loss while convexity prevents an increase in the average penalty.)
Let $n=2$, $x_1=(1,1)^\mathsf{T}$, $x_2=(1,-1)^\mathsf{T}$, $x_3=(1,0)^\mathsf{T}$, $y=(3,0)^\mathsf{T}$, and $\lambda=0.5$. Check whether $\beta=(1,1,0)^\mathsf{T}$ satisfies the KKT conditions, and calculate its objective. (Answer: Yes; the fit is $(2,0)^\mathsf{T}$, the residual scores are $0.5,0.5,0.5$, and the zero coefficient satisfies the inequality. The loss is $1/4=0.25$, the penalty is $0.5(2)=1$, and the objective is 1.25.)

What You'll Learn