Exact Duplicates and Lasso Solutions

Why Exact Duplicates Matter

Lasso can select variables by shrinking some fitted coefficients to zero. But when two predictor columns are exactly identical, the data cannot distinguish their separate contributions to the fitted values. Their coefficients enter every prediction through their sum. The lasso penalty can also depend only on that sum for a whole range of coefficient allocations, so more than one coefficient vector can attain the same minimum objective.

This is a precise special case of ambiguity in coefficient allocation, not evidence that the fitted predictions must differ. The distinction matters when interpreting a selected variable: if two columns contain the same information, a nonzero coefficient assigned to one rather than the other may be an arbitrary feature of the solution returned by a fitting procedure.

Definition: Two predictors are exact duplicates if their design-matrix columns have the same value in every observation. For duplicate columns \(x_1=x_2=x\), with coefficients \(\beta_1\) and \(\beta_2\), their combined coefficient is \(\gamma=\beta_1+\beta_2\). Their contribution to the fitted value for observation \(i\) is \(x_i\gamma\).

The Lasso Objective for Duplicate Columns

Use the lasso objective with an unpenalized intercept and, for clarity, a centered response and centered predictor columns. With the intercept then equal to zero, the objective for \(n\) observations is

Q(\beta_1,\beta_2) = \frac{1}{2n}\sum_{i=1}^{n} \left(y_i-x_i\beta_1-x_i\beta_2\right)^2 +\lambda\left(|\beta_1|+|\beta_2|\right), \qquad \lambda>0.

The first term is the scaled residual sum of squares; the second is the \(L_1\) penalty. The factor \(1/(2n)\) is part of the objective convention here, so numerical answers depend on using it consistently. Because \(x_i\beta_1+x_i\beta_2=x_i(\beta_1+\beta_2)\), the residual term depends on the two coefficients only through \(\gamma\). The penalty, however, could depend on how \(\gamma\) is divided.

Key result: For a fixed combined coefficient \(\gamma=\beta_1+\beta_2\), the smallest possible penalty contribution is \(\lambda|\gamma|\). It is attained by every allocation of \(\gamma\) between the duplicate coefficients that has no cancellation: both coefficients are nonnegative when \(\gamma>0\), both are nonpositive when \(\gamma<0\), or either coefficient is zero. Thus, when the optimal \(\gamma\) is nonzero, there can be multiple coefficient pairs with the same fitted values and objective value.

Why the Penalty Can Be Shared

The result follows from the triangle inequality. For any two coefficients that sum to \(\gamma\),

|\beta_1|+|\beta_2| \geq |\beta_1+\beta_2| =|\gamma|.

Equality holds when the two terms do not oppose one another: \(\beta_1\) and \(\beta_2\) have the same sign, or one is zero. For \(\gamma>0\), all pairs \((\beta_1,\beta_2)=(t,\gamma-t)\) with \(0\leq t\leq\gamma\) meet this condition. Each pair has the same fitted values because its coefficients sum to \(\gamma\), and each has the same penalty because \(|t|+|\gamma-t|=\gamma\). If the coefficients have opposite signs, their absolute values sum to more than \(|\gamma|\), so the penalty is larger while the fitted values stay unchanged.

Consequently, the best objective for a fixed \(\gamma\) is obtained by using the reduced one-coefficient expression

Q_{\mathrm{reduced}}(\gamma) = \frac{1}{2n}\sum_{i=1}^{n}(y_i-x_i\gamma)^2 +\lambda|\gamma|.

This reduction is a way to calculate the best combined coefficient in these examples. It does not say that every allocation with the same sum has the same penalty: only allocations attaining the triangle-inequality bound do. The examples below calculate both the combined coefficient and the possible allocations.

Worked Example: A Positive Combined Coefficient

Suppose there are three observations, with duplicate centered predictor columns \(x_1=x_2=(-1,0,1)\), centered response \(y=(-2,0,2)\), and \(\lambda=2/3\). Find the optimal combined coefficient, give three optimal coefficient allocations, and calculate the objective value.

Method. Set \(\gamma=\beta_1+\beta_2\). Since \(y_i=2x_i\), the residuals are \((2-\gamma)x_i\). The sum of squared predictor values is \((-1)^2+0^2+1^2=2\), so the residual sum of squares is \(2(2-\gamma)^2\). With \(n=3\), the reduced objective is

Q_{\mathrm{reduced}}(\gamma) = \frac{2(2-\gamma)^2}{6} +\frac{2}{3}|\gamma| = \frac{(2-\gamma)^2}{3} +\frac{2}{3}|\gamma|.

For \(\gamma>0\), differentiate the quadratic expression: \(Q'_{\mathrm{reduced}}(\gamma)=-2(2-\gamma)/3+2/3\). Setting this equal to zero gives \(2-\gamma=1\), hence \(\gamma=1\). The solution is in the positive region assumed in the calculation.

Every nonnegative pair summing to 1 is an optimal allocation. Three examples are \((\beta_1,\beta_2)=(1,0)\), \((0.4,0.6)\), and \((0,1)\). Each produces the same fitted vector \((-1,0,1)\). For each pair, the residual vector is \(y-\hat{y}=(-1,0,1)\), giving a residual sum of squares of \(1+0+1=2\). The loss term is \(2/(2\cdot3)=1/3\), and the penalty is \((2/3)(1)=2/3\). Therefore, each objective value is \(1/3+2/3=1\).

Check. The combined coefficient is 1 for all three pairs, and every pair has absolute coefficients summing to 1. The fitted values and objective calculations agree. The common mistake avoided here is treating \((1,0)\) as the only possible optimal allocation.

Worked Example: An Allocation with Cancellation Costs More

Keep the duplicate columns and response from the previous calculation, but consider two coefficient pairs: \((0.8,0.2)\) and \((1.3,-0.3)\). Both have combined coefficient \(\gamma=1\). With \(\lambda=2/3\), compare their fitted values and objective values.

Method. Since both coefficient sums equal 1, both fitted vectors are \((-1,0,1)\), and both have the same residual sum of squares, 2. Their loss terms are therefore both \(2/6=1/3\). For the first pair, the absolute coefficients sum to \(0.8+0.2=1\); its penalty is \((2/3)(1)=2/3\), giving objective \(1/3+2/3=1\).

For the second pair, the absolute coefficients sum to \(1.3+0.3=1.6\); its penalty is \((2/3)(1.6)=16/15\). Its objective is \(1/3+16/15=5/15+16/15=21/15=7/5=1.4\). Thus the two pairs give identical predictions, but the cancellation pair has a larger objective by \(1.4-1=0.4\).

Check. The lower bound on the absolute-coefficient sum is \(|\gamma|=1\). The first pair attains it; the second has a larger sum because its coefficients oppose each other. A shared fitted contribution does not make every possible coefficient allocation equally good under the lasso penalty.

Negative Combined Coefficients

The same reasoning applies if the optimal combined coefficient is negative. In that case, equality in the triangle inequality requires both coefficients to be nonpositive, or one to be zero. Splitting a negative total between two negative coefficients preserves both the fitted values and the minimum penalty. The next example checks the sign and objective explicitly.

Worked Example: Splitting a Negative Coefficient

Let \(n=3\), \(x_1=x_2=(-1,0,1)\), \(y=(1,0,-1)=-x_1\), and \(\lambda=1/3\). Find the optimal combined coefficient and calculate the objective for two different allocations.

Method. The residual vector is \(-x_1-\gamma x_1=-(1+\gamma)x_1\). Since \(\sum_i x_i^2=2\), the residual sum of squares is \(2(1+\gamma)^2\), and the reduced objective is

Q_{\mathrm{reduced}}(\gamma) = \frac{(1+\gamma)^2}{3} +\frac{1}{3}|\gamma|.

For \(\gamma<0\), the derivative is \(2(1+\gamma)/3-1/3\). Setting it to zero gives \(2(1+\gamma)=1\), so \(\gamma=-1/2\), which is indeed negative. The residual sum of squares is \(2(1/2)^2=1/2\), and the loss term is \((1/2)/6=1/12\). The minimum penalty is \((1/3)(1/2)=1/6\), so the objective is \(1/12+1/6=3/12=1/4\).

Two allocations are \((-1/2,0)\) and \((-0.2,-0.3)\). Both sum to \(-1/2\), and both have absolute coefficients summing to \(1/2\), so each has objective \(1/4\) and fitted vector \((0.5,0,-0.5)\). By contrast, \((-0.8,0.3)\) also sums to \(-1/2\), but its absolute coefficients sum to \(1.1\). Its penalty is \(1.1/3=11/30\), and its objective is \(1/12+11/30=27/60=0.45\), which is greater than \(1/4\).

Check. The optimal combined coefficient is negative, so the equal-penalty allocations have nonpositive entries. The mixed-sign pair has the right sum but pays more penalty.

Common Exam Traps and Interpretation

Exact duplicates are a useful test of whether coefficient, prediction, and objective are being kept distinct. The lasso selects coefficients by minimizing a stated objective; it does not guarantee that the coefficients attached to indistinguishable columns are uniquely allocated.

Reporting one allocation as the only solution. A fitting routine may return \((\gamma,0)\), but with exact duplicates and \(\gamma>0\), every nonnegative split summing to \(\gamma\) has the same fitted values and minimum penalty.
Changing the coefficient sum. The equality of fitted values depends on keeping \(\beta_1+\beta_2\) fixed. If that sum changes, the duplicate predictors' contribution to each fitted value changes as well.
Ignoring signs in the penalty. For a fixed sum, opposite signs create cancellation and raise \(|\beta_1|+|\beta_2|\). Equal predictions alone do not imply equal objective values.
Using a different loss scaling without adjusting calculations. The examples use residual sum of squares divided by \(2n\). If an implementation or problem uses another scaling, the numerical meaning of \(\lambda\) changes; do not carry over a threshold or coefficient calculation without matching the stated objective.
Over-interpreting a chosen variable. When two columns are exact duplicates, a nonzero coefficient on one and zero on the other does not establish that the selected column has a distinct predictive contribution.

For a practical check, calculate the coefficient sum, compare fitted values, and separately compare the \(L_1\) penalty. This makes clear whether two candidate coefficient vectors are truly equivalent for the objective, rather than merely producing the same predictions.

Key takeaway: With exact duplicate columns, predictions depend on the sum of the duplicate coefficients. For a fixed sum, the smallest lasso penalty is attained when the coefficients do not cancel. When the combined coefficient is nonzero, that minimum can be shared across multiple allocations, so the coefficient vector need not be uniquely determined.

Check Your Understanding

For each problem, use the objective convention stated in the question and check predictions and penalty separately.

Let \(n=2\), \(x_1=x_2=(-1,1)\), \(y=(-2,2)\), and \(Q=\mathrm{RSS}/(2n)+\lambda(|\beta_1|+|\beta_2|)\) with \(\lambda=0.5\). Find the optimal combined coefficient and the minimum objective. (Answer: \(\gamma=1.5\), minimum objective \(0.875\).)
Two exact duplicate columns have coefficients \((0.8,0.2)\) and \((1.3,-0.3)\) in two candidate fits. If \(\lambda=0.4\), how much larger is the second candidate's penalty, and do the fitted values differ? (Answer: The second penalty is larger by \(0.24\); the fitted values do not differ.)
For \(n=3\), \(x_1=x_2=(-1,0,1)\), \(y=2x_1\), and \(Q=\mathrm{RSS}/(2n)+(1/3)(|\beta_1|+|\beta_2|)\), find the optimal combined coefficient and objective. Give one allocation with both coefficients nonzero. (Answer: \(\gamma=1.5\), minimum objective \(7/12\); for example, \((0.6,0.9)\).)
For duplicate predictors and a fixed combined coefficient \(\gamma=-0.7\), compare the penalty factors \(|\beta_1|+|\beta_2|\) for allocations \((-0.2,-0.5)\) and \((-1.0,0.3)\). Which achieves the minimum penalty factor? (Answer: The factors are \(0.7\) and \(1.3\); \((-0.2,-0.5)\) achieves the minimum.)
Suppose a lasso fit assigns coefficients \((2,0)\) to two exact duplicate columns, with \(\lambda>0\). State another allocation with the same fitted values and minimum penalty, and explain why the fitted values match. (Answer: \((1,1)\); both coefficient sums are 2, so the duplicate columns contribute the same amount to every fitted value, and both allocations have absolute coefficients summing to 2.)

What You'll Learn