A Worked Lasso Example With Orthogonal Predictors

Why Work Through an Orthogonal Lasso Fit?

Lasso estimates are generally found by numerical optimization, but an orthogonal design gives a useful exact case: each coefficient can be calculated separately. Working through a complete example makes the penalty’s effect visible. A coefficient is reduced toward zero, and it becomes exactly zero once its predictor-response inner product is no larger in absolute value than the penalty.

The calculations below use a particular, explicit objective. Lasso software can use different normalizations, so the numerical meaning of a penalty value depends on the objective being minimized. Here there is no intercept, and $\lambda$ is the penalty in the stated formula.

Definition: For response vector $y$, design matrix $X$, coefficient vector $\beta$, and penalty $\lambda\geq 0$, the lasso estimate minimizes $$ J(\beta)=\frac{1}{2}\|y-X\beta\|_2^2+\lambda\sum_{j=1}^p|\beta_j|. $$ The columns $x_j$ of $X$ are orthogonal if $x_j^\mathsf{T}x_k=0$ whenever $j\ne k$. In this tutorial, each column has positive squared length $d_j=x_j^\mathsf{T}x_j$. A predictor is selected when its estimated coefficient is nonzero.

The Formula for an Orthogonal Design

Write $c_j=x_j^\mathsf{T}y$, the inner product of predictor $j$ with the response. With orthogonal columns, expanding the squared error separates the coefficient-dependent terms into one expression per predictor. Terms not depending on $\beta_j$ can be ignored when minimizing over that coefficient.

\frac{1}{2}\|y-X\beta\|_2^2+\lambda\sum_j|\beta_j| \quad\longrightarrow\quad \sum_j\left(\frac{d_j}{2}\beta_j^2-c_j\beta_j+\lambda|\beta_j|\right).

For a single coefficient, the derivative of the smooth part is $d_j\beta_j-c_j$. If the minimizing coefficient is positive, the derivative of $\lambda|\beta_j|$ is $\lambda$, so the first-order condition gives $d_j\beta_j-c_j+\lambda=0$. If it is negative, that derivative is $-\lambda$, giving $d_j\beta_j-c_j-\lambda=0$. At zero, the absolute-value penalty has subgradients from $-\lambda$ to $+\lambda$; zero minimizes the expression when $|c_j|\leq\lambda$. These cases give the soft-thresholding formula.

Theorem: Under the stated lasso objective and orthogonal design, the minimizing coefficient for predictor $j$ is $$ \widehat{\beta}_j = \frac{\operatorname{sign}(c_j)(|c_j|-\lambda)_+}{d_j}, \qquad (a)_+=\max(a,0). $$ Thus predictor $j$ is selected exactly when $|c_j|>\lambda$. If $\lambda=0$, the formula reduces to the least-squares coefficient $c_j/d_j$. The separable formula does not generally apply when the design columns are correlated.

A second way to check the result is through the lasso optimality conditions. Let $r=y-X\widehat{\beta}$ be the residual vector. For a nonzero coefficient, $x_j^\mathsf{T}r=\lambda\operatorname{sign}(\widehat{\beta}_j)$. For a zero coefficient, $|x_j^\mathsf{T}r|\leq\lambda$. These conditions verify both the signs of selected coefficients and whether omitted predictors satisfy the required bound.

Worked Calculations With a Small Design

Consider three predictors with four observations. Their columns and the response are

x_1=(1,-1,-1,1)^\mathsf{T},\quad x_2=(1,-1,1,-1)^\mathsf{T},\quad x_3=(1,1,-1,-1)^\mathsf{T},\quad y=(5,-4,-2,1)^\mathsf{T}.

Each column has squared length 4. Their pairwise inner products are $x_1^\mathsf{T}x_2=1+1-1-1=0$, $x_1^\mathsf{T}x_3=1-1+1-1=0$, and $x_2^\mathsf{T}x_3=1-1-1+1=0$. The response inner products are $c_1=12$, $c_2=6$, and $c_3=2$. For example, $c_2=5+4-2-1=6$. Ordinary least squares therefore gives $(12/4,6/4,2/4)=(3,1.5,0.5)$.

Worked Example: A Small Penalty and Three Selected Predictors

Fit the lasso to this design at $\lambda=0.5$. Calculate the coefficients and fitted vector, then verify the result using residual correlations. The method is soft-thresholding of each response inner product; it avoids treating the penalty as a multiplier on the least-squares coefficients.

Step 1: Threshold each inner product. The three values are $c_j=(12,6,2)$. Subtract $0.5$ from each absolute value, retaining the original positive signs: $11.5$, $5.5$, and $1.5$.

Step 2: Divide by each squared column length. Since $d_j=4$ for all three columns, $$ \widehat{\beta}=(11.5/4,\;5.5/4,\;1.5/4)=(2.875,\;1.375,\;0.375). $$ All three coefficients are nonzero, so all three predictors are selected. The least-squares coefficients were $(3,1.5,0.5)$, which are larger in absolute value in each coordinate.

Step 3: Calculate the fitted vector. The contributions from predictors 2 and 3 are $1.375x_2+0.375x_3=(1.75,-1,1,-1.75)^\mathsf{T}$: the second entry is $-1.375+0.375=-1$, and the fourth is $-1.375-0.375=-1.75$. Adding $2.875x_1=(2.875,-2.875,-2.875,2.875)^\mathsf{T}$ gives $$ \widehat{y}=X\widehat{\beta}=(4.625,-3.875,-1.875,1.125)^\mathsf{T}. $$

Step 4: Check the residual and the objective. The residual is $r=y-\widehat{y}=(0.375,-0.125,-0.125,-0.125)^\mathsf{T}$. Its inner products with $x_1,x_2,x_3$ are each $0.5$, matching $\lambda\operatorname{sign}(\widehat{\beta}_j)=0.5$ for all three positive coefficients. Also, $\|r\|_2^2=0.375^2+3(0.125^2)=0.1875$, and $\sum_j|\widehat{\beta}_j|=4.625$. Thus $J=0.5(0.1875)+0.5(4.625)=2.40625$. The residual-correlation check confirms the fit.

Worked Example: Comparing Two Penalty Values

Using the same design, calculate the lasso estimates at $\lambda=1$ and $\lambda=2$. State which predictors are selected and compare the fitted vectors with the least-squares fit. The key is to threshold the inner products first and divide by 4 second.

At $\lambda=1$: The thresholded values are $12-1=11$, $6-1=5$, and $2-1=1$. Dividing by 4 gives $\widehat{\beta}=(2.75,1.25,0.25)$, so all three predictors remain selected. Combining the columns gives $\widehat{y}=(4.25,-3.75,-1.75,1.25)^\mathsf{T}$. The residual is $(0.75,-0.25,-0.25,-0.25)^\mathsf{T}$, with squared norm $0.75^2+3(0.25^2)=0.75$. Its inner product with each column is 1, as required for three positive coefficients at this penalty.

At $\lambda=2$: The thresholded values are $12-2=10$, $6-2=4$, and $(2-2)_+=0$. Therefore $\widehat{\beta}=(2.5,1,0)$: predictors 1 and 2 are selected, while predictor 3 is not. The fitted vector is $2.5x_1+x_2=(3.5,-3.5,-1.5,1.5)^\mathsf{T}$. The residual is $(1.5,-0.5,-0.5,-0.5)^\mathsf{T}$. Its inner products with $x_1$ and $x_2$ are both 2; its inner product with $x_3$ is 2, satisfying the inactive-predictor condition $|x_3^\mathsf{T}r|\leq2$.

Least squares fits $y$ exactly in this example, with residual sum of squares zero. The lasso fits above have nonzero residuals because the penalty changes the coefficient estimates. At $\lambda=1$, no predictor has yet been removed; at $\lambda=2$, the third coefficient is exactly zero.

Worked Example: Finding a Selection Boundary

For the same design, determine the lasso estimates at $\lambda=6$ and $\lambda=12$, and explain why the selected set changes. This calculation identifies selection directly from $|c_j|>\lambda$, rather than assuming that every predictor survives a positive penalty.

At $\lambda=6$, the thresholded inner products are $12-6=6$, $(6-6)_+=0$, and $(2-6)_+=0$. Dividing by 4 gives $\widehat{\beta}=(1.5,0,0)$, so only predictor 1 is selected. The residual is $y-1.5x_1=(3.5,-2.5,-0.5,-0.5)^\mathsf{T}$. Its correlations with the columns are 6, 6, and 2, respectively. The active correlation equals $\lambda$; the inactive correlations have absolute values no greater than $\lambda$.

At $\lambda=12$, all three thresholded values are zero: $(12-12)_+=0$, $(6-12)_+=0$, and $(2-12)_+=0$. Hence $\widehat{\beta}=(0,0,0)$. The all-zero fit is valid because the absolute response correlations, 12, 6, and 2, are all at most 12. At the boundary $\lambda=12$, predictor 1 is also zero; for any $\lambda>12$, the all-zero solution remains the minimizer.

How to Read the Results and Avoid Traps

For this fixed design, the selection boundaries are the absolute inner products: 12 for predictor 1, 6 for predictor 2, and 2 for predictor 3. As $\lambda$ increases past a predictor’s boundary, its coefficient becomes zero. At equality, the coefficient is already zero. Below the boundary, its coefficient’s magnitude is reduced by $\lambda/d_j$ relative to its least-squares magnitude.

The table summarizes the calculations. Each coefficient is in the order $(\widehat{\beta}_1,\widehat{\beta}_2,\widehat{\beta}_3)$.

Penalty $\lambda$	Lasso coefficients	Selected predictors
0	(3, 1.5, 0.5)	1, 2, 3
0.5	(2.875, 1.375, 0.375)	1, 2, 3
1	(2.75, 1.25, 0.25)	1, 2, 3
2	(2.5, 1, 0)	1, 2
6	(1.5, 0, 0)	1
12	(0, 0, 0)	None

A common sign trap is to carry an incorrect sign from an inner product into the thresholding step. Here $x_2^\mathsf{T}y=6$, so at $\lambda=1$ its thresholded value is $6-1=5$, not $-5$; at $\lambda=2$ it is $6-2=4$, not $-4$. The coefficient is then $5/4$ or $4/4$, respectively. If an inner product is negative, retain that negative sign after reducing its magnitude: for example, $c=-6$ at $\lambda=1$ gives $-5$, not $+5$.

Another trap is mixing objective conventions. The result above uses $\frac12\|y-X\beta\|_2^2+\lambda\|\beta\|_1$. If instead the objective is $\frac{1}{2n}\|y-X\beta\|_2^2+\lambda\|\beta\|_1$, multiplying the whole objective by $n$ shows that the equivalent penalty in the first convention is $n\lambda$. A numerical penalty cannot be transferred between these forms without adjusting for that factor. The orthogonal formula also depends on the column lengths: divide the thresholded inner product by $d_j$, not by a common value unless the columns really have equal squared lengths.

Key takeaway: Under the specified objective, calculate $c_j=x_j^\mathsf{T}y$, threshold its magnitude by $\lambda$, restore its sign, and divide by $x_j^\mathsf{T}x_j$. Then check active residual correlations against $\lambda$ and inactive ones against the bound $|x_j^\mathsf{T}r|\leq\lambda$. This exact separation relies on orthogonal predictors.

Check Your Understanding

Use the stated objective and, unless a question says otherwise, the orthogonal design above. Work through the thresholding and division before checking the final answer.

For the stated design, find the coefficients and selected predictors at $\lambda=3$. (Answer: $(2.25,0.75,0)$; predictors 1 and 2.)
For the stated design, find the coefficients at $\lambda=5$. Which predictors are selected? (Answer: $(1.75,0.25,0)$; predictors 1 and 2.)
For the stated design, calculate the coefficients at $\lambda=2.5$, and state why predictor 3 is excluded. (Answer: $(2.375,0.875,0)$; predictor 3 is excluded because $|c_3|=2\leq2.5$.)
In a different orthogonal design, one column has squared length $d_j=9$ and response inner product $c_j=-5$. At $\lambda=2$, calculate its coefficient and say whether it is selected. (Answer: $-3/9=-1/3$; it is selected.)
A software objective is $\frac{1}{2n}\|y-X\beta\|_2^2+\lambda\|\beta\|_1$, with $n=5$. For an orthogonal column with squared length 10 and response inner product 8, calculate its coefficient at $\lambda=0.4$. (Answer: $(8-5(0.4))/10=0.6$.)

What You'll Learn