Why Orthogonality Simplifies the Lasso
The lasso usually fits all coefficients together: changing one coefficient can alter the residuals relevant to the others. When predictor columns are orthogonal, that coupling disappears. The objective separates into independent one-coefficient problems, and each coefficient can be calculated directly from its unpenalized least-squares value.
This special case gives a precise account of what the lasso penalty does to a coefficient: it reduces its magnitude by an amount determined by the penalty, while preserving its sign until it reaches zero. The result is called soft thresholding. It is useful both as an exact solution under orthogonality and as a reference for understanding why correlated predictors require a joint fit.
Use the penalized least-squares objective \( \frac{1}{2n}\lVert y-X\beta\rVert_2^2+\lambda\sum_{j=1}^{p}|\beta_j| \), where \(y\) is the response vector, \(X\) is the design matrix, \(\beta_j\) is the coefficient for predictor \(j\), and \(\lambda\geq 0\) is the lasso penalty. Assume an intercept is absent or has been handled by centering the response and predictor columns. The intercept, if included, is not penalized.
The formula is a property of the optimization problem, not a claim about a probability model for the response. No normality or constant-variance assumption is needed to derive the minimizing coefficients. The design and response are treated as observed quantities.
Deriving the Coordinatewise Solution
Let \(d_j=x_j^\mathsf{T}x_j>0\) and \(c_j=x_j^\mathsf{T}y\). Expanding the squared residual gives \( \lVert y-X\beta\rVert_2^2=y^\mathsf{T}y-2\sum_j c_j\beta_j+\sum_j d_j\beta_j^2+2\sum_{j<k}\beta_j\beta_kx_j^\mathsf{T}x_k \). Orthogonality makes every cross-product term zero. The term \(y^\mathsf{T}y/(2n)\) is constant with respect to \(\beta\), so it does not affect the minimizer. The remaining objective is a sum of separate terms, one for each coefficient.
For a single coordinate, define \(z_j=c_j/n\) and \(q_j=d_j/n\), with \(q_j>0\). Completing the square shows that minimizing the coordinate’s contribution is equivalent to minimizing \( \frac{q_j}{2}(\beta_j-z_j/q_j)^2+\lambda|\beta_j| \). The unpenalized least-squares coefficient for that coordinate is \(z_j/q_j=c_j/d_j\). The absolute-value term changes the solution near zero, where its derivative is not defined; its subgradient at zero is the interval \([-1,1]\).
To verify the result, consider the three possible regions. For \(\beta_j>0\), differentiating the coordinate objective gives \(q_j\beta_j-z_j+\lambda=0\), so a positive solution is \((z_j-\lambda)/q_j\), which is valid when \(z_j>\lambda\). For \(\beta_j<0\), the derivative is \(q_j\beta_j-z_j-\lambda=0\), giving \((z_j+\lambda)/q_j\), valid when \(z_j<-\lambda\). At \(\beta_j=0\), zero minimizes the coordinate objective precisely when \(0\) belongs to its subgradient, or equivalently when \(|z_j|\leq\lambda\). Combining these cases gives the stated formula. Because \(q_j>0\), each coordinate objective is strictly convex, so its minimizer is unique.
Worked Examples
Worked Example: Thresholding Standardized Orthogonal Predictors
Suppose a hypothetical data set has \(n=4\) observations and three predictor columns \(x_1=(1,1,-1,-1)^\mathsf{T}\), \(x_2=(1,-1,1,-1)^\mathsf{T}\), and \(x_3=(1,-1,-1,1)^\mathsf{T}\). Each column has squared length 4, and every pair has inner product 0. Let the hypothetical response be \(y=2.4x_1-1.2x_2+0.5x_3\). Thus, \(x_j^\mathsf{T}x_j=n\), and the columns are orthogonal.
Since the columns are orthogonal, \(x_1^\mathsf{T}y/4=2.4\), \(x_2^\mathsf{T}y/4=-1.2\), and \(x_3^\mathsf{T}y/4=0.5\). At \(\lambda=0.7\), apply the standardized formula:
The first coefficient remains positive and is reduced by 0.7; the second remains negative and its magnitude is reduced by 0.7. The third is set to zero because its unpenalized coordinate value has magnitude smaller than the penalty. These are coefficient estimates on the response scale per one unit of each stated predictor.
Worked Example: Following Coefficients Across Penalties
Use the same hypothetical orthogonal design and response. Because the normalized predictor scores are \(2.4\), \(-1.2\), and \(0.5\), the coefficients at any chosen penalty follow directly from the soft-thresholding formula. For example, at \(\lambda=1.0\), the estimates are \(1.4\), \(-0.2\), and \(0\); at \(\lambda=1.2\), they are \(1.2\), \(0\), and \(0\); and at \(\lambda=2.4\), all three are zero.
| Penalty \(\lambda\) | \(\hat{\beta}_1\) | \(\hat{\beta}_2\) | \(\hat{\beta}_3\) |
|---|---|---|---|
| 0 | 2.4 | -1.2 | 0.5 |
| 0.4 | 2.0 | -0.8 | 0.1 |
| 1.0 | 1.4 | -0.2 | 0 |
| 1.2 | 1.2 | 0 | 0 |
| 2.4 | 0 | 0 | 0 |
For instance, at \(\lambda=0.4\), the first estimate is \(S(2.4,0.4)=2.4-0.4=2.0\); the second is \(S(-1.2,0.4)=-(1.2-0.4)=-0.8\); and the third is \(S(0.5,0.4)=0.1\). As the penalty rises, each coefficient moves toward zero without changing sign, and a coefficient stays zero once the penalty is at least as large as the magnitude of its score.
Worked Example: Orthogonal Predictors With Unequal Lengths
Suppose a hypothetical data set has \(n=3\), with \(x_1=(1,1,0)^\mathsf{T}\), \(x_2=(1,-1,1)^\mathsf{T}\), and response \(y=(2,0,1)^\mathsf{T}\). The columns are orthogonal because \(x_1^\mathsf{T}x_2=1-1+0=0\), but their squared lengths differ: \(d_1=2\) and \(d_2=3\). The response inner products are \(c_1=2\) and \(c_2=3\).
At \(\lambda=0.2\), the normalized scores and curvature factors are \(z_1=2/3\), \(q_1=2/3\), \(z_2=1\), and \(q_2=1\). Therefore,
The unequal column lengths matter: it would be incorrect to use \(S(z_j,\lambda)\) directly for both coefficients. That simpler expression applies when the column squared lengths equal \(n\). Scaling predictors before penalization, as in the earlier material on predictor scaling, makes coefficient penalties more comparable, but the formula must still use the actual design normalization.
What the Formula Shows—and When It Applies
For a fixed coordinate, the solution is piecewise linear in \(\lambda\). If \(z_j>0\), its estimate is \( (z_j-\lambda)/q_j \) while \(0\leq\lambda<z_j\), then zero for \(\lambda\geq z_j\). If \(z_j<0\), its estimate is \( (z_j+\lambda)/q_j \) while \(0\leq\lambda<|z_j|\), then zero for \(\lambda\geq|z_j|\). Thus, for each coefficient the magnitude is nonincreasing as the penalty grows, and its sign cannot flip under this orthogonal design.
The simplification depends on more than the absolute-value penalty: it depends on the cross-products \(x_j^\mathsf{T}x_k\) being zero. With correlated columns, the squared-error expansion contains terms involving \(\beta_j\beta_k\), so minimizing one coordinate while holding the others fixed does not yield the full solution by thresholding each unpenalized score independently. The lasso KKT conditions still apply, but the coefficients are coupled. Coordinatewise soft thresholding is then an update used within an algorithm, not generally the final solution from one pass.
There is also a fundamental high-dimensional limitation. Nonzero mutually orthogonal vectors in \(\mathbb{R}^n\) can number at most \(n\). Therefore, if \(p>n\), all \(p\) predictor columns cannot be exactly orthogonal. The formula remains useful for understanding an idealized design or a subset of orthogonal predictors, but it cannot be assumed to give the exact full lasso fit in a \(p>n\) data set.
Diagnostics and Common Pitfalls
- Assuming predictors are orthogonal because they were centered. Centering makes column means zero; it does not make pairwise inner products zero. Check the cross-products or correlations before using the closed-form solution.
- Ignoring column length. The formula \(S(x_j^\mathsf{T}y/n,\lambda)\) assumes squared column length \(n\). With unequal lengths, use \(S(x_j^\mathsf{T}y/n,\lambda)/(x_j^\mathsf{T}x_j/n)\).
- Interpreting a coefficient of zero as evidence of no biological effect. It means the penalized fit assigns zero under this design, response, scaling, and penalty. It does not establish that a gene or other predictor has no biological role.
- Applying the exact result to correlated predictors. If cross-products are nonzero, independently thresholding the scores generally does not minimize the full objective. Fit the joint penalized problem instead.
- Expecting exact orthogonality when \(p>n\). There cannot be more than \(n\) nonzero orthogonal columns in \(n\)-dimensional observation space. Treat the result as a special case, not a general high-dimensional shortcut.
Check Your Understanding
Answer each question using the orthogonal-design formula. Full solutions follow the questions.
- For normalized orthogonal columns, a predictor has score \(z_j=-1.6\). What is its lasso coefficient at \(\lambda=0.5\)?
- For a normalized orthogonal predictor with score \(0.9\), at what penalty does its coefficient first become zero?
- A column has \(x_j^\mathsf{T}x_j/n=2\) and \(x_j^\mathsf{T}y/n=1.5\). Find its coefficient at \(\lambda=0.5\), assuming orthogonality to the other columns.
- Why is independent soft thresholding not generally the exact solution when two predictor columns are correlated?
- Can a design with \(n=8\) observations have 10 nonzero mutually orthogonal predictor columns? Explain.
Solutions. (1) \(S(-1.6,0.5)=-(1.6-0.5)=-1.1\). (2) The coefficient becomes zero when \(\lambda\geq|z_j|\), so the threshold is \(0.9\). (3) Here \(z_j=1.5\) and \(q_j=2\), so \(\hat{\beta}_j=S(1.5,0.5)/2=(1.5-0.5)/2=0.5\). (4) Correlation creates cross-products in the residual sum of squares, coupling the coefficients; separate one-coordinate minimizations do not generally minimize the joint objective. (5) No. In \(\mathbb{R}^8\), at most eight nonzero vectors can be mutually orthogonal.