The Lasso Coefficient Path

Why Follow a Lasso Coefficient Path?

Lasso does more than produce one set of regression coefficients. Its penalty parameter controls how strongly coefficients are shrunk, so changing the penalty can change both their sizes and which predictors are included. A coefficient path records each coefficient as a function of the penalty. It helps identify the penalty values at which predictors enter the model and shows how their estimates change afterward.

This tutorial studies the path under an orthogonal-predictor assumption. That special structure makes the lasso solution separable across predictors, so the path can be derived exactly. The examples are hypothetical regression calculations, not empirical claims. No interest, mortality, or response-error distribution assumption is needed to calculate this optimization path; the stated design and objective are the assumptions used here.

Definition: Let $y$ be a centered response vector and let $X$ be a matrix of centered predictor columns satisfying $X^\mathsf{T}X=I$, where $I$ is the identity matrix. Thus each column has unit length and distinct columns are orthogonal. With an unpenalized intercept (which is zero after centering), define the lasso estimate to minimize $$ \frac{1}{2}\lVert y-X\beta\rVert_2^2+\lambda\sum_{j=1}^{p}|\beta_j|,\qquad \lambda\geq 0. $$ Here $\lambda$ is the penalty strength, $p$ is the number of predictors, and $z_j=X_j^\mathsf{T}y$ is the unpenalized least-squares coefficient for predictor $j$ under this unit-length design.

The normalization matters. With unit-length columns and the objective above, the threshold is $|z_j|$. If a different objective scaling or predictor scaling is used, the numerical value of the penalty and its threshold change accordingly. Do not transfer a threshold from one convention to another without checking the objective.

Deriving the Orthogonal Lasso Path

Since $X^\mathsf{T}X=I$, expanding the squared-error term gives

\frac{1}{2}\lVert y-X\beta\rVert_2^2 =\frac{1}{2}\lVert y\rVert_2^2-\sum_{j=1}^{p}z_j\beta_j +\frac{1}{2}\sum_{j=1}^{p}\beta_j^2.

The first term does not depend on $\beta$. The remaining objective is a sum of separate one-variable objectives, one for each predictor:

\sum_{j=1}^{p}\left(\frac{1}{2}\beta_j^2-z_j\beta_j+\lambda|\beta_j|\right).

To minimize the contribution for a particular $j$, consider positive, negative, and zero values of $\beta_j$. If $\beta_j>0$, differentiating gives $\beta_j-z_j+\lambda=0$, so a positive solution is possible when $z_j>\lambda$, and then $\beta_j=z_j-\lambda$. If $\beta_j<0$, the derivative is $\beta_j-z_j-\lambda=0$, so a negative solution is possible when $z_j<-\lambda$, and then $\beta_j=z_j+\lambda$. At zero, the absolute-value penalty has subgradients from $-\lambda$ to $\lambda$; zero minimizes the objective exactly when $|z_j|\leq\lambda$. Combining these cases yields the soft-threshold formula.

Theorem: Under the centered, orthogonal-design setup and the stated objective, the lasso coefficient path is $$ \hat{\beta}_j(\lambda) =\operatorname{sign}(z_j)(|z_j|-\lambda)_+,\qquad (a)_+=\max(a,0). $$ If $|z_j|>0$, predictor $j$ is zero at $\lambda=|z_j|$ and is active for $0\leq\lambda<|z_j|$. If $z_j=0$, it remains zero for every allowed $\lambda$ and never becomes active.

A knot is a penalty value at which the path changes its piecewise-linear expression. For an orthogonal design, the possible entry knots are the positive values $|z_j|$. As $\lambda$ decreases through a predictor’s knot, its coefficient begins to move away from zero in the sign of $z_j$. If multiple predictors have the same positive $|z_j|$, they enter at the same knot. At the knot itself, each tied coefficient is still zero.

Between knots, the active set is fixed and every active coefficient is linear in $\lambda$. As the penalty decreases, active coefficients grow in magnitude without changing sign. Consequently, under these assumptions a nonzero-correlation predictor enters and remains active; it does not leave later. This nested active-set behavior is a property of the orthogonal setup, not a general promise for correlated predictors.

Worked Example: Locating Entry Knots and Reading the Path

A hypothetical standardized regression has four orthonormal predictors and unpenalized coefficients $z=(3,-2,0.5,0)$. Find the entry knots, the coefficient vector at $\lambda=2.5$, the active set at $\lambda=1$, and the endpoint at $\lambda=0$. Use the stated unit-length design and objective; no probability distribution is needed. The method is to apply soft thresholding component by component. A common mistake to avoid is declaring a coefficient active at its knot.

Step 1: Find thresholds. The absolute correlations are $(3,2,0.5,0)$. The positive entry knots are therefore $3$, $2$, and $0.5$. The fourth predictor has $z_4=0$, so it has no positive entry knot and never becomes active.

Step 2: Evaluate at $\lambda=2.5$. For predictor 1, $\operatorname{sign}(3)(3-2.5)_+=0.5$. For predictor 2, $\operatorname{sign}(-2)(2-2.5)_+=0$. For predictor 3, $\operatorname{sign}(0.5)(0.5-2.5)_+=0$, and predictor 4 remains zero. Thus $\hat{\beta}(2.5)=(0.5,0,0,0)$; only predictor 1 is active.

Step 3: Evaluate at $\lambda=1$. The coefficients are $3-1=2$, $-\,(2-1)=-1$, $0$ because $0.5-1<0$, and $0$. Thus $\hat{\beta}(1)=(2,-1,0,0)$, with predictors 1 and 2 active.

Step 4: Check the endpoint. At $\lambda=0$, soft thresholding returns $z$, so $\hat{\beta}(0)=(3,-2,0.5,0)$, the orthogonal-design least-squares solution. At the knot $\lambda=2$, predictor 2 is still zero; it becomes active only when $\lambda<2$. This verifies both the threshold convention and the endpoint.

Worked Paths, Ties, and Zero Correlations

The coefficient path contains two related but distinct kinds of information: coefficient values at a chosen penalty and the order in which predictors become active as the penalty is reduced. The next examples illustrate why an active-set statement should always be checked against the actual threshold.

Worked Example: Finding the Active Set at a Given Penalty

In a hypothetical orthogonal model for a portfolio’s monthly loss index, the predictor correlations are $z=(2.4,-1.6,0.8)$. At $\lambda=1$, calculate the coefficients and identify the next entry knot as $\lambda$ decreases. The model uses the same unit-length columns and objective defined above; no distributional assumption is used. The method is direct soft thresholding. The common mistake to avoid is treating every nonzero $z_j$ as active at every positive penalty.

Predictor 1: $z_1=2.4>0$, so $\hat{\beta}_1(1)=2.4-1=1.4$.

Predictor 2: $z_2=-1.6$, so $\hat{\beta}_2(1)=-\,(1.6-1)=-0.6$.

Predictor 3: $|z_3|=0.8<1$, so $(0.8-1)_+=0$ and $\hat{\beta}_3(1)=0$.

Therefore $\hat{\beta}(1)=(1.4,-0.6,0)$, and the active set is predictors 1 and 2 only. Their entry knots are $2.4$ and $1.6$, respectively. The next entry as the penalty decreases from 1 is predictor 3 at $\lambda=0.8$; it is zero at $0.8$ and becomes active only below $0.8$. As a check, each nonzero coefficient has the sign of its corresponding $z_j$, and each inactive predictor has $|z_j|\leq\lambda$.

Worked Example: Simultaneous Entry at a Tied Knot

A hypothetical pricing regression has orthogonal predictors with $z=(1.5,-1.5,0.4)$. Determine the tied entry event, the coefficients at $\lambda=1.2$, and the active set and coefficients at $\lambda=0.2$. Use the same objective and design assumptions. The method is to sort the absolute correlations and then apply the formula at each requested penalty. The mistake to avoid is breaking a tie arbitrarily or making a tied coefficient nonzero exactly at the knot.

Step 1: Identify the knots. The absolute correlations are $1.5,1.5,0.4$. Predictors 1 and 2 share the first knot, $\lambda=1.5$; predictor 3 has the later knot $\lambda=0.4$. At $\lambda=1.5$, both tied coefficients are zero.

Step 2: Calculate at $\lambda=1.2$. Predictor 1 has coefficient $1.5-1.2=0.3$. Predictor 2 has coefficient $-\,(1.5-1.2)=-0.3$. Predictor 3 is zero because $0.4<1.2$. Thus $\hat{\beta}(1.2)=(0.3,-0.3,0)$, with predictors 1 and 2 active.

Step 3: Calculate at $\lambda=0.2$. The coefficients are $1.5-0.2=1.3$, $-\,(1.5-0.2)=-1.3$, and $0.4-0.2=0.2$. Hence $\hat{\beta}(0.2)=(1.3,-1.3,0.2)$, and all three predictors are active. Each coefficient has the expected sign, and the two tied predictors entered together below their common knot.

How to Interpret the Path Without Overstating It

A path plot usually places $\lambda$ (or an equivalent measure of penalty strength) along one axis and coefficient values along the other. With the convention used here, larger $\lambda$ means stronger shrinkage. For orthogonal predictors, each path is zero until its threshold is crossed, then is a straight line whose sign is fixed. The maximum entry knot, $\max_j|z_j|$, is the point above which every coefficient is zero. If all $z_j=0$, every coefficient is zero for all $\lambda\geq0$.

The path is useful for describing what the penalty does, but it does not by itself select an optimal penalty. A prediction or model-selection criterion is needed for that decision, and the coefficient path alone does not establish that a selected predictor has a causal effect. In particular, the simple threshold ordering comes from orthogonality: the coefficient for one predictor does not have to adjust to account for another predictor’s contribution.

When reading a path, keep the following distinctions clear:

At a positive knot: the coefficient is still zero.
Below a positive knot: the coefficient is active, with sign matching $z_j$.
At zero correlation: the coefficient remains zero, including at $\lambda=0$.
At tied positive knots: all predictors with that threshold enter together as $\lambda$ moves below it.
For this orthogonal path: active predictors do not later leave, but do not assume this behavior beyond the stated setup.

Key takeaway: Under the unit-length orthogonal design and objective stated here, each predictor has its own threshold $|z_j|$. A predictor with $|z_j|>0$ is zero at that threshold and active only below it; a predictor with $z_j=0$ never enters. The path is piecewise linear, and tied thresholds produce simultaneous entry.

Keeping the threshold condition and the exact penalty convention in view is especially important for short calculations. An answer that lists the right coefficient vector but labels a zero coefficient as active is still internally inconsistent. These path calculations are aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.

Check Your Understanding

For each problem, use centered responses, orthonormal unit-length predictor columns, an unpenalized intercept, and the objective defined above. No response-error distribution is required.

A hypothetical model has $z=(2,-1,0)$. At $\lambda=1$, give the coefficient vector and active set. (Answer: $(1,0,0)$; predictor 1 only.)
A model has $z=(-2.2,1.1,0.6)$. Give the positive entry knots in descending order, and state the coefficient vector at $\lambda=0.8$. (Answer: Knots $2.2,1.1,0.6$; coefficients $(-1.4,0.3,0)$.)
For $z=(1.3,-1.3,0.2)$, state which predictors enter at the first knot and give the coefficient vector at $\lambda=1.0$. (Answer: Predictors 1 and 2 enter together at $1.3$; coefficients $(0.3,-0.3,0)$.)
For $z=(0,-0.7,0.7)$, find the active set at $\lambda=0.7$, then at $\lambda=0.5$. (Answer: At $0.7$, none are active; at $0.5$, predictors 2 and 3 are active.)
A model has $z=(1.8,-1.0,0.4)$. At $\lambda=0.6$, calculate all coefficients and identify the next entry knot if the penalty is reduced further. (Answer: Coefficients $(1.2,-0.4,0)$; predictor 3 enters at $\lambda=0.4$, remaining zero at the knot itself.)

What You'll Learn