Why Follow a Lasso Path?
A lasso fit is not just a single set of coefficients. It is a family of fits indexed by the penalty strength \(\lambda\). At large \(\lambda\), shrinkage can leave every coefficient at zero; as \(\lambda\) decreases, predictors may become active, change magnitude, or leave the active set. Following this path helps explain how the fitted model changes, rather than treating variable selection as a one-time decision.
The central feature is that, under suitable conditions, each coefficient is a linear function of \(\lambda\) between particular event points. A path is therefore piecewise linear: its coefficients change continuously along a segment, but the slope can change at a kink when the active predictors or their signs change.
Objective, Assumptions, and KKT Conditions
Use a centered response vector \(y\) and a centered design matrix \(X\), with each predictor column scaled so that \(x_j^\mathsf{T}x_j/n=1\). The intercept is unpenalized and is omitted from the notation because the variables are centered. Consider the lasso objective
Here, \(\lambda\) controls the penalty, and larger values favor more shrinkage. Define the residual \(r=y-X\hat{\beta}(\lambda)\), the sample predictor-response correlations \(c=X^\mathsf{T}y/n\), and the predictor Gram matrix \(G=X^\mathsf{T}X/n\). The active set is \(A=\{j:\hat{\beta}_j(\lambda)\ne 0\}\). For \(j\in A\), let \(s_j=\operatorname{sign}(\hat{\beta}_j)\).
These conditions hold for a minimizer of the stated convex objective. To obtain the simple path formula below, assume that on the segment being considered the active columns are linearly independent, so \(G_{AA}\) is invertible, and that the active set and coefficient signs do not change within the segment. A general-position design typically avoids ambiguous ties and yields a well-defined path; without suitable uniqueness conditions, a fitted coefficient path may not be unique. The essential argument applies to each interval where the active set is fixed and its Gram matrix is invertible.
At the all-zero fit, the inactive KKT inequalities require \(|x_j^\mathsf{T}y/n|\leq\lambda\) for every \(j\). Thus the null fit is optimal for \(\lambda\geq\lambda_{\max}\), where \(\lambda_{\max}=\max_j|c_j|\). This gives the starting point for tracing the path downwards.
Why Each Segment Is Linear
Suppose that, over some interval of \(\lambda\), the active set \(A\) and sign vector \(s_A\) stay fixed. The active KKT equalities can be written as \(X_A^\mathsf{T}(y-X_A\hat{\beta}_A)/n=\lambda s_A\). Rearranging gives
This is an affine function of \(\lambda\), while all inactive coefficients remain zero. Its slope on the segment is \(-G_{AA}^{-1}s_A\). This is the key justification for piecewise linearity: when the active set and signs are fixed, the KKT equations are linear in \(\lambda\), so solving them gives coefficients that are linear in \(\lambda\).
The formula is valid only while it satisfies the conditions that defined the segment. An active coefficient cannot cross zero while keeping its original sign. An inactive predictor must continue to satisfy its KKT inequality. The segment ends at the first value of \(\lambda\) where either condition becomes an equality. Such an event changes the active set or signs and can change the slope.
Worked Example: A Predictor Enters the Path
Consider a hypothetical standardized expression dataset with two gene predictors. Suppose their Gram matrix and predictor-response correlations are \(G=\begin{pmatrix}1&0.5\\0.5&1\end{pmatrix}\) and \(c=(2,1.4)^\mathsf{T}\). The matrix is positive definite, since its determinant is \(1-0.5^2=0.75>0\). The largest absolute correlation is \(2\), so the zero fit is optimal for \(\lambda\geq2\).
Just below \(\lambda=2\), gene 1 is active with positive sign, while gene 2 is inactive. The one-predictor KKT equation gives \(\hat{\beta}_1=2-\lambda\). Gene 2’s residual correlation on this segment is \(c_2-G_{21}\hat{\beta}_1=1.4-0.5(2-\lambda)=0.4+0.5\lambda\). For gene 2 to remain inactive, this must be no greater than \(\lambda\), and it is positive on the segment. The boundary is therefore found by solving \(0.4+0.5\lambda=\lambda\), which gives \(\lambda=0.8\). Gene 2 enters as \(\lambda\) decreases through \(0.8\).
Below that entry point, both genes have positive coefficients. Solving the active KKT equations gives \(\hat{\beta}_A=G^{-1}(c-\lambda(1,1)^\mathsf{T})\), so \(\hat{\beta}_1=(1.3-0.5\lambda)/0.75\) and \(\hat{\beta}_2=(0.4-0.5\lambda)/0.75\). At \(\lambda=0.8\), these are \(1.2\) and \(0\), respectively, matching the entry event. At \(\lambda=0.5\), they are \(1.4\) and \(0.2\). Both are positive, as assumed. The coefficients meet continuously at the kink, but their slopes change when gene 2 becomes active.
Entry Events, Drop-Outs, and Kinks
An entry event occurs when an inactive predictor’s absolute residual correlation reaches \(\lambda\). For a fixed active set, the inactive residual correlation is \(x_j^\mathsf{T}r/n=c_j-G_{jA}\hat{\beta}_A(\lambda)\). Because \(\hat{\beta}_A(\lambda)\) is affine on the segment, this correlation is also affine in \(\lambda\). An entry candidate is found by solving \(x_j^\mathsf{T}r/n=\lambda\) or \(x_j^\mathsf{T}r/n=-\lambda\), then checking that the resulting value lies on the current segment and that no earlier event intervenes.
A drop-out event occurs when an active coefficient reaches zero. Using the segment formula, its candidate value solves \((G_{AA}^{-1}c_A)_j-\lambda(G_{AA}^{-1}s_A)_j=0\) for some \(j\in A\). A drop-out may change the active set even though \(\lambda\) is decreasing. Thus the number of active predictors need not increase monotonically along the path: a predictor may enter and later leave, or enter again after a later change in the fit.
At an event, a coefficient path is generally continuous when the solution is unique, but its slope can change. The term “kink” refers to this change in slope, not to a jump in the coefficient. If two predictors reach their entry boundaries together, or a drop-out and entry coincide, several KKT equalities change status at once. Such ties require care when tracing a path numerically.
Worked Example: Checking a Candidate Entry Point
Return to the hypothetical two-gene design above. At \(\lambda=1\), gene 1 is active with \(\hat{\beta}_1=2-1=1\), and gene 2 is inactive. Its residual correlation is \(1.4-0.5(1)=0.9\). The inactive KKT condition is satisfied because \(|0.9|\leq1\), so the proposed active set is valid at this penalty value.
At \(\lambda=0.8\), the residual correlation becomes \(1.4-0.5(1.2)=0.8\), exactly the KKT boundary. This checks the calculated entry point. At \(\lambda=0.7\), continuing the one-gene formula would give gene 2 a residual correlation of \(0.4+0.5(0.7)=0.75\), which exceeds \(\lambda=0.7\). The inactive KKT condition fails, so the one-gene segment cannot extend to \(0.7\). Gene 2 must be included in the active set.
Worked Example: Reading the Slope Change
For the same hypothetical design, between \(\lambda=2\) and \(\lambda=0.8\), gene 1 follows \(\hat{\beta}_1=2-\lambda\). Its slope with respect to \(\lambda\) is \(-1\); gene 2 remains zero. Below \(0.8\), the two-active-predictor formulas give slopes \(-0.5/0.75=-0.6667\) for both genes.
For example, decreasing \(\lambda\) from \(0.8\) to \(0.5\) changes it by \(-0.3\). The predicted coefficient changes on the two-active segment are \((-0.6667)(-0.3)=0.2000\) for each gene. Starting from \((1.2,0)\) at the kink gives \((1.4,0.2)\) at \(\lambda=0.5\), as calculated earlier. The first gene’s slope changes from \(-1\) to \(-0.6667\), and the second changes from zero to \(-0.6667\). These different slopes are the numerical signature of the kink.
Diagnostics, Alternatives, and Limitations
A proposed segment should be checked rather than extended by visual intuition. Verify the active KKT equalities, the inactive KKT inequalities, and the assumed signs at representative penalty values. Calculate the next candidate entry and drop-out values, and use the first valid event encountered as the segment endpoint. After updating the active set or signs, solve the new active equations and repeat.
Common warning signs include an inactive residual correlation exceeding \(\lambda\), an active coefficient having the wrong sign, or a reported coefficient discontinuity where the fitted solution is expected to be unique. These can indicate that the segment was extended beyond an event, a sign was recorded incorrectly, variables were not scaled as assumed, or a numerical algorithm has not converged adequately. In high-dimensional designs, near-collinearity can make calculations unstable; if the active Gram matrix is singular, the inverse-based segment formula does not apply directly. The existence of nonunique coefficients under collinearity requires separate treatment.
An exact path-tracing approach identifies event values and follows the linear segments between them. In contrast, a numerical fitting routine may solve the optimization problem on a chosen grid of penalty values. A grid is convenient for fitting and prediction comparisons, but it can miss a kink between grid points and may not reveal the exact entry value. Neither a path plot nor an event establishes that a gene is biologically important: entry is a property of this penalized fit, conditional on the response, design, scaling, and penalty.
The piecewise-linear result depends on the squared-error lasso objective and the fixed-design assumptions above. It does not mean that a path has few segments; in some designs, the number can be very large. Nor does it guarantee that every entry is a permanent addition or that the selected predictors are stable under small changes to the data. Interpret the path as a structured description of how this optimization problem’s solution changes with \(\lambda\), not as an inferential test or a causal conclusion.
A frequent mistake is to say, “As the penalty decreases, predictors only enter, so the active set can only grow.” The correction is that inactive predictors can enter at residual-correlation boundaries, but active coefficients can also reach zero and drop out. The active-set size is therefore not necessarily monotone.
Compute \(\lambda_{\max}=\max_j|c_j|\); the zero fit is optimal at and above this value.
For fixed active predictors and signs, use \(\hat{\beta}_A=G_{AA}^{-1}(c_A-\lambda s_A)\).
Check when an inactive KKT inequality becomes equality or an active coefficient reaches zero.
Change the active set or signs at the event, derive the next segment, and recheck all KKT conditions.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- For the objective in this tutorial, what condition must every inactive predictor satisfy at a given \(\lambda\)?
- If \(G_{AA}^{-1}c_A=(2,1)^\mathsf{T}\) and \(G_{AA}^{-1}s_A=(0.5,0.25)^\mathsf{T}\), at what penalty values do the two active coefficients first reach zero when following this segment toward smaller \(\lambda\)? Assume both coefficients and signs are valid down to those points.
- In the two-gene worked example, why does the one-gene segment fail at \(\lambda=0.7\)?
- Does a kink necessarily mean that a coefficient jumps? Explain.
Solutions. 1. Each inactive predictor must satisfy \(|x_j^\mathsf{T}r/n|\leq\lambda\); equality marks a possible entry boundary. 2. The coefficients are \(2-0.5\lambda\) and \(1-0.25\lambda\). Setting them to zero gives \(\lambda=4\) and \(\lambda=4\), respectively. Under the stated assumption, both reach zero simultaneously at \(\lambda=4\). 3. At \(0.7\), gene 2’s residual correlation on the one-gene segment is \(0.4+0.5(0.7)=0.75\), and \(0.75>0.7\), violating its inactive KKT condition. 4. No. With a unique solution, coefficients are generally continuous at an event; the slope changes because the active set or signs change, creating the kink.