Reading Ridge and Lasso Solution Paths

What a Coefficient Path Shows

A penalized regression fit is not just one set of coefficients. It is a collection of fits, each made with a different penalty strength. A coefficient path plots each predictor’s fitted coefficient against the penalty parameter $\lambda$. Following a curve shows how that coefficient changes as the fitting procedure places more or less emphasis on keeping coefficients small.

Paths help describe how a model responds to penalization: which coefficients shrink steadily, which lasso coefficients become exactly zero, and whether signs or relative magnitudes change. A path does not, by itself, identify the best value of $\lambda$. Choosing a penalty for predictive performance is a separate task.

Model and convention: For observations $i=1,\ldots,n$, write $y_i=\beta_0+\sum_{j=1}^{p}x_{ij}\beta_j+\epsilon_i$. The intercept $\beta_0$ is not penalized. Assume the predictors have been centered and scaled as needed before penalization, as in the earlier material on scaling predictors. For a consistent illustration, use the objectives $$ \text{Ridge: }\frac{1}{2n}\sum_{i=1}^{n}(y_i-\beta_0-\sum_{j=1}^{p}x_{ij}\beta_j)^2+\frac{\lambda}{2}\sum_{j=1}^{p}\beta_j^2, \qquad \text{Lasso: }\frac{1}{2n}\sum_{i=1}^{n}(y_i-\beta_0-\sum_{j=1}^{p}x_{ij}\beta_j)^2+\lambda\sum_{j=1}^{p}|\beta_j|. $$ Here $\lambda\geq 0$. These objectives specify the penalty convention used below; software may use a different scaling. The fitted paths are deterministic functions of the data and this convention. No normality assumption is needed just to compute or read them.

A common plot puts coefficient values on the vertical axis and $\lambda$, often on a logarithmic scale, on the horizontal axis. The path is frequently drawn from strong penalty on the left to weak penalty on the right, so $\lambda$ decreases as the reader moves right. Other plots reverse the direction or label the horizontal axis with a transformed quantity such as $1/\lambda$. Always inspect the axis label and tick marks before describing which direction means stronger penalization.

Reading the Weak- and Strong-Penalty Ends

At $\lambda=0$, neither objective penalizes the slopes, so both reduce to least squares. If the design matrix has full column rank, the least-squares coefficient vector is unique and is the weak-penalty limit for both paths. If least squares has multiple solutions, a limit need not be described as one unique ordinary least-squares vector; the details depend on the data and penalty.

As $\lambda$ becomes large, both methods increasingly favor small slopes. Ridge coefficients approach zero continuously as $\lambda$ grows, but a ridge coefficient is generally not exactly zero at any finite $\lambda$. Lasso can set one or more coefficients exactly to zero at finite penalty values. With centered response and centered predictors, the intercept is zero in the centered representation; in the original response scale, the unpenalized intercept supplies the fitted mean when all slopes are zero.

Key takeaway: A strong penalty shrinks slopes toward zero, not necessarily the intercept. Ridge paths approach zero gradually; lasso paths can reach zero exactly and remain there over a range of penalty values.

Worked Example: Following Paths for Orthogonal Predictors

Suppose three centered, scaled predictor columns are orthogonal, with $X^\mathsf{T}X/n=I$, and their correlations with the centered response are $z=(3,\ 1.5,\ -0.5)$. For ridge under the stated objective, the coefficient for predictor $j$ is $z_j/(1+\lambda)$. For lasso, orthogonality gives the soft-threshold formula $\operatorname{sign}(z_j)(|z_j|-\lambda)_+$, where $a_+=\max(a,0)$.

At $\lambda=1$, the ridge coefficients are $$ (3/2,\ 1.5/2,\ -0.5/2)=(1.5,\ 0.75,\ -0.25). $$ At $\lambda=5$, they are $(3/6,\ 1.5/6,\ -0.5/6)=(0.5,\ 0.25,\ -0.0833)$. They have become smaller, but none is exactly zero.

For lasso at $\lambda=1$, the coefficients are $(3-1,\ 1.5-1,\ -(0.5-1)_+)=(2,\ 0.5,\ 0)$. At $\lambda=2$, they are $(1,\ 0,\ 0)$; at $\lambda=3$, all three are zero. Thus predictor 3 reaches zero first, predictor 2 reaches zero at $\lambda=1.5$, and predictor 1 reaches zero at $\lambda=3$. This illustrates how exact zeros create visible endpoints in lasso paths.

The numerical $\lambda$ values should not be compared as though ridge and lasso penalties had identical units or scales. Their penalty terms differ. The useful comparison here is the path behavior under each stated objective.

Why the Path Shapes Differ

Ridge uses a squared-coefficient penalty, which changes the least-squares solution by shrinking its directions. Lasso uses the sum of absolute coefficients; its geometry permits fitted slopes to land exactly at zero. Earlier material on ridge shrinkage and lasso optimality conditions explains these mechanisms. For reading paths, the practical consequence is the central visual contrast: ridge usually has smooth, nonzero curves, while lasso curves can meet the horizontal zero line and stay there.

The behavior at the all-zero lasso solution has a precise threshold. If the predictors and response are centered, the predictors are scaled so that $x_j^\mathsf{T}x_j/n=1$, and $\beta_0$ is unpenalized, then all lasso slopes are zero whenever $$ \lambda\geq\lambda_{\max}=\max_j\left|\frac{x_j^\mathsf{T}y}{n}\right|. $$ To see why, at zero slopes the derivative of the squared-error part with respect to $\beta_j$ is $-x_j^\mathsf{T}y/n$. The lasso penalty can balance this derivative at zero exactly when its subgradient, the interval $[-\lambda,\lambda]$, contains $x_j^\mathsf{T}y/n$ for every $j$. That condition is precisely $\lambda\geq\max_j|x_j^\mathsf{T}y/n|$. This threshold gives the leftmost all-zero end of a lasso path under these conventions.

Ridge has no analogous finite threshold that sets every slope to zero. Its normal equations, with centered variables, are $$ (X^\mathsf{T}X/n+\lambda I)\hat{\beta}_{\text{ridge}}=X^\mathsf{T}y/n. $$ As $\lambda$ grows, the added $\lambda I$ dominates the left side, so the solution approaches zero. This explains why a ridge path can look very close to zero at a large plotted penalty even though its coefficients are not exactly zero.

Worked Example: Interpreting a Lasso Path with Correlated Predictors

Consider a hypothetical lasso path for three standardized predictors A, B, and C. The values below are illustrative fitted coefficients read at selected penalties; the intercept is not shown.

$\lambda$	A	B	C
0	1.60	1.10	-0.42
0.50	1.03	0.62	0
1.00	0.58	0.15	0
1.50	0.10	0	0
2.00	0	0	0

At $\lambda=0.50$, the fitted slopes for A and B are positive and the slope for C is exactly zero. The model therefore has two nonzero slopes at that point. By $\lambda=1.50$, only A remains nonzero, at $0.10$; by $\lambda=2.00$, all three slopes are zero. The path indicates that C is removed before B, and B before A, as the penalty strengthens.

The changes in coefficient values are not percentage changes in a predictor’s importance, and a zero on the path is a property of this penalized fit at this $\lambda$, not evidence that the predictor has no relationship with the response in every model. Also, when predictors are correlated, coefficient paths can reflect how the predictors share information. The path alone does not settle which of several related predictors is scientifically preferable.

What to Check When Reading a Plot

Use the plotted axes and coefficient scale as part of the interpretation. A coefficient measured per standardized predictor unit is not on the same scale as one measured per original unit. If predictors were standardized before fitting, a path usually reports coefficients for those standardized predictors; check whether the response was also scaled. When the horizontal axis is $\log(\lambda)$, equal horizontal distances represent multiplicative rather than additive changes in $\lambda$.

A path also summarizes coefficient estimates, not uncertainty intervals or causal effects. It does not show that a predictor causes a change in the response, and it does not by itself indicate how well the fitted model will predict new observations. Those questions require other analyses. For the path itself, focus on the direction and scale of the axes, the location of zero crossings, and changes in sign or relative coefficient size.

Worked Example: Avoiding an Axis-Direction Error

Suppose a plotted lasso path has $\log(\lambda)$ on the horizontal axis, with tick marks from $2$ at the left to $-4$ at the right. A predictor’s coefficient is $0$ at the left, $0.35$ at the middle tick $-1$, and $0.90$ at the right.

Because $\lambda=\exp(\log(\lambda))$, the left side has stronger penalization: $\exp(2)\approx7.389$, while the right side has weaker penalization: $\exp(-4)\approx0.01832$. Moving right therefore means decreasing $\lambda$, not increasing it. The coefficient grows from $0$ to $0.90$ as the penalty weakens, which is consistent with less shrinkage.

A reader who assumes every path is drawn with increasing $\lambda$ from left to right would reverse the interpretation. The correction is simple: read the axis values first, then describe the coefficient changes.

Common Mistakes

Calling a small ridge coefficient zero. A plotted value may look indistinguishable from zero at the graph’s resolution. Ridge generally approaches zero without reaching it at a finite penalty.
Assuming paths always run from weak to strong penalties left to right. Check whether the axis shows $\lambda$, $\log(\lambda)$, or another quantity and note its tick values.
Reading a lasso zero as proof of no association. It means the penalized fit sets that coefficient to zero at the displayed penalty, conditional on the other predictors and the fitting setup.
Comparing ridge and lasso at the same numerical $\lambda$ as if the penalties were equivalent. The penalty terms differ, so interpret each path using its stated convention.
Using the path alone to choose $\lambda$. A path describes coefficient behavior; it does not determine which fit best meets a prediction or analysis objective.

Key takeaway: Read a solution path from its axes and scaling, then follow each coefficient as penalization changes. Ridge usually shrinks slopes gradually toward zero; lasso can set slopes exactly to zero at finite penalties. Neither path alone selects the most useful model.

Check Your Understanding

Use the definitions and path-reading principles above to answer these questions.

Under what condition does a ridge path reduce to the ordinary least-squares solution at $\lambda=0$?
Why can a lasso path show an exact zero coefficient at a finite value of $\lambda$, while ridge generally does not?
With centered, scaled predictors, what quantity determines the threshold $\lambda_{\max}$ above which all lasso slopes are zero?
A plot uses $\log(\lambda)$ and its tick values decrease from left to right. Which direction corresponds to stronger penalization?
Why should the same numerical value of $\lambda$ not automatically be treated as an equivalent penalty for ridge and lasso?