What a Ridge Coefficient Path Shows
A ridge coefficient path displays fitted predictor coefficients over a sequence of penalty values, \(\lambda\). It makes shrinkage visible: as \(\lambda\) increases, the ridge penalty becomes stronger and the fitted coefficient vector generally becomes shorter. A path is useful for seeing the joint movement of coefficients, not for declaring that every coefficient must move steadily toward zero.
The distinction matters when predictors are correlated. Ridge fitting couples their coefficients, so an individual coefficient can change direction or even cross zero as \(\lambda\) changes. The overall coefficient vector has a cleaner property: for a fixed design and response, its Euclidean length is nonincreasing as positive \(\lambda\) increases. That statement concerns the vector as a whole, not each coordinate separately.
In R, the glmnet ridge model uses \(\alpha=0\). For a design matrix \(X\) with \(n\) rows, response vector \(y\), and predictor coefficient vector \(\beta\), its ridge objective can be written as
Here \(\beta_0\) is the intercept, which is not penalized. The penalty term makes large predictor coefficients costly. A larger \(\lambda\) therefore exerts more shrinkage, while a smaller positive \(\lambda\) allows the fitted coefficients to be closer to their unpenalized least-squares values. This objective and its normalization are the conventions used in this tutorial.
Creating a Path in R
A path is fitted over multiple values of \(\lambda\), rather than at only one penalty value. The following code uses an explicitly decreasing sequence of positive penalties. Assume that x is a numeric predictor matrix and y is a numeric response vector with one entry per row of x.
R commands: lam <- exp(seq(log(20), log(0.01), length.out = 100)); fit <- glmnet::glmnet(x, y, alpha = 0, lambda = lam, standardize = TRUE, intercept = TRUE); plot(fit, xvar = "lambda", label = TRUE).
This fits ridge regression because \(\alpha=0\), asks glmnet to standardize predictors internally, and includes an unpenalized intercept. The supplied \(\lambda\) values are positive and decreasing, as glmnet expects for a path. The plot labels each predictor’s coefficient path and uses lambda as the horizontal variable. Read the axis ticks to determine which end corresponds to larger \(\lambda\); do not infer that direction from left or right alone.
With predictor standardization enabled, glmnet fits using standardized predictors and reports coefficients on the original predictor scale. Consequently, the displayed vertical scales can differ substantially across predictors with different units. A coefficient measured in dollars per unit and one measured in dollars per year are not directly comparable in magnitude simply because they appear on the same graph. A path plot shows how the fitted model’s reported coefficients change; the meaning of each coefficient still depends on its predictor scale and the other predictors in the model.
Worked Example: Reading a One-Predictor Path
Suppose a centered, standardized predictor \(x\) satisfies \(x^\mathsf{T}x/n=1\), and \(x^\mathsf{T}y/n=2\). Compute its ridge coefficient at \(\lambda=0.25,\ 0.5,\ 1,\) and \(3\). This calculation illustrates the direction and amount of shrinkage when there is just one predictor.
Method. The ridge normal equation for this one-predictor case is \((1+\lambda)\hat{\beta} = 2\), so \(\hat{\beta}(\lambda)=2/(1+\lambda)\). The intercept is excluded because the variables are centered.
At \(\lambda=0.25\), \(\hat{\beta}=2/1.25=1.6\). At \(\lambda=0.5\), \(\hat{\beta}=2/1.5=1.3333\). At \(\lambda=1\), \(\hat{\beta}=2/2=1\). At \(\lambda=3\), \(\hat{\beta}=2/4=0.5\). Values are rounded to four decimal places only where needed.
Check. Each coefficient has the same positive sign as the unpenalized coefficient, and the coefficient decreases in magnitude as \(\lambda\) increases. A common mistake is to read a plotted coefficient value as the penalty value; they are different quantities, shown on different axes.
Why the Coefficient Vector Shrinks Overall
The usual ridge equations make precise what it means for shrinkage to increase. Center the predictors and response to separate the intercept, let \(A=X^\mathsf{T}X/n\), and let \(c=X^\mathsf{T}y/n\). For \(\lambda>0\), the ridge coefficient vector solves
The matrix \(A\) is positive semidefinite. By its spectral decomposition, it has orthonormal eigenvectors with eigenvalues \(d_j\geq 0\). Write \(q_j\) for the component of \(c\) in eigenvector direction \(j\). In that direction, the fitted coefficient has component \(q_j/(d_j+\lambda)\). Since the eigenvectors are orthonormal, the squared length of the coefficient vector is the sum of the squared components:
For every \(j\), \(d_j+\lambda\) is positive and increases with \(\lambda\). Thus each nonnegative term \(q_j^2/(d_j+\lambda)^2\) is nonincreasing as \(\lambda\) increases. Equivalently, differentiating for \(\lambda>0\) gives
This establishes nonincrease of the squared length, and therefore of the length, for positive \(\lambda\). It does not establish that every original predictor coefficient decreases in absolute value. Eigenvector directions are generally combinations of predictors, and a change in those directions can produce a nonmonotone path for a particular predictor.
Worked Example: Orthogonal Predictors and the Whole-Vector Length
Suppose two centered, standardized predictors are orthogonal, so \(X^\mathsf{T}X/n=I\), and \(X^\mathsf{T}y/n=(2,-1)^\mathsf{T}\). Find the two coefficients and the squared coefficient-vector length at \(\lambda=0,\ 1,\) and \(3\).
Method. Because the predictors are orthogonal, \(A=I\), and each coefficient is its corresponding component of \(c\) divided by \(1+\lambda\). Thus \(\hat{\beta}(\lambda)=(2/(1+\lambda),-1/(1+\lambda))^\mathsf{T}\).
At \(\lambda=0\), the coefficient vector is \((2,-1)^\mathsf{T}\), and its squared length is \(2^2+(-1)^2=5\). At \(\lambda=1\), it is \((1,-0.5)^\mathsf{T}\), with squared length \(1^2+(-0.5)^2=1.25\). At \(\lambda=3\), it is \((0.5,-0.25)^\mathsf{T}\), with squared length \(0.5^2+(-0.25)^2=0.3125\).
Check. Both coefficients and the vector length move toward zero in this orthogonal example. The zero-penalty fit is well-defined here because \(A=I\) is positive definite. The example illustrates a simple path, but it is not a rule that each coordinate must behave this way when predictors are correlated.
Correlated Predictors Can Produce Nonmonotone Paths
With correlated predictors, the ridge equations link the coefficients. The next example shows why an individual path can approach zero, cross zero, and then move away from zero even as the penalty continues to increase. Its overall coefficient-vector length still decreases.
Worked Example: A Correlated-Predictor Path That Crosses Zero
Suppose the centered predictor cross-product matrix divided by \(n\) and the response cross-product vector divided by \(n\) are
Compute both ridge coefficients at \(\lambda=0,\ 0.5,\ 2,\ 3,\) and \(4\), and interpret the first coefficient’s path.
Method. The eigenvalues of \(A\) are \(1.8\) and \(0.2\), so \(A\) is positive definite and the fit at \(\lambda=0\) is unique. Inverting \(A+\lambda I\) gives
At \(\lambda=0\), the denominator is \(1-0.64=0.36\), giving \(\hat{\beta}_1=-0.6/0.36=-1.6667\) and \(\hat{\beta}_2=0.84/0.36=2.3333\). At \(\lambda=0.5\), the denominator is \(2.25-0.64=1.61\), giving \(\hat{\beta}_1=-0.5/1.61=-0.3106\) and \(\hat{\beta}_2=1.34/1.61=0.8323\).
At \(\lambda=2\), the denominator is \(9-0.64=8.36\), giving \(\hat{\beta}_1=-0.2/8.36=-0.0239\) and \(\hat{\beta}_2=2.84/8.36=0.3397\). At \(\lambda=3\), the denominator is \(16-0.64=15.36\), giving \(\hat{\beta}_1=0/15.36=0\) and \(\hat{\beta}_2=3.84/15.36=0.25\). At \(\lambda=4\), the denominator is \(25-0.64=24.36\), giving \(\hat{\beta}_1=0.2/24.36=0.0082\) and \(\hat{\beta}_2=4.84/24.36=0.1987\). Coefficients are rounded to four decimal places.
Check and interpretation. The first coefficient starts negative, approaches zero, reaches zero at \(\lambda=3\), and is positive at \(\lambda=4\). Its absolute value therefore does not decrease at every step. The second coefficient’s magnitude decreases over the displayed values. A common mistake is to treat the correlated case as if each coefficient were \(c_j/(1+\lambda)\); that would give \(\hat{\beta}_1=0.2/1.5=0.1333\) at \(\lambda=0.5\), not the correct \(-0.3106\). The off-diagonal entries of \(A\) couple the coefficient estimates.
How to Read and Report the Plot
When interpreting a ridge path plot, connect a visual observation to the fitted model and the plotted scale. Use the following checks.
Confirm that the axis represents lambda, and use its tick labels to identify which positions have larger penalties. A log-scaled lambda axis can make equal horizontal distances represent multiplicative, not additive, changes in lambda.
Use the labels, where available, to associate each curve with a predictor. Do not assume that the visually highest path belongs to the predictor with the greatest practical influence; units and scaling matter.
State whether a coefficient’s sign changes, whether its magnitude appears to grow or shrink over the displayed range, and whether several paths move together. Avoid claiming that all paths are monotone unless the plot supports that claim.
Note whether predictors were standardized and that the intercept is unpenalized. A path plot describes fitted coefficients; by itself it does not establish predictive performance or select a preferred lambda.
The displayed path is a diagnostic view of how the estimates respond to the penalty. It does not, on its own, identify the best penalty for prediction. That requires an appropriate model-evaluation procedure, considered separately from reading coefficient paths. For an exam response, distinguish direct observations from broader claims: “the coefficient for predictor A crosses zero in the displayed range” is a plot-based observation, while “ridge has selected predictor A out of the model” is not a valid interpretation of a ridge path.
- Do not confuse lambda with the coefficient. Lambda is the penalty setting on the horizontal axis; the fitted coefficients are on the vertical axis.
- Do not claim every coefficient is monotone. The squared length of the coefficient vector is nonincreasing for positive lambda, but correlated predictors can have individual paths that turn or cross zero.
- Do not include the intercept in the shrinkage claim. The intercept is ordinarily unpenalized in glmnet’s ridge fit.
- Do not treat a plotted path as proof of predictive quality. The plot describes coefficient movement, not out-of-sample error.
Check Your Understanding
Work each problem before checking its answer.
- A centered, standardized predictor has \(x^\mathsf{T}x/n=1\) and \(x^\mathsf{T}y/n=1.5\). What is its ridge coefficient at \(\lambda=0.5\)? (Answer: \(1.0000\).)
- Two centered, standardized orthogonal predictors have \(X^\mathsf{T}y/n=(3,-2)^\mathsf{T}\). At \(\lambda=1\), find the coefficient vector and its squared length. (Answer: \((1.5,-1)^\mathsf{T}\), squared length \(3.25\).)
- For \(A=\begin{pmatrix}1&0.8\\0.8&1\end{pmatrix}\), \(c=(0.2,1)^\mathsf{T}\), and \(\lambda=1\), calculate \(\hat{\beta}_1\) and \(\hat{\beta}_2\). (Answer: \(\hat{\beta}_1=-0.4/3.36=-0.1190\), \(\hat{\beta}_2=1.84/3.36=0.5476\), rounded to four decimals.)
- A rank-deficient design has multiple least-squares coefficient vectors at \(\lambda=0\). Which zero-penalty solution is the limit of ridge coefficient vectors as positive \(\lambda\) decreases to zero? (Answer: The minimum-norm least-squares solution.)
- A ridge path plot shows one coefficient approaching zero and then crossing to a small positive value as lambda increases. Is this inconsistent with the nonincrease result for the coefficient-vector length? (Answer: No. That result applies to the vector’s overall length, not to every individual coefficient’s absolute value.)