Comparing Fits Across Penalty Strengths

Why Compare Fits at Several Penalty Strengths?

A ridge or lasso fit is not determined by the data alone: it also depends on the penalty strength. A smaller penalty generally permits a closer fit to the training data, while a larger penalty discourages large coefficients. For ridge, that discouragement is smooth; for lasso, some coefficients can become exactly zero. Comparing fits at several fixed values of the penalty makes these effects concrete.

The purpose here is to calculate and interpret fits, not to choose a penalty using a validation procedure. A fair numerical comparison requires keeping the objective’s scaling convention fixed and knowing how the predictors were scaled. The examples use centered predictors, so the intercept is the response mean and does not enter the penalty.

Definition: Let \(y_i\) be the response and \(x_i\) the vector of \(p\) centered predictor values for observation \(i\). The ridge estimate at penalty strength \(\lambda \geq 0\) minimizes \( \frac{1}{2n}\sum_{i=1}^n (y_i-\beta_0-x_i^\mathsf{T}\beta)^2+\frac{\lambda}{2}\sum_{j=1}^p\beta_j^2 \). The lasso estimate minimizes \( \frac{1}{2n}\sum_{i=1}^n (y_i-\beta_0-x_i^\mathsf{T}\beta)^2+\lambda\sum_{j=1}^p|\beta_j| \). In both objectives, the intercept \(\beta_0\) is unpenalized. In this centered setup, \(\beta_0=\bar y\). The value of \(\lambda\) is meaningful only in relation to the stated objective scaling and predictor scale.

Working From Cross-Products

When predictors and response are centered, define \(G=X^\mathsf{T}X/n\), \(c=X^\mathsf{T}y/n\), and \(s_y=y^\mathsf{T}y/n\). Here \(G\) summarizes predictor cross-products, \(c\) summarizes predictor-response cross-products, and \(s_y\) is the centered response sum of squares divided by \(n\). These summaries are enough to calculate the ridge coefficients and, for a specified coefficient vector, the training mean squared error (MSE).

\hat\beta_{\text{ridge}}=(G+\lambda I)^{-1}c, \qquad \operatorname{MSE}_{\text{train}}=s_y-2c^\mathsf{T}\hat\beta+\hat\beta^\mathsf{T}G\hat\beta.

The ridge formula follows by differentiating the objective with respect to \(\beta\). The derivative of the residual term is \(G\beta-c\); the derivative of the penalty is \(\lambda\beta\). Setting their sum to zero gives \((G+\lambda I)\beta=c\). If \(G+\lambda I\) is invertible, solving this system gives the displayed estimate. For \(\lambda>0\), it is invertible for any positive semidefinite \(G\). At \(\lambda=0\), invertibility requires full column rank.

For lasso, the absolute-value penalty is not differentiable at zero. A nonzero coefficient must satisfy the ordinary derivative condition with its sign, while a zero coefficient must satisfy an inequality. In the centered setup, the conditions are \(G\hat\beta-c+\lambda s=0\), where \(s_j=\operatorname{sign}(\hat\beta_j)\) for a nonzero coefficient and \(s_j\in[-1,1]\) for a zero coefficient. The zero-coefficient condition is especially useful for checking whether a proposed sparse fit is valid.

Worked Example: Comparing Ridge and Lasso From the Same Summaries

A hypothetical risk study uses two centered predictors. Suppose \(G=\begin{pmatrix}1&0.5\\0.5&1\end{pmatrix}\), \(c=(1,0.6)^\mathsf{T}\), \(s_y=2\), and \(\bar y=10\). Calculate ridge fits at \(\lambda=0.5\) and \(1\), and lasso fits at \(\lambda=0.1\), \(0.2\), and \(0.3\). For each fit, calculate the prediction at centered predictor profile \(x=(1,1)^\mathsf{T}\). Also report training MSE.

Method. Use the fixed objective conventions above. For ridge, solve the two-by-two system \( (G+\lambda I)\hat\beta=c\). For lasso, check the active-coefficient equations and the zero-coefficient inequality. The profile prediction is \(10+\hat\beta_1+\hat\beta_2\).

Ridge at \(\lambda=0.5\). The system is \(1.5\hat\beta_1+0.5\hat\beta_2=1\) and \(0.5\hat\beta_1+1.5\hat\beta_2=0.6\). Its determinant is \(1.5(1.5)-0.5(0.5)=2\). Thus \(\hat\beta_1=(1.5(1)-0.5(0.6))/2=0.6\) and \(\hat\beta_2=(1.5(0.6)-0.5(1))/2=0.2\). The profile prediction is \(10+0.6+0.2=10.8\). The training MSE is \(2-2(1(0.6)+0.6(0.2))+(0.6^2+2(0.5)(0.6)(0.2)+0.2^2)=2-1.44+0.52=1.08\).

Ridge at \(\lambda=1\). The system has diagonal entries \(2\) and off-diagonal entries \(0.5\); its determinant is \(4-0.25=3.75\). Therefore \(\hat\beta_1=(2(1)-0.5(0.6))/3.75=0.4533\) and \(\hat\beta_2=(2(0.6)-0.5(1))/3.75=0.1867\), rounded to four decimals. The prediction is \(10+0.4533+0.1867=10.6400\). Using the unrounded coefficients in the MSE formula gives \(1.1943\), rounded to four decimals.

Lasso at \(\lambda=0.1\). Try both coefficients positive. The active equations are \(G\hat\beta=c-(0.1,0.1)^\mathsf{T}=(0.9,0.5)^\mathsf{T}\). Since the determinant of \(G\) is \(1-0.25=0.75\), \(\hat\beta_1=(0.9-0.5(0.5))/0.75=0.8667\) and \(\hat\beta_2=(0.5-0.5(0.9))/0.75=0.0667\), rounded to four decimals. Both are positive, as assumed. The prediction is \(10+0.8667+0.0667=10.9333\). Substitution into the MSE formula, using unrounded coefficients, gives \(1.0000\).

Lasso at \(\lambda=0.2\). Try \(\hat\beta_2=0\) and \(\hat\beta_1>0\). The first active equation gives \(\hat\beta_1=1-0.2=0.8\). For the second coefficient to be zero, its residual correlation must have absolute value at most \(0.2\). It is \(c_2-G_{21}\hat\beta_1=0.6-0.5(0.8)=0.2\), so the boundary condition holds. The prediction is \(10.8\), and the MSE is \(2-2(0.8)+0.8^2=1.04\).

Lasso at \(\lambda=0.3\). Again set \(\hat\beta_2=0\). The first equation gives \(\hat\beta_1=1-0.3=0.7\). The second residual correlation is \(0.6-0.5(0.7)=0.25\), which is less than \(0.3\), so zero is valid. The prediction is \(10.7\), and the MSE is \(2-2(0.7)+0.7^2=1.09\).

The results show two different patterns: ridge reduces both coefficients without setting either to zero, while lasso removes the second coefficient at these larger penalties. The lasso fit at \(0.2\) has greater training MSE than the fit at \(0.1\); it uses a simpler coefficient pattern, not a better fit to the training responses.

What Must Change as the Penalty Increases?

An individual coefficient does not have to decrease in absolute value at every step along a ridge or lasso path, particularly when predictors are related. More reliable comparisons concern the total penalty quantity and training error. These comparisons follow directly from the fact that each estimate minimizes its stated objective.

Result: For either method, hold the data and objective scaling fixed. If \(0\leq\lambda_1<\lambda_2\), then the penalty quantity at the minimizing fit cannot be larger at \(\lambda_2\) than at \(\lambda_1\), and the training residual sum of squares cannot be smaller at \(\lambda_2\). For ridge the penalty quantity is \(\frac12\sum_j\hat\beta_j^2\); for lasso it is \(\sum_j|\hat\beta_j|\). This result does not say that every coefficient moves monotonically or that a larger penalty improves prediction on new observations.

To justify the result, write \(R(\beta)\) for the residual sum of squares divided by \(2n\), and \(P(\beta)\) for the relevant penalty quantity. Let \(\hat\beta_1\) and \(\hat\beta_2\) minimize \(R+\lambda_1P\) and \(R+\lambda_2P\), respectively. Optimality gives two inequalities:

R(\hat\beta_1)+\lambda_1P(\hat\beta_1) \leq R(\hat\beta_2)+\lambda_1P(\hat\beta_2), \qquad R(\hat\beta_2)+\lambda_2P(\hat\beta_2) \leq R(\hat\beta_1)+\lambda_2P(\hat\beta_1).

Adding them and rearranging yields \((\lambda_2-\lambda_1)\{P(\hat\beta_1)-P(\hat\beta_2)\}\geq0\). Since \(\lambda_2-\lambda_1>0\), the penalty quantity cannot increase. Substituting that ordering into the first optimality inequality gives \(R(\hat\beta_1)\leq R(\hat\beta_2)\), so the training residual sum of squares cannot decrease. This argument compares fits for the same data and same penalty definition; it does not allow direct interpretation of a ridge \(\lambda\) as numerically equivalent to a lasso \(\lambda\).

Worked Example: Checking a Lasso Active Set and Its Prediction

A hypothetical portfolio study has two centered predictors with \(G=\begin{pmatrix}2&1\\1&2\end{pmatrix}\), \(c=(3,2)^\mathsf{T}\), and response mean \(20\). At lasso penalty \(\lambda=0.5\), suppose both coefficients are positive; at \(\lambda=1.25\), check whether the second coefficient can be zero. Calculate the resulting predictions at profile \(x=(1,-1)^\mathsf{T}\).

At \(\lambda=0.5\). With both coefficients positive, the active equations are \(G\hat\beta=(3-0.5,2-0.5)^\mathsf{T}=(2.5,1.5)^\mathsf{T}\). The determinant of \(G\) is \(2(2)-1(1)=3\). Hence \(\hat\beta_1=(2(2.5)-1(1.5))/3=3.5/3=7/6\) and \(\hat\beta_2=(2(1.5)-1(2.5))/3=0.5/3=1/6\). Both are positive, confirming the assumed signs. The centered fitted contribution at the requested profile is \((7/6)(1)+(1/6)(-1)=1\), so the prediction is \(20+1=21\).

At \(\lambda=1.25\). Set \(\hat\beta_2=0\). The first active equation is \(2\hat\beta_1=3-1.25=1.75\), giving \(\hat\beta_1=0.875\). The residual correlation for the second coefficient is \(c_2-G_{21}\hat\beta_1=2-0.875=1.125\). Since \(1.125\leq1.25\), the zero-coefficient condition is satisfied. The fitted contribution at \((1,-1)^\mathsf{T}\) is \(0.875\), and the prediction is \(20.875\).

The second fit has one nonzero coefficient rather than two, and its prediction differs because the profile gives the predictors opposite signs. A prediction comparison must therefore use the specified profile; the coefficient sum alone would not answer this question.

Interpreting a Table of Fits

A useful comparison table keeps the method, penalty, coefficients, and fitted quantity together. The following calculations use a one-predictor centered example with \(G=2\), \(c=1.4\), response mean \(5\), and profile \(x=2\). The ridge coefficient is \(c/(G+\lambda)\). The lasso coefficient, when positive, is \((c-\lambda)/G\), and it is zero when \(c\leq\lambda\).

Worked Example: Reading Shrinkage and Training Error Together

For the one-predictor summaries above, compare ridge and lasso at \(\lambda=0.4\) and \(\lambda=1\). The profile prediction is \(5+2\hat\beta\). Take \(s_y=3\) and calculate training MSE as \(3-2(1.4)\hat\beta+2\hat\beta^2\).

At \(\lambda=0.4\), ridge. The coefficient is \(1.4/(2+0.4)=1.4/2.4=0.5833\), rounded to four decimals. The prediction is \(5+2(0.5833)=6.1667\), using the unrounded coefficient before rounding. The MSE is \(3-2(1.4)(7/12)+2(7/12)^2=2.0472\), rounded to four decimals.

At \(\lambda=0.4\), lasso. The positive coefficient is \((1.4-0.4)/2=0.5\). The prediction is \(5+2(0.5)=6\), and the MSE is \(3-2(1.4)(0.5)+2(0.5)^2=2.1\).

At \(\lambda=1\), ridge. The coefficient is \(1.4/(2+1)=0.4667\), rounded to four decimals. The prediction is \(5+2(1.4/3)=5.9333\), and the MSE is \(3-2(1.4)(1.4/3)+2(1.4/3)^2=2.1289\), rounded to four decimals.

At \(\lambda=1\), lasso. The positive coefficient is \((1.4-1)/2=0.2\). The prediction is \(5+2(0.2)=5.4\), and the MSE is \(3-2(1.4)(0.2)+2(0.2)^2=2.52\).

For both methods, the larger penalty produces a smaller coefficient and a larger training MSE in this example. But this does not imply that ridge and lasso with the same numerical \(\lambda\) are equally penalized: their penalty terms differ. Nor does training MSE alone establish which model predicts better outside the data used to fit it.

Common Mistakes and Practical Checks

Changing the objective convention mid-calculation. Under the lasso objective used here, the threshold in the one-predictor formula is \(\lambda\). If the loss or penalty is scaled differently, the numerical threshold changes.
Assuming every lasso coefficient is positive and active. Solve under the proposed active set, check the signs, then check the absolute residual correlations for coefficients set to zero.
Assuming ridge makes coefficients exactly zero. Ridge shrinks coefficients, but its squared penalty does not generally set individual coefficients to zero.
Calling a lower training error a better predictive model. Training error is expected to be nondecreasing as the penalty increases under the fixed objectives. It does not by itself measure performance on new data.
Comparing penalty values without checking scaling. Changing predictor units or the loss normalization changes the numerical meaning of \(\lambda\). State the scaling convention before interpreting its value.

Key takeaway: For each fixed penalty, calculate the coefficients under one stated objective, verify lasso active-set conditions, and then compute fitted values at the predictor profile of interest. Across increasing penalties, expect the total penalty quantity to fall and training error not to fall; do not infer that each coefficient changes monotonically or that one fit generalizes better from these facts alone.

Check Your Understanding

For each problem, use the ridge and lasso objective conventions stated in this tutorial. Round coefficients and predictions to four decimals when needed.

A centered one-predictor fit has \(G=3\), \(c=1.8\), response mean \(7\), and profile \(x=2\). Calculate the ridge coefficient and prediction at \(\lambda=0.6\), then the lasso coefficient and prediction at \(\lambda=0.3\). (Answer: ridge \(\hat\beta=0.5000\), prediction \(8.0000\); lasso \(\hat\beta=0.5000\), prediction \(8.0000\))
For lasso, let \(G=\begin{pmatrix}1&0.4\\0.4&1\end{pmatrix}\), \(c=(0.9,0.5)^\mathsf{T}\), and \(\lambda=0.2\). Check the both-positive active solution, including its signs, and report the coefficients. (Answer: \(\hat\beta=(0.6905,0.0238)^\mathsf{T}\), both positive)
For ridge, let \(G=\begin{pmatrix}2&0.5\\0.5&1\end{pmatrix}\), \(c=(1.5,0.8)^\mathsf{T}\), and \(\lambda=0.5\). Solve for both coefficients. (Answer: \(\hat\beta=(0.5286,0.3571)^\mathsf{T}\))
For lasso, let \(G=\begin{pmatrix}1&0.5\\0.5&1\end{pmatrix}\), \(c=(1,0.6)^\mathsf{T}\), and \(\lambda=0.25\). Check the candidate with the second coefficient zero, and report the coefficient vector if the KKT condition holds. (Answer: \(\hat\beta=(0.7500,0)^\mathsf{T}\); the inactive residual correlation is \(0.225\leq0.25\))
A centered model has intercept \(12\), coefficients \((0.4,0.2)^\mathsf{T}\), and a new profile \(x=(2,-1)^\mathsf{T}\). Calculate its fitted value, showing why the second predictor’s contribution is negative. (Answer: \(12+0.4(2)+0.2(-1)=12.6000\))

What You'll Learn