Why Compare the Three Fits?
When predictors carry nearly the same information, several coefficient combinations can describe the training data almost equally well. Ordinary least squares (OLS) chooses the combination that minimizes the residual sum of squares, but small changes in the data can then produce large changes in individual coefficients. Ridge and lasso add penalties to the fitting objective. Comparing all three methods on one hypothetical dataset makes the trade-offs concrete: the methods can give different coefficients, lasso can set a coefficient to zero, and their prediction errors can differ on held-out cases.
The comparison below uses a deliberately extreme two-predictor example. Its unusually strong collinearity will make the contrast clear. The held-out cases are also deliberately unusual relative to the training predictors; that feature will matter when interpreting their prediction errors.
Model and Comparison Setup
These objectives use the same residual-loss scale, but their penalty terms differ. Ridge penalizes squared coefficient sizes and generally shrinks coefficients without making them exactly zero. Lasso penalizes absolute coefficient sizes and can produce exact zeros. The numerical value of \(\lambda\) depends on the objective convention, sample size, and predictor scale, so a ridge \(\lambda\) and a lasso \(\lambda\) are not directly comparable.
Consider hypothetical centered training data with two predictors. The following cross-products summarize the training data; they are sufficient to calculate the OLS and ridge coefficients. The predictors have equal sums of squares and a correlation of \(499995/500005\), approximately \(0.99998\), so they are extremely collinear.
For this example, the training cross-products correspond to OLS coefficients approximately \(12.00016\) and \(8.00016\). The different \(\lambda\) values used below are illustrative fits, not values chosen by comparing the held-out outcomes. In an applied analysis, tuning should be performed using training data and an appropriate validation procedure; a final held-out set should be reserved for evaluation.
Worked Example: Calculate the OLS Coefficients
For centered predictors, the OLS normal equations are \(X^\mathsf{T}X\hat{\beta}_{\mathrm{OLS}}=X^\mathsf{T}y\). Using the cross-products above gives
The solution is approximately \(\hat{\beta}_{1,\mathrm{OLS}}=12.00016\) and \(\hat{\beta}_{2,\mathrm{OLS}}=8.00016\). Substitution into the first equation gives \(500005(12.00016)+499995(8.00016)\approx10000175.942\); the second gives \(499995(12.00016)+500005(8.00016)\approx10000135.942\). The small difference between these coefficients and 12 and 8 is retained in the later calculations.
This fit identifies a useful combined direction: the sum of the coefficients is about \(20.00031\). Because the predictors move together, the training data provide much less information about how that sum should be divided between the two individual coefficients. This is why the OLS coefficients should not automatically be read as stable estimates of separate predictor effects.
How Ridge Changes the Coefficients
Ridge regression replaces the OLS normal equations with \((X^\mathsf{T}X+\lambda I)\hat{\beta}_{\mathrm{ridge}}=X^\mathsf{T}y\). For the example, choose \(\lambda\) to be approximately \(15.5942\), with its unrounded value defined by \(10(4/1.5628526-1)\). The resulting coefficients are approximately \(10.7814263\) and \(9.2185737\).
A short derivation shows why the two coefficients move toward one another. The matrix \(X^\mathsf{T}X\) has a common-predictor direction \((1,1)\) with eigenvalue \(1000000\), and a contrast direction \((1,-1)\) with eigenvalue \(10\). Ridge multiplies an OLS component in an eigenvector direction with eigenvalue \(d\) by \(d/(d+\lambda)\). Thus, it shrinks the contrast component far more strongly than the common component here.
The OLS coefficient sum is approximately \(20.0003119\), and the difference is \(4\). After ridge shrinkage, the sum is \(20\) to the displayed precision and the difference is \(1.5628526\). Converting back from sum and difference gives \((20+1.5628526)/2=10.7814263\) and \((20-1.5628526)/2=9.2185737\). The near-unchanged sum and noticeably reduced difference illustrate ridge’s direction-specific shrinkage in this collinear design.
Worked Example: Calculate Ridge Held-Out Error
Suppose two held-out cases have predictor profiles \((1,0)\) and \((0,1)\), with centered observed responses \(10.900\) and \(9.200\). The ridge predictions are the corresponding fitted coefficients: \(10.7814263\) and \(9.2185737\). The held-out mean squared prediction error (MSE) is the average of the squared prediction errors:
The result is approximately \(0.007202\) squared response units. It is the average squared error on these two specified cases, not an estimate guaranteed to describe every future prediction. In particular, these profiles have one predictor equal to zero while the other is nonzero, unlike the nearly matched predictor values that dominate the collinear training data.
Lasso Can Set a Coefficient to Zero
For lasso, use the same training cross-products and choose \(\lambda=9500000\). To check whether a solution with the second coefficient equal to zero is valid, suppose \(\hat{\beta}_{1,\mathrm{lasso}}=b>0\) and \(\hat{\beta}_{2,\mathrm{lasso}}=0\). Minimizing the objective with respect to the first coefficient gives \(500005b=10000175.942-\lambda\), so \(b\approx1.000342\).
That calculation alone does not establish that the second coefficient should be zero. For a zero lasso coefficient, the absolute value of its residual cross-product with the corresponding predictor must be no greater than \(\lambda\). Here the residual cross-product is approximately \(9499970\), which is less than \(9500000\). The first coefficient is positive and satisfies its lasso equation; the second satisfies the zero-coefficient condition. Thus the fitted lasso coefficients are approximately \(1.000342\) and \(0\).
Worked Example: Check Lasso Selection and Compare Test Errors
For the proposed lasso fit, the first coefficient is
The residual cross-product for the second predictor is
This inequality verifies that setting the second coefficient to zero is compatible with the lasso optimality condition. It is a selected-variable result for this fit, not evidence that the second predictor is irrelevant in every population or under every penalty choice.
Now compare all three models on the same held-out cases. Their predictions are the fitted coefficients at profiles \((1,0)\) and \((0,1)\). For OLS they are approximately \(12.00016\) and \(8.00016\); for ridge they are \(10.7814263\) and \(9.2185737\); for lasso they are approximately \(1.000342\) and \(0\). Averaging each pair of squared errors gives:
On these two cases, ridge has the smallest MSE, OLS has a larger MSE, and this strongly penalized lasso fit has the largest MSE. The result describes only these held-out cases. Their predictor profiles differ sharply from the training pattern, so this comparison should not be treated as a general ranking of the methods. In a practical analysis, a representative test set is important for meaningful evaluation.
What the Comparison Shows
The estimates and errors answer different questions. OLS minimizes training residual sum of squares without a coefficient penalty. Ridge accepts some bias from shrinkage to stabilize coefficients, especially in directions that the data estimate weakly. Lasso can additionally give a sparse fit, but the predictors it retains can depend on the penalty and on the arrangement of correlated predictors. The selected variable in this example does not establish that it is uniquely important; a similar predictor may carry overlapping information.
The held-out MSE compares prediction errors on a defined set of cases. A low value is useful evidence about performance on cases like those held out, provided those cases represent the intended prediction setting. It does not establish that the coefficient estimates are close to any particular population coefficients. Prediction accuracy and coefficient recovery are distinct goals.
Common Mistake
A related mistake is to announce that the model with the smallest MSE in a tiny held-out sample must predict best in general. Here there are only two held-out cases, and their predictor profiles are unusual relative to training. Report the comparison in context, consider whether the evaluation cases represent the intended use, and do not use the held-out set to choose \(\lambda\).
Check Your Understanding
Use the definitions and calculations above to answer these questions.
- Why can OLS coefficients be unstable when two predictors are nearly collinear, even if their combined contribution is estimated more reliably?
- Which coefficient direction is shrunk more strongly by ridge in this example, and why?
- What condition verifies that the second lasso coefficient can equal zero in the worked fit?
- Why is the ridge held-out MSE approximately \(0.007202\), rather than \(0.007203\), when rounded to six decimal places?
- What limitation should be noted when comparing the three held-out MSE values in this example?