Why Shrinkage Can Help
When predictors contain much of the same information, a regression model may have difficulty deciding which predictor should receive credit for an observed pattern in the response. The fitted values can remain similar while individual coefficient estimates change substantially. This is one reason that a model can predict reasonably well even when its coefficients are unstable.
Ridge regression addresses this problem by adding a penalty for large coefficients to the least-squares objective. The penalty pulls estimates toward zero, which introduces some bias, but it can reduce the sensitivity of predictions to small changes in the data. Whether that trade-off is useful depends on the aim: stabilizing predictions is not the same as estimating a distinct, unpenalized effect for each predictor.
The Model and the Ridge Estimate
For context, suppose a response \(Y\) is modeled as a linear function of \(p\) predictors: \(Y_i=\beta_0+\beta_1X_{i1}+\cdots+\beta_pX_{ip}+\varepsilon_i\). For a prediction interpretation, assume the conditional mean of the response is adequately represented by this linear form and that new observations are drawn from the same process as the observations used to fit the model. The errors need not be normally distributed to calculate ridge estimates. However, the prediction benefit of shrinkage is not guaranteed by the formula alone: it depends on the data and on how well the fitted model represents the prediction setting.
Write \(X\) for the design matrix of predictor values and \(y\) for the response vector. Assume the columns of \(X\) and the response have been centered, so the intercept can be handled separately. If the predictors are also standardized, their columns are on a common scale. For \(\lambda=0\), minimizing the objective gives the least-squares estimate. For \(\lambda>0\), the ridge estimate is
Here \(I\) is the identity matrix. When \(\lambda>0\), adding \(\lambda I\) makes the matrix invertible, even if some predictor columns are exactly redundant. The method therefore gives a unique penalized estimate in situations where the separate least-squares coefficients may not be uniquely determined. Ridge regression usually shrinks coefficients toward zero but does not generally set them exactly to zero.
The basic use of this formula assumes that the model’s predictors and response are defined consistently for fitting and prediction. In practice, the penalty’s meaning also depends on the units of the predictors. Standardizing predictors before fitting makes the penalty apply on a more comparable scale; an intercept, when included, remains unpenalized.
Worked Example: Shrinking Coefficients for Two Redundant Predictors
Suppose a hypothetical centered data set has two similarly scaled, highly correlated predictors. Its predictor cross-products and predictor-response cross-products are \(X^\mathsf{T}X=\begin{pmatrix}10&9\\9&10\end{pmatrix}\) and \(X^\mathsf{T}y=\begin{pmatrix}20\\18\end{pmatrix}\). The off-diagonal value 9 is close to the diagonal values 10, indicating substantial overlap between the predictors. Compare the least-squares estimate with ridge regression at \(\lambda=1\).
For least squares, the determinant is \(10(10)-9(9)=19\), so \[ (X^\mathsf{T}X)^{-1}=\frac{1}{19}\begin{pmatrix}10&-9\\-9&10\end{pmatrix}. \] Thus \[ \hat{\boldsymbol{\beta}}_{\text{LS}} =\frac{1}{19}\begin{pmatrix}10(20)-9(18)\\-9(20)+10(18)\end{pmatrix} =\begin{pmatrix}2\\0\end{pmatrix}. \]
For ridge regression with \(\lambda=1\), add 1 to each diagonal entry: \(X^\mathsf{T}X+I=\begin{pmatrix}11&9\\9&11\end{pmatrix}\). Its determinant is \(11(11)-9(9)=40\), giving \[ \hat{\boldsymbol{\beta}}_{\text{ridge}} =\frac{1}{40}\begin{pmatrix}11&-9\\-9&11\end{pmatrix} \begin{pmatrix}20\\18\end{pmatrix} =\frac{1}{40}\begin{pmatrix}220-162\\-180+198\end{pmatrix} =\begin{pmatrix}1.45\\0.45\end{pmatrix}. \]
The least-squares fit assigns the entire coefficient to the first predictor; ridge distributes the fitted contribution across both. For a new case with centered predictor values \(x_1=x_2=1\), the least-squares prediction from these slopes is \(2(1)+0(1)=2\); the ridge prediction is \(1.45(1)+0.45(1)=1.90\). This calculation illustrates a changed prediction as well as shrunken coefficients. It does not establish that ridge predicts better; that must be assessed using data not used to fit the model.
Why Shrinkage Can Stabilize the Fit
The effect of the penalty is clearest when the predictors are expressed in directions of information. A direction well supported by the data has a relatively large eigenvalue of \(X^\mathsf{T}X\); a direction that is poorly determined has a small one. If \(v\) is a unit eigenvector with eigenvalue \(d\), then along that direction the ridge estimate scales the least-squares component by \(d/(d+\lambda)\). This follows directly from the ridge formula: in that direction, \(X^\mathsf{T}X+\lambda I\) multiplies by \(d+\lambda\), rather than \(d\).
For \(\lambda>0\), this factor is between 0 and 1. When \(d\) is small, it is much less than 1, so ridge strongly reduces the component that the data determine poorly. When \(d\) is large relative to \(\lambda\), the factor is closer to 1, so that component is changed less. With highly correlated predictors, the poorly determined direction often corresponds to increasing one coefficient while decreasing another. Ridge can therefore reduce large, opposing coefficient movements that have little effect on fitted values.
This is a stabilization argument, not a guarantee of improved prediction. Shrinkage adds bias: even if the linear model is appropriate, the ridge coefficients do not generally equal the population regression coefficients. It can still improve prediction if the reduction in variability outweighs the extra bias for the prediction task at hand.
Worked Example: A Small Data Change and an Unstable Coefficient Split
Continue with the cross-products in the first example. Imagine a small change in the response measurements changes \(X^\mathsf{T}y\) from \((20,18)^\mathsf{T}\) to \((20.5,17.5)^\mathsf{T}\), while \(X^\mathsf{T}X\) stays the same. Calculate the new least-squares and \(\lambda=1\) ridge estimates.
Using the same least-squares inverse, \[ \hat{\boldsymbol{\beta}}_{\text{LS,new}} =\frac{1}{19}\begin{pmatrix}10(20.5)-9(17.5)\\-9(20.5)+10(17.5)\end{pmatrix} =\frac{1}{19}\begin{pmatrix}205-157.5\\-184.5+175\end{pmatrix} =\begin{pmatrix}2.50\\-0.500\end{pmatrix}. \] The first coefficient has increased from 2.00 to 2.50, and the second has changed from 0 to \(-0.500\). Yet their sum remains 2.00, as it did before the data change.
For ridge regression, \[ \hat{\boldsymbol{\beta}}_{\text{ridge,new}} =\frac{1}{40}\begin{pmatrix}11&-9\\-9&11\end{pmatrix} \begin{pmatrix}20.5\\17.5\end{pmatrix} =\frac{1}{40}\begin{pmatrix}225.5-157.5\\-184.5+192.5\end{pmatrix} =\begin{pmatrix}1.70\\0.200\end{pmatrix}. \] The coefficient changes are \(1.70-1.45=0.25\) and \(0.20-0.45=-0.25\), half the least-squares changes in this example. Their sum remains 1.90. Thus, for predictions at \(x_1=x_2=1\), the ridge prediction remains 1.90 across these two sets of cross-products, while the least-squares prediction remains 2.00. The example shows reduced movement in the coefficient split; it does not show that the ridge prediction is closer to an unknown true response.
Scaling and the Meaning of the Penalty
The ridge penalty is a sum of squared coefficient values, so changing a predictor’s units changes the penalty unless predictors are scaled consistently. Consider two highly correlated measurements used to predict an outcome. Suppose a fitted model has coefficient \(0.40\) per hour for the first measurement and \(0.020\) per minute for the second. The second measurement can instead be recorded in hours: \(x_{\text{hour}}=x_{\text{minute}}/60\). To preserve its contribution to the prediction, its coefficient must be \(0.020(60)=1.20\) per hour.
The two models make exactly the same predictions after this conversion. But the squared-coefficient penalty for these two slopes changes from \(0.40^2+0.020^2=0.1604\) to \(0.40^2+1.20^2=1.60\). A ridge fit on raw measurements could therefore change simply because one measurement was recorded in hours instead of minutes. Standardizing predictors before applying the penalty avoids letting arbitrary measurement units determine their relative shrinkage. When using the fitted model later, apply the same centering and scaling used for training data.
Worked Example: Choosing Whether Ridge Matches the Question
Suppose a hypothetical analyst has two standardized, strongly correlated measurements recorded at baseline and wants to predict a later outcome for new cases. The fitted ridge slopes are 0.70 and 0.55, and the selected penalty is \(\lambda=2\). The analyst should first verify that the measurements were standardized using the training data and that the intercept was not penalized. The practical question is then whether predictions improve at the chosen penalty.
For example, if the analyst has a validation set with \(n=4\) cases, the mean squared prediction error is \[ \mathrm{MSE}=\frac{1}{4}\sum_{i=1}^{4}(y_i-\hat{y}_i)^2. \] If the squared prediction errors on that set are \(1,4,1,2\), then \(\mathrm{MSE}=(1+4+1+2)/4=2.00\) outcome-units squared. The analyst can compare this value with the validation-set MSE for a least-squares model fitted to the same training data. A smaller validation MSE supports ridge for this prediction setting; it does not prove that ridge will be better for every future sample.
Now change the question: suppose the analyst needs to estimate the separate unpenalized coefficient for each baseline measurement. The ridge slopes 0.70 and 0.55 are not those unpenalized estimates: they were obtained by minimizing an objective that deliberately penalizes large slopes. Ridge may be useful for prediction, but it does not resolve the underlying lack of information about separate effects. The analyst should report the limitation and use an approach suited to the separate-effect question rather than interpreting the ridge slopes as ordinary least-squares effects.
When Ridge Helps—and When It Does Not
Ridge is especially worth considering when predictors are redundant, the goal is accurate prediction for cases from a comparable setting, and small changes in the data cause large coefficient shifts. The amount of shrinkage is controlled by \(\lambda\): at \(\lambda=0\) the fit is least squares, and increasing \(\lambda\) shrinks the slopes more strongly. A value for \(\lambda\) can be chosen by comparing predictive performance across candidate values using validation data or cross-validation. The final prediction performance should be assessed on data not used to choose the penalty.
Ridge is not a way to make collinearity disappear or to establish which redundant predictor has a uniquely important role. If the main aim is to estimate separate effects, shrinkage changes the target of the fitting procedure. The estimates are biased relative to the unpenalized least-squares target, and the separate coefficients may still be difficult to interpret. Combining measures into a defensible score can be a different modeling choice when the scientific meaning supports a single composite predictor.
A common mistake is to reason, “Ridge made the coefficients smaller and more similar, so it has identified each predictor’s real separate effect.” That conclusion does not follow. Ridge deliberately changes the estimates to stabilize a penalized fit. Its usefulness must be judged against the goal—most often prediction—not by whether the resulting coefficients look more plausible.
Check Your Understanding
Answer these questions using the ideas and calculations in the tutorial.
- What does the ridge penalty add to the least-squares objective, and why is the intercept usually left out of that penalty?
- Why can a ridge fit change individual coefficients while leaving predictions for some predictor combinations nearly unchanged?
- If a predictor is converted from minutes to hours, why can a ridge fit change when the numerical coefficient is not adjusted for the unit conversion?
- In the small-change example, how did the least-squares coefficient changes compare with the ridge coefficient changes?
- Why does a lower validation-set prediction error support a prediction use for ridge but not establish that its slopes are the separate unpenalized effects?