Limiting Cases of Ridge Regression

Why the Endpoints of the Ridge Path Matter

Ridge regression adds a penalty to least squares so that large slope coefficients are discouraged. Understanding the two endpoints of its penalty parameter, $ \lambda $, gives a useful check on both the algebra and the interpretation: at $ \lambda=0 $, ridge reduces to ordinary least squares when the least-squares coefficients are uniquely determined; as $ \lambda $ grows without bound, the slopes shrink toward zero. The intercept behaves differently because the standard ridge objective does not penalize it.

These are limiting statements about a specified objective, not promises that any finite positive penalty makes the coefficients exactly zero. The examples use hypothetical data and the following convention throughout: minimize the sum of squared residuals plus $ \lambda $ times the sum of squared slopes, leaving the intercept unpenalized.

Definition: For a response vector $y$, an $n$-row design matrix $X$, an intercept $ \beta_0 $, and a slope vector $ \boldsymbol{\beta} $, define the ridge objective as $$ Q(\beta_0,\boldsymbol{\beta}) = \sum_{i=1}^{n}(y_i-\beta_0-\mathbf{x}_i^\mathsf{T}\boldsymbol{\beta})^2 +\lambda\boldsymbol{\beta}^\mathsf{T}\boldsymbol{\beta}, \qquad \lambda\geq 0. $$ Here $ \mathbf{x}_i $ is the vector of predictors for observation $i$. The penalty applies to the slopes, not the intercept. A different objective, such as one that penalizes the intercept, need not have the same limiting intercept.

Centering Separates the Intercept From the Slopes

Write $ \bar{y} $ for the response mean and $ \bar{\mathbf{x}} $ for the vector of predictor means. For any fixed slope vector, minimizing the sum of squared residuals with respect to the unpenalized intercept gives $ \widehat{\beta}_0=\bar{y}-\bar{\mathbf{x}}^\mathsf{T}\widehat{\boldsymbol{\beta}} $. Thus, if the predictors have been centered so that each column of $X_c$ has mean zero, the fitted intercept is $ \bar{y} $ for every value of $ \lambda $. The corresponding centered response is $ y_c=y-\bar{y}\mathbf{1} $.

After centering, the slope problem is to minimize $ \lVert y_c-X_c\boldsymbol{\beta}\rVert^2+\lambda\boldsymbol{\beta}^\mathsf{T}\boldsymbol{\beta} $. When $ \lambda>0 $, its unique solution is $ (X_c^\mathsf{T}X_c+\lambda I)^{-1}X_c^\mathsf{T}y_c $. When $ \lambda=0 $, this inverse formula gives the ordinary least-squares solution only if $X_c^\mathsf{T}X_c$ is invertible—that is, if the centered predictors are linearly independent.

Worked Example: The Zero-Penalty Endpoint With Independent Predictors

A hypothetical pricing analysis uses two centered predictors, $x_1=(-1,0,1)$ and $x_2=(1,-2,1)$, to model three centered response values $y_c=(-3,2,1)$. Find the ridge coefficients at $ \lambda=0 $. State the intercept if the uncentered response mean is 5.

Method. At $ \lambda=0 $, solve the ordinary least-squares normal equations. First check the predictor cross-products: $$ x_1^\mathsf{T}x_1=1+0+1=2,\qquad x_2^\mathsf{T}x_2=1+4+1=6, $$ $$ x_1^\mathsf{T}x_2=(-1)(1)+(0)(-2)+(1)(1)=0. $$ The centered predictors are orthogonal and each has a positive sum of squares, so the design has full column rank.

The cross-products with the response are $$ x_1^\mathsf{T}y_c=(-1)(-3)+(0)(2)+(1)(1)=4, \qquad x_2^\mathsf{T}y_c=(1)(-3)+(-2)(2)+(1)(1)=-6. $$ The normal equations therefore give $ \widehat{\beta}_1=4/2=2 $ and $ \widehat{\beta}_2=-6/6=-1 $. Since the predictors were centered, $ \widehat{\beta}_0=\bar{y}=5 $.

Check. The fitted values are $5+2(-1)-1(1)=2$, $5+2(0)-1(-2)=7$, and $5+2(1)-1(1)=6$. These are not the stated observations because the centered response values are $(-3,2,1)$, not $(-3,2,1)+5$ evaluated with the listed predictors. Rechecking the problem data reveals an inconsistency: with the stated centered response, the response mean is zero and the least-squares slopes are $2,-1$, so the fitted centered values are $(-3,2,1)$. Adding the separately stated response mean of 5 gives actual responses $(2,7,6)$, whose fitted values are exactly $(2,7,6)$. The intercept and slopes are therefore consistent with the uncentered responses $(2,7,6)$.

The common mistake this method avoids is using the inverse formula at $ \lambda=0 $ without checking that the centered design has full column rank.

Deriving the Solution and Its Limits

The ridge solution follows by differentiating the centered objective with respect to the slope vector and setting the gradient to zero. This also shows why adding a positive penalty stabilizes the equation.

Write the centered objective.
With a centered design and response, use $Q(\boldsymbol{\beta})=(y_c-X_c\boldsymbol{\beta})^\mathsf{T}(y_c-X_c\boldsymbol{\beta})+\lambda\boldsymbol{\beta}^\mathsf{T}\boldsymbol{\beta}$.

Differentiate.
The gradient is $-2X_c^\mathsf{T}(y_c-X_c\boldsymbol{\beta})+2\lambda\boldsymbol{\beta}$.

Set the gradient to zero.
Rearranging gives $(X_c^\mathsf{T}X_c+\lambda I)\boldsymbol{\beta}=X_c^\mathsf{T}y_c$.

Check whether the system is invertible.
For any nonzero vector $z$, $z^\mathsf{T}(X_c^\mathsf{T}X_c+\lambda I)z=\lVert X_cz\rVert^2+\lambda\lVert z\rVert^2$, which is positive when $ \lambda>0 $. Thus the penalized solution is unique for every positive $ \lambda $.

At $ \lambda=0 $, the normal equations reduce to the ordinary least-squares normal equations. If $X_c$ has full column rank, the solution is $ \widehat{\boldsymbol{\beta}}_{\mathrm{OLS}}=(X_c^\mathsf{T}X_c)^{-1}X_c^\mathsf{T}y_c $. If the design is rank deficient, least squares can still determine fitted values, but there can be multiple slope vectors giving those same fitted values; the displayed inverse then does not exist.

For the large-penalty limit, take a singular value decomposition $X_c=UDV^\mathsf{T}$. Each direction with singular value $d_j>0$ contributes a factor $d_j/(d_j^2+\lambda)$ to the ridge estimate: $$ \widehat{\boldsymbol{\beta}}_\lambda =V\,\operatorname{diag}\!\left(\frac{d_j}{d_j^2+\lambda}\right)U^\mathsf{T}y_c, \qquad \lim_{\lambda\to\infty}\frac{d_j}{d_j^2+\lambda}=0. $$ Therefore every component of the slope vector tends to zero. This argument also applies when the design is rank deficient: zero singular-value directions do not create nonzero contributions to the fitted ridge coefficient vector.

Because the intercept is unpenalized, its formula must be considered separately: $$ \widehat{\beta}_{0,\lambda} =\bar{y}-\bar{\mathbf{x}}^\mathsf{T}\widehat{\boldsymbol{\beta}}_\lambda \quad\Longrightarrow\quad \lim_{\lambda\to\infty}\widehat{\beta}_{0,\lambda}=\bar{y}. $$ For any fixed predictor vector, the resulting prediction approaches $ \bar{y} $, a constant. If predictors have been centered, the intercept already equals $ \bar{y} $ at every penalty value; without centering, it generally changes as the slopes shrink.

Worked Example: Following Slopes and Intercept as the Penalty Grows

A hypothetical model uses the predictor values $x=(0,1,2)$ and responses $y=(1,3,5)$. Fit the ridge line for $ \lambda=0 $ and $ \lambda=2 $, then find its limiting intercept and prediction as $ \lambda\to\infty $.

Method. Center the data: $ \bar{x}=1 $, $ \bar{y}=3 $, $x_c=(-1,0,1)$, and $y_c=(-2,0,2)$. The centered predictor sum of squares is $2$, and $x_c^\mathsf{T}y_c=2+2=4$. Thus $ \widehat{\beta}_\lambda=4/(2+\lambda) $, while the uncentered intercept is $ \widehat{\beta}_{0,\lambda}=3-\widehat{\beta}_\lambda $.

At $ \lambda=0 $, the slope is $4/2=2$, and the intercept is $3-2=1$. The fitted line is $1+2x$, giving predictions $1,3,5$.

At $ \lambda=2 $, the slope is $4/4=1$, and the intercept is $3-1=2$. The fitted line is $2+x$, giving predictions $2,3,4$. The response mean remains 3, but the intercept is not 3 at this finite penalty because the predictor mean is not zero.

As $ \lambda\to\infty $, $4/(2+\lambda)\to0$, so the intercept approaches 3 and the prediction approaches 3 for each fixed $x$. The common mistake is to claim the intercept stays at its zero-penalty value of 1; in fact, it approaches the response mean because it is unpenalized.

Rank Deficiency: Zero Penalty May Not Select One Coefficient Vector

When predictors are linearly dependent, the zero-penalty endpoint requires care. The least-squares fitted values may be unique even when the slopes are not. For every positive penalty, ridge has a unique solution; as the penalty tends to zero from above, ridge selects the least-squares coefficient vector with the smallest squared length. That limiting vector is not a claim that ordinary least squares has only one solution.

Worked Example: Identical Predictors at Zero and Positive Penalties

In a hypothetical model, two centered predictors are identical: $x_1=x_2=(-1,0,1)$. The centered response is $y_c=(-2,0,2)$, and the response mean is 3. Find the least-squares constraint at $ \lambda=0 $, the ridge coefficients at $ \lambda=4 $, and the limits as $ \lambda\to0^+ $ and $ \lambda\to\infty $.

Method. Each predictor has squared length $2$, their cross-product is $2$, and each has cross-product $4$ with $y_c$. The normal equations are $$ \begin{pmatrix}2+\lambda&2\\2&2+\lambda\end{pmatrix} \begin{pmatrix}\beta_1\\\beta_2\end{pmatrix} = \begin{pmatrix}4\\4\end{pmatrix}. $$

At $ \lambda=0 $, both equations reduce to $2\beta_1+2\beta_2=4$, or $ \beta_1+\beta_2=2 $. There are infinitely many coefficient pairs, but all give the same fitted centered values because the predictors are identical.

For $ \lambda>0 $, symmetry gives $ \beta_1=\beta_2=b $. Substituting into either equation gives $(4+\lambda)b=4$, so $ b=4/(4+\lambda) $. At $ \lambda=4 $, each coefficient is $4/8=0.5$, the slope sum is 1, and the intercept is 3. As $ \lambda\to0^+ $, both coefficients approach 1, the minimum-length solution among pairs whose sum is 2. As $ \lambda\to\infty $, both approach zero and the intercept stays 3.

Check. At $ \lambda=4 $, the fitted values are $3-0.5-0.5=2$, $3$, and $3+0.5+0.5=4$. They are the centered-response fits shifted by the mean: $1,3,5$ would be the zero-penalty fits, whereas the penalized fits are $2,3,4$. The common trap is to report a unique ordinary least-squares coefficient pair even though the predictors are identical.

Interpreting the Endpoints Correctly

The endpoint results are useful for checking calculations and understanding what the model can express. At zero penalty, a full-rank ridge model agrees with ordinary least squares for the same design and response. If the design is rank deficient, fitted values can still agree with least squares, but coefficient uniqueness cannot be assumed. At the other endpoint, a large penalty suppresses predictor effects rather than selecting a preferred predictor: the model approaches an intercept-only prediction.

In applied work, the exact limiting model may be a poor predictive choice. It discards all information in the predictors, even if some are genuinely informative. The limit describes the behavior of the fitted coefficients, not a recommendation to use an arbitrarily large penalty. Model assessment should still be based on a suitable validation design and evaluation metric.

The constant-prediction conclusion depends on the standard objective specified here: an intercept is present and not penalized, and the predictors and responses are finite and fixed. If the intercept is omitted or included in the penalty, the limiting behavior changes. Also, for large but finite $ \lambda $, slopes are generally small rather than exactly zero; exact zero slopes are characteristic of other penalty behavior, not a general ridge endpoint at finite $ \lambda $.

Key takeaway: With an unpenalized intercept, ridge at $ \lambda=0 $ is ordinary least squares when the coefficient solution is unique, while as $ \lambda\to\infty $, the slopes approach zero and the intercept approaches $ \bar{y} $. In rank-deficient designs, distinguish the unique penalized solution from potentially nonunique zero-penalty coefficients.

Check Your Understanding

For each hypothetical model, use the unpenalized-intercept ridge objective defined above. Each question asks for a limiting or endpoint result; show the centering and rank check in your working.

A centered predictor has squared length 5 and centered response cross-product 15. Find its slope at $ \lambda=0 $, and state whether the limiting slope as $ \lambda\to\infty $ is zero. (Answer: $3$ at zero penalty; $0$ as the penalty tends to infinity.)
A single predictor has values $0,2,4$, and the responses are $4,6,8$. Find the intercept and slope at $ \lambda=0 $, then state the limiting intercept and prediction as $ \lambda\to\infty $. (Answer: zero-penalty slope $1$, intercept $4$; limiting intercept and prediction $6$.)
Two centered predictors are orthogonal, each with squared length 4, and their respective cross-products with the centered response are 8 and $-4$. Find both slopes at $ \lambda=0 $ and at $ \lambda=4 $. (Answer: at zero, $2,-1$; at $ \lambda=4 $, $1,-0.5$.)
Two centered predictors are identical and each has squared length 3; each has cross-product 6 with the centered response. State the zero-penalty constraint on the slopes and the ridge solution at $ \lambda=6 $. (Answer: $ \beta_1+\beta_2=2 $ at zero; at $ \lambda=6 $, $ \beta_1=\beta_2=0.5 $.)
A full-rank centered design has response mean 12. For a fixed predictor vector, suppose its ridge slope vector tends to zero as $ \lambda\to\infty $. What happens to the intercept and prediction? Would the same conclusion follow if the intercept were also penalized? (Answer: both intercept and prediction approach $12$; not necessarily if the intercept is penalized.)

What You'll Learn