Ridge Regression as Directional Shrinkage

Why Ridge Shrinkage Depends on Direction

When predictors are strongly related, many combinations of their coefficients can produce very similar fitted values. Least squares can then be sensitive to small changes in the data: the fitted predictions may change little while the individual coefficient estimates change substantially. Ridge regression addresses this instability by adding a penalty on coefficient size to the least-squares objective.

The penalty acts on the coefficient vector, not directly on the fitted values. As a result, ridge does not generally shrink every coefficient by the same percentage. How much the estimates change depends on how the predictors relate to one another and to the response. The derivation below gives the centered coefficient solution and shows why adding $\lambda I$ stabilizes the equations.

Definition: For a response vector $\mathbf y$ and a predictor matrix $\mathbf X$ whose columns have been centered, the centered ridge estimate at penalty value $\lambda \geq 0$ minimizes $$ \|\mathbf y-\mathbf X\boldsymbol{\beta}\|^2+\lambda\|\boldsymbol{\beta}\|^2. $$ Here $\boldsymbol{\beta}$ is the vector of predictor coefficients, $\|\cdot\|$ is Euclidean length, and $I$ denotes the identity matrix with one row and column per predictor. The penalty is on the slopes; an intercept is not included in it.

The usual working regression model describes the response as a linear mean plus an error, with an intercept and predictor slopes. Centering each predictor and the response removes the intercept from the slope calculation. The ridge computation itself is the minimization problem in the definition; it does not require a normal-error assumption. For the penalty to treat predictor coefficients comparably, their scales must be handled appropriately, as in the earlier material on scaling predictors before penalization.

Deriving the Centered Ridge Solution

Write $Q(\boldsymbol{\beta})$ for the objective being minimized. Expanding the squared residual term gives

Q(\boldsymbol{\beta}) =(\mathbf y-\mathbf X\boldsymbol{\beta})^\mathsf{T} (\mathbf y-\mathbf X\boldsymbol{\beta}) +\lambda\boldsymbol{\beta}^\mathsf{T}\boldsymbol{\beta}.

Differentiate with respect to the coefficient vector and set the gradient equal to zero. The derivative of the residual term is $-2\mathbf X^\mathsf{T}\mathbf y+2\mathbf X^\mathsf{T}\mathbf X\boldsymbol{\beta}$; the derivative of the penalty is $2\lambda\boldsymbol{\beta}$. Thus the minimizing coefficients satisfy

-2\mathbf X^\mathsf{T}\mathbf y +2\mathbf X^\mathsf{T}\mathbf X\boldsymbol{\beta} +2\lambda\boldsymbol{\beta}=\mathbf 0, $$ $$ (\mathbf X^\mathsf{T}\mathbf X+\lambda I)\boldsymbol{\beta} =\mathbf X^\mathsf{T}\mathbf y.

For $\lambda>0$, the matrix on the left is invertible, even if the predictors are linearly dependent. To see this, take any nonzero vector $\mathbf v$. Then

\mathbf v^\mathsf{T}(\mathbf X^\mathsf{T}\mathbf X+\lambda I)\mathbf v =\|\mathbf X\mathbf v\|^2+\lambda\|\mathbf v\|^2>0.

The final inequality holds because the first term is nonnegative and the second is strictly positive. Therefore the matrix is positive definite and has a unique inverse. The centered ridge solution is

\widehat{\boldsymbol{\beta}}_{\mathrm{ridge}} =(\mathbf X^\mathsf{T}\mathbf X+\lambda I)^{-1} \mathbf X^\mathsf{T}\mathbf y.

At $\lambda=0$, this reduces to the usual least-squares normal equations; the inverse formula applies only when $\mathbf X^\mathsf{T}\mathbf X$ is invertible. With positive $\lambda$, adding $\lambda I$ lifts the zero or very small eigenvalues of $\mathbf X^\mathsf{T}\mathbf X$. This makes the coefficient equations less sensitive to small perturbations, especially when the predictors are nearly redundant. The tradeoff is that the estimates are pulled toward zero.

Worked Example: Shrinkage With Orthogonal Predictors

Suppose two centered, suitably scaled predictor columns are orthogonal and each has squared length 1. Thus $\mathbf X^\mathsf{T}\mathbf X=I$. Suppose their cross-products with the centered response are $\mathbf X^\mathsf{T}\mathbf y=(2,-1)^\mathsf{T}$. Find the ridge coefficients for $\lambda=1$, and compare them with the least-squares coefficients.

The ridge equation is $$ (I+1I)\widehat{\boldsymbol{\beta}}_{\mathrm{ridge}} =\begin{pmatrix}2\\-1\end{pmatrix}. $$ Since $I+1I=2I$, the solution is $$ \widehat{\boldsymbol{\beta}}_{\mathrm{ridge}} =\frac{1}{2}\begin{pmatrix}2\\-1\end{pmatrix} =\begin{pmatrix}1\\-0.5\end{pmatrix}. $$

With $\lambda=0$, the least-squares coefficients are $(2,-1)^\mathsf{T}$. Here both coordinates are halved because the two predictor columns have the same squared length and are orthogonal. This special case can make ridge look like a common multiplier on all coefficients, but that is not generally true.

How the Penalty Stabilizes a Difficult Fit

The stabilizing effect is especially visible when two predictors are almost copies of one another. Then the data can provide strong information about their combined contribution but little information about how to divide that contribution between them. The next calculation uses a two-predictor example to show how ridge changes estimates in the poorly determined direction.

Worked Example: Nearly Redundant Predictors

Suppose the centered predictor columns each have squared length 1 and have correlation $0.99$. Let their cross-products with the centered response be $(1,-1)^\mathsf{T}$. Find the least-squares and ridge coefficients for $\lambda=0.1$, and compare the stability of the two systems.

The predictor cross-product matrix and response cross-product vector are $$ \mathbf X^\mathsf{T}\mathbf X= \begin{pmatrix}1&0.99\\0.99&1\end{pmatrix}, \qquad \mathbf X^\mathsf{T}\mathbf y= \begin{pmatrix}1\\-1\end{pmatrix}. $$ The vector $(1,-1)^\mathsf{T}$ is the contrast between the two predictors. Multiplying the cross-product matrix by this vector gives $0.01(1,-1)^\mathsf{T}$. Thus the least-squares equation requires coefficients in that direction to be multiplied by only $0.01$ to produce the observed cross-product vector. The least-squares solution is $$ \widehat{\boldsymbol{\beta}}_{\mathrm{LS}} =\begin{pmatrix}100\\-100\end{pmatrix}, \qquad 0.01\begin{pmatrix}100\\-100\end{pmatrix} =\begin{pmatrix}1\\-1\end{pmatrix}. $$

For $\lambda=0.1$, the ridge system is $$ \begin{pmatrix}1.1&0.99\\0.99&1.1\end{pmatrix} \widehat{\boldsymbol{\beta}}_{\mathrm{ridge}} =\begin{pmatrix}1\\-1\end{pmatrix}. $$ The contrast direction now has multiplier $1.1-0.99=0.11$, so $$ \widehat{\boldsymbol{\beta}}_{\mathrm{ridge}} =\begin{pmatrix}1/0.11\\-1/0.11\end{pmatrix} \approx\begin{pmatrix}9.091\\-9.091\end{pmatrix}. $$ The estimates are still large, but much smaller in magnitude than the least-squares estimates. The matrix eigenvalues are $1.99$ and $0.01$ without the penalty, giving a condition number of $1.99/0.01=199$. After adding $0.1I$, they are $2.09$ and $0.11$, giving a condition number of $2.09/0.11=19.0$. The smaller condition number reflects a less ill-conditioned coefficient system.

The same calculation also illustrates directional shrinkage. Along the contrast direction $(1,-1)^\mathsf{T}$, the response cross-product is divided by $0.11$ rather than $0.01$, greatly reducing the coefficient magnitude. In the common direction $(1,1)^\mathsf{T}$, the corresponding multipliers are $2.09$ with ridge and $1.99$ without it, so the relative change is much smaller. The penalty affects all coefficient directions, but its practical effect is most dramatic where the data provide the weakest information.

More generally, the ridge penalty adds $\lambda\|\boldsymbol{\beta}\|^2$ to the residual sum of squares, so larger penalties favor shorter coefficient vectors. One way to make this monotonic tradeoff precise is to compare two penalty values. Let $\boldsymbol{\beta}_1$ and $\boldsymbol{\beta}_2$ minimize the objective at $0\leq\lambda_1<\lambda_2$, respectively. Write $R(\boldsymbol{\beta})=\|\mathbf y-\mathbf X\boldsymbol{\beta}\|^2$. Optimality gives

R(\boldsymbol{\beta}_1)+\lambda_1\|\boldsymbol{\beta}_1\|^2 \leq R(\boldsymbol{\beta}_2)+\lambda_1\|\boldsymbol{\beta}_2\|^2, $$ $$ R(\boldsymbol{\beta}_2)+\lambda_2\|\boldsymbol{\beta}_2\|^2 \leq R(\boldsymbol{\beta}_1)+\lambda_2\|\boldsymbol{\beta}_1\|^2.

Adding the inequalities and cancelling the residual terms yields

(\lambda_2-\lambda_1) \left(\|\boldsymbol{\beta}_2\|^2-\|\boldsymbol{\beta}_1\|^2\right)\leq 0.

Since $\lambda_2-\lambda_1>0$, the squared coefficient-vector length cannot increase when the penalty increases. This does not mean every individual coefficient must move monotonically toward zero: coefficients can change relative to one another as the fit adjusts. It does establish that the overall coefficient vector becomes no longer as the penalty grows.

Centering and the Intercept

When an intercept is included, it is ordinarily left unpenalized. Centering provides a direct way to calculate the slopes without including the intercept in the ridge system. After estimating the slopes from centered variables, recover the intercept from the sample means:

\widehat{\beta}_0=\bar{y}-\sum_{j=1}^{p}\widehat{\beta}_j\bar{x}_j.

This ensures that the fitted values have the appropriate overall level. Penalizing the intercept instead would make the fitted level depend on the choice of response origin, which is not the intended ridge penalty on predictor effects.

Worked Example: Recovering an Unpenalized Intercept

Consider three hypothetical observations with predictor values $-1,0,1$ and response values $1,2,3$. Fit a one-predictor ridge model with $\lambda=2$, including an unpenalized intercept.

The predictor mean is $\bar{x}=0$, and the response mean is $\bar{y}=2$. The centered predictor values remain $(-1,0,1)$, and the centered responses are $(-1,0,1)$. Therefore $$ \mathbf X^\mathsf{T}\mathbf X=(-1)^2+0^2+1^2=2, \qquad \mathbf X^\mathsf{T}\mathbf y=(-1)(-1)+0(0)+1(1)=2. $$ The centered ridge solution is $$ \widehat{\beta}_{\mathrm{ridge}} =\frac{\mathbf X^\mathsf{T}\mathbf y} {\mathbf X^\mathsf{T}\mathbf X+\lambda} =\frac{2}{2+2}=0.5. $$

Recover the intercept using the sample means: $$ \widehat{\beta}_0=\bar{y}-\widehat{\beta}_{\mathrm{ridge}}\bar{x} =2-(0.5)(0)=2. $$ The fitted line is $\widehat{y}=2+0.5x$, with fitted values $1.5,2,2.5$ at $x=-1,0,1$. The intercept remains at the response mean because the predictor mean is zero; it was not shrunk by the penalty.

Common Mistakes and Practical Meaning

Assuming ridge multiplies every coefficient by the same factor. That happens in special cases such as orthogonal predictors with equal column lengths. With correlated predictors, the coefficient changes depend on the predictor cross-products, as the nearly redundant example shows.
Thinking a positive penalty guarantees exact zeros. Ridge generally makes coefficients smaller, but it does not ordinarily set them exactly to zero. Its purpose here is to stabilize and shrink the fit, not to provide a list of excluded predictors.
Penalizing the intercept by accident. The centered formula penalizes slopes only. With an intercept in the model, center the variables for the slope calculation and then recover the intercept from their means.
Ignoring predictor scales. A squared-coefficient penalty depends on the units of each predictor. The predictors should be scaled appropriately before penalization; otherwise, changing a predictor's units can change how strongly its coefficient is penalized.

Ridge is useful when the goal is a more stable regression fit, especially when predictors overlap in the information they carry. Its central compromise is explicit in the objective: it accepts some increase in residual error in exchange for smaller, less volatile coefficients. The penalty does not make the predictors independent, nor does it prove that a particular coefficient is a causal effect. It changes the estimation problem to control coefficient size.

Key takeaway: For centered predictors and response, ridge solves $(\mathbf X^\mathsf{T}\mathbf X+\lambda I)\widehat{\boldsymbol{\beta}}=\mathbf X^\mathsf{T}\mathbf y$. A positive $\lambda$ makes this system invertible and discourages large coefficient vectors. Because predictor relationships differ by direction, the resulting shrinkage is generally not a common percentage applied separately to every coefficient.

Check Your Understanding

Try these questions without looking back at the worked calculations.

Write the centered ridge objective and derive the normal equations by differentiating with respect to the coefficient vector.
Why is $\mathbf X^\mathsf{T}\mathbf X+\lambda I$ invertible when $\lambda>0$, even if predictor columns are linearly dependent?
In the nearly redundant predictor example, why are the least-squares coefficients large despite the response cross-products being only 1 and $-1$?
What is the role of centering when fitting ridge regression with an intercept, and how is that intercept recovered?
Does increasing $\lambda$ guarantee that every individual coefficient decreases in absolute value? Explain the distinction between coefficient-vector length and an individual coefficient.