Deriving Ridge Shrinkage in a Singular-Value Basis

Why Use a Singular-Value Basis?

Ridge regression is often introduced through its matrix formula, but that formula can hide what the penalty does to different directions in the predictor space. The singular-value decomposition (SVD) makes the effect explicit: directions associated with large singular values are retained more strongly, while directions associated with small singular values are shrunk more heavily. A zero singular value identifies a direction that the data cannot distinguish at all.

The calculations below concern the ridge objective itself. No probability distribution for the response or errors is required to minimize this objective. When the linear model is given a stochastic interpretation, assumptions about errors are needed for inference, but not for the SVD derivation here.

Definition: For a design matrix $X$ with $n$ rows and $p$ columns, the singular-value decomposition is $X=U\Sigma V^\top$, where the columns of $U$ and $V$ are orthonormal and the diagonal entries of $\Sigma$ are the singular values. The positive singular values $d_1,\ldots,d_r$ correspond to directions that $X$ can identify; $r$ is the rank of $X$. A zero singular value corresponds to a direction in the null space of $X$.

Ridge Regression in SVD Coordinates

Assume that the response vector $y$ and the columns of $X$ have been centered, or that an intercept is handled separately and is not penalized. For a fixed $\lambda>0$, define the ridge coefficient estimate as the minimizer of the residual sum of squares plus a squared-coefficient penalty:

\hat{\beta}_{\lambda} =\arg\min_{\beta\in\mathbb{R}^p} \left\{\|y-X\beta\|^2+\lambda\|\beta\|^2\right\}.

This convention puts $\lambda$ beside the unscaled residual sum of squares. If instead the objective is $n^{-1}\|y-X\beta\|^2+\lambda\|\beta\|^2$, its equivalent penalty in the displayed objective is $n\lambda$. A calculation must use the convention specified in the problem.

Write the SVD using orthonormal vectors $u_j$ and $v_j$, so that $Xv_j=d_j u_j$ for each positive singular value $d_j$. Define the response coordinate $c_j=u_j^\top y$ and the coefficient coordinate $z_j=v_j^\top\beta$. The response component perpendicular to the column space of $X$ cannot be fitted by any coefficient vector. Within each identifiable direction, the problem reduces to choosing $z_j$ to balance the coordinate residual against the penalty.

Theorem: For $\lambda>0$, the ridge coefficient estimate and fitted response are $$ \hat{\beta}_{\lambda} =\sum_{j=1}^{r}\frac{d_j}{d_j^2+\lambda}(u_j^\top y)v_j, \qquad \hat{y}_{\lambda}=X\hat{\beta}_{\lambda} =\sum_{j=1}^{r}\frac{d_j^2}{d_j^2+\lambda}(u_j^\top y)u_j. $$ Every coefficient coordinate in the null space of $X$ is zero. Response components perpendicular to the column space of $X$ also receive fitted value zero.

The coefficient formula is written in the original predictor coordinates, while the fitted-value formula is written in the response directions $u_j$. In the $v_j$ direction, the unpenalized coefficient coordinate is $c_j/d_j$. Ridge multiplies that coordinate by $d_j^2/(d_j^2+\lambda)$. The same factor multiplies the unpenalized fitted response coordinate $c_j$. It is therefore the coefficient coordinate shrinkage factor, not the coefficient multiplier $d_j/(d_j^2+\lambda)$ when starting directly from $c_j$.

Why the Shrinkage Formula Holds

The SVD changes coordinates without changing Euclidean lengths: $U$ and $V$ are orthogonal. Let $z=V^\top\beta$ and $c=U^\top y$. For a direction with singular value $d_j$, the predicted response coordinate is $d_jz_j$, and its contribution to the objective is $(c_j-d_jz_j)^2+\lambda z_j^2$. Different directions contribute separate squared terms, so each coordinate can be minimized independently.

Minimize one coordinate.
Differentiate with respect to $z_j$: $-2d_j(c_j-d_jz_j)+2\lambda z_j=0$. Hence $(d_j^2+\lambda)z_j=d_jc_j$, so $z_j=d_jc_j/(d_j^2+\lambda)$.

Recover coefficients and fitted values.
Since $\beta=Vz$, the coefficient contribution is $z_jv_j$. Multiplying by $X$ gives $X(z_jv_j)=d_jz_ju_j=[d_j^2/(d_j^2+\lambda)]c_ju_j$, which yields the stated formulas.

This derivation also settles the zero-singular-value case. If $d_j=0$, changing $z_j$ does not change $X\beta$, so the corresponding objective term is $c_j^2+\lambda z_j^2$, with minimum at $z_j=0$ when $\lambda>0$. If $p>n$, there can also be coefficient directions beyond the response coordinates; these lie in the null space and likewise have minimum-penalty coordinate zero. Conversely, a response component perpendicular to the column space of $X$ is a residual that ridge cannot remove.

Worked Calculations

Worked Example: Two Identifiable Directions

Suppose a centered design has orthonormal singular directions $u_1,u_2$ and $v_1,v_2$, with singular values $d_1=3$ and $d_2=1$. The response coordinates are $u_1^\top y=6$ and $u_2^\top y=2$. Using the objective $\|y-X\beta\|^2+\lambda\|\beta\|^2$, calculate ridge coefficient and fitted-value coordinates for $\lambda=3$. No error distribution is assumed or needed.

Method. Apply the SVD-coordinate formulas direction by direction. For the first coefficient coordinate, $z_1=3(6)/(3^2+3)=18/12=1.5$. For the second, $z_2=1(2)/(1^2+3)=2/4=0.5$.

The coefficient shrinkage factors relative to the unpenalized coordinates are $9/(9+3)=0.75$ and $1/(1+3)=0.25$. Indeed, the unpenalized coordinates are $6/3=2$ and $2/1=2$, and multiplying by the factors gives $1.5$ and $0.5$.

The fitted response coordinates are $3(1.5)=4.5$ and $1(0.5)=0.5$. Equivalently, the fitted-value formula gives $0.75(6)=4.5$ and $0.25(2)=0.5$. Check: both calculations agree. The method avoids confusing the coefficient multiplier $d_j/(d_j^2+\lambda)$ with the shrinkage factor $d_j^2/(d_j^2+\lambda)$.

Worked Example: A Zero Singular Value

Let $X=\operatorname{diag}(2,0,0)$, $y=(5,3,4)^\top$, and $\lambda=1$, using the unscaled ridge objective. The first coordinate is identifiable; the second and third coefficient directions are in the null space of $X$. Find the ridge coefficients, fitted response, and residual. The calculation is deterministic and assumes no probability distribution.

The singular values are $2,0,0$, and the response coordinates in the left singular basis are $5,3,4$. For the positive singular value, $z_1=2(5)/(2^2+1)=10/5=2$. Each zero-singular-value coefficient coordinate minimizes $c_j^2+z_j^2$, so $z_2=z_3=0$. Thus $\hat{\beta}_\lambda=(2,0,0)^\top$.

Multiplication by $X$ gives $\hat{y}_\lambda=(4,0,0)^\top$, and the residual is $y-\hat{y}_\lambda=(1,3,4)^\top$. Check: $X\hat{\beta}_\lambda=(4,0,0)^\top$, as required. The second and third response entries remain residual; zero coefficient directions do not allow them to be fitted.

Worked Example: Converting From SVD Coordinates to Original Coefficients

Suppose $U=I$, the singular values are $2$ and $1$, and $V=2^{-1/2}\begin{pmatrix}1&1\\1&-1\end{pmatrix}$. Let $y=(4,2)^\top$ and use $\lambda=1$ in the unscaled objective. Find the ridge coefficients in the original predictor coordinates and the fitted response. No distributional assumptions are needed.

Since $U=I$, the response coordinates are $c_1=4$ and $c_2=2$. The SVD coefficient coordinates are $z_1=2(4)/(4+1)=8/5$ and $z_2=1(2)/(1+1)=1$.

Transform back using $\hat{\beta}_\lambda=Vz$: $\hat{\beta}_\lambda=2^{-1/2}(13/5,3/5)^\top=(13/(5\sqrt{2}),3/(5\sqrt{2}))^\top$. The fitted response coordinates are $2(8/5)=16/5$ and $1(1)=1$, so $\hat{y}_\lambda=(16/5,1)^\top$.

Check: Here $X=U\Sigma V^\top=\Sigma V^\top$. Multiplying this $X$ by the displayed coefficient vector gives $(16/5,1)^\top$. The transformation by $V$ is essential: $z$ gives coordinates along singular vectors, not generally the original predictor coefficients. The calculation avoids treating SVD coordinates as if they were already in the original predictor basis.

Rank Deficiency and the Zero-Penalty Limit

For every $\lambda>0$, the ridge objective has a unique minimizer, including when $X$ is rank deficient. The penalty makes changes in null-space coefficient coordinates costly even though those changes do not affect the fitted response. Ridge therefore selects zero for each such coordinate.

At $\lambda=0$, the objective is ordinary least squares. If $X$ is rank deficient, there can be multiple coefficient vectors with the same least-squares fit: adding any vector in the null space of $X$ leaves $X\beta$ unchanged. As $\lambda$ decreases toward zero through positive values, the ridge estimate converges to the least-squares solution with minimum Euclidean norm. Its fitted response is the projection of $y$ onto the column space of $X$; the component perpendicular to that space remains residual.

\lim_{\lambda\downarrow0}\hat{\beta}_{\lambda} =\sum_{j=1}^{r}\frac{u_j^\top y}{d_j}v_j, \qquad \lim_{\lambda\downarrow0}\hat{y}_{\lambda} =\sum_{j=1}^{r}(u_j^\top y)u_j.

This limit does not mean that a zero singular value can be divided by. The coefficient sum includes only positive singular values. Null-space coefficients remain zero in the minimum-norm limit, while response directions outside the column space remain unfitted.

Common Exam Traps

Using the wrong shrinkage factor. The multiplier from response coordinate $c_j$ to coefficient coordinate $z_j$ is $d_j/(d_j^2+\lambda)$. The factor multiplying the unpenalized coefficient coordinate $c_j/d_j$, and the factor multiplying the unpenalized fitted response coordinate $c_j$, is $d_j^2/(d_j^2+\lambda)$.
Assigning a coefficient to a null direction. For $d_j=0$ and $\lambda>0$, the minimizing coefficient coordinate is zero; it is not obtained by dividing by $d_j$.
Fitting response components outside the column space. Ridge changes the fitted values within the column space but cannot fit its orthogonal complement.
Changing the penalty convention mid-calculation. An objective with mean squared error uses a different numerical penalty from an objective with an unscaled residual sum of squares. Translate the convention before inserting $\lambda$.
Ignoring the predictor basis. The $z_j$ are coefficients along the right singular vectors. Convert with $\beta=Vz$ before reporting original predictor coefficients.

Key takeaway: In singular direction $j$, ridge multiplies the unpenalized fitted response by $d_j^2/(d_j^2+\lambda)$. Large singular values are shrunk less than small ones. A zero singular value produces zero ridge coefficient for positive $\lambda$, and contributes no fitted response.

Check Your Understanding

Use the unscaled objective $\|y-X\beta\|^2+\lambda\|\beta\|^2$ unless a question specifies a different convention. All response coordinates below are given in the left singular-vector basis.

A design has singular values $4,2$, response coordinates $3,-2$, and $\lambda=4$. Find both ridge coefficient coordinates and fitted response coordinates. (Answer: coefficient coordinates $0.6,-0.5$; fitted coordinates $2.4,-1$.)
A rank-one design has singular values $3,0$, response coordinates $6,5$, and $\lambda=3$. Find the ridge coefficient coordinates and fitted response coordinates. (Answer: coefficient coordinates $1.5,0$; fitted coordinates $4.5,0$.)
A design has singular values $1,3$, response coordinates $2,-6$, and $\lambda=1$. Find the coefficient and fitted response coordinates, and state the shrinkage factor in each direction. (Answer: coefficient coordinates $1,-1.8$; fitted coordinates $1,-5.4$; shrinkage factors $0.5,0.9$.)
An objective is $n^{-1}\|y-X\beta\|^2+0.2\|\beta\|^2$, with $n=5$. In one singular direction, $d=2$ and $u^\top y=4$. Find the ridge coefficient and fitted response coordinate. (Answer: equivalent unscaled penalty $\lambda=1$; coefficient coordinate $1.6$; fitted response coordinate $3.2$.)
A rank-deficient design has one positive singular value $2$, with response coordinate $4$ in that direction, and response component $3$ perpendicular to the column space. As $\lambda$ decreases to zero, give the minimum-norm coefficient coordinate, fitted coordinate in the identifiable direction, and the perpendicular fitted component. (Answer: coefficient coordinate $2$; identifiable fitted coordinate $4$; perpendicular fitted component $0$, leaving perpendicular residual $3$.)

What You'll Learn