Tutorials › Exam PA - Predictive Analytics › Ridge Shrinkage Through Singular Values

Home  /  Tutorials  /  Exam PA / Penalties and Ridge Regression

Exam PA · Penalties and Ridge Regression · Tutorial 400 of 966

Ridge Shrinkage Through Singular Values

Use the singular value decomposition to express ridge coefficients and fitted values direction by direction, then calculate and interpret the amount of shrinkage.

Intermediate 11 min read

What You'll Learn

  • State the centered linear-model setup and ridge objective used for the derivation
  • Express ridge coefficients and fitted values in singular-vector coordinates
  • Calculate how each singular value and the penalty determine a shrinkage factor
  • Distinguish shrinkage of coefficient components from shrinkage of fitted-response components
  • Identify what happens in zero-singular-value directions and why ridge is stable
  • Avoid confusing large and small singular values or comparing them without regard to the penalty

Why Examine Ridge by Direction?

Ridge regression does not generally shrink every feature coefficient by the same percentage. Its effect is clearer when the predictor matrix is described in terms of directions: some combinations of predictors are strongly represented in the data, while others are weakly represented. The singular value decomposition (SVD) identifies these directions and measures their strength.

This view makes a practical fact precise: for a fixed positive ridge penalty, a direction associated with a small singular value is shrunk more than a direction associated with a large singular value. The comparison is relative to the penalty, so the size of a singular value alone does not say how much shrinkage occurs.

Definition: For a predictor matrix \(X\), a singular value decomposition is a factorization \(X=UDV^\mathsf{T}\), where the columns of \(U\) and \(V\) are orthonormal and \(D\) contains the singular values. The positive singular values \(d_1,\ldots,d_r\) measure how strongly \(X\) maps the corresponding right-singular directions \(v_j\) into the left-singular directions \(u_j\): \(Xv_j=d_j u_j\).

Model and Ridge Objective

Use the linear model \(y=X\beta+\varepsilon\), with \(n\) observations, response vector \(y\), design matrix \(X\), and coefficient vector \(\beta\). To focus on the SVD calculation, assume the response and predictor columns have been centered, and consider the slope coefficients without an intercept. The ridge estimate minimizes residual sum of squares plus a squared-coefficient penalty:

$$ \widehat{\beta}_{\lambda} =\arg\min_{\beta}\left\{\|y-X\beta\|_2^2+\lambda\|\beta\|_2^2\right\}, \qquad \lambda>0. $$

Here \(\lambda\) is the penalty parameter. A larger \(\lambda\) increases the cost of nonzero coefficients. This tutorial uses the unscaled sum of squares in the objective; if an objective is instead written with residual sum of squares divided by \(n\), the numerical penalty must be interpreted on that scale. The SVD formulas below apply to the stated objective.

Let \(r\) be the rank of \(X\), and write its positive-singular-value part as \(X=U_rD_rV_r^\mathsf{T}\). Thus \(D_r\) has diagonal entries \(d_j>0\) for \(j=1,\ldots,r\). Any remaining coefficient directions lie in the null space of \(X\): changing \(\beta\) in one of those directions does not change \(X\beta\).

Deriving the Direction-Specific Formula

Represent the part of \(\beta\) in the row space of \(X\) as \(V_r\gamma\), where \(\gamma_j\) is its coefficient in direction \(v_j\). A component of \(\beta\) in the null space can only increase the penalty without changing the fitted values, so a ridge minimizer has no such component. Also, \(X V_r\gamma=U_rD_r\gamma\). Since the columns of \(U_r\) are orthonormal, the residual contributions along these directions separate.

1
Project the response onto the left-singular directions.
Define \(z_j=u_j^\mathsf{T}y\). The squared residual in direction \(u_j\) is \((z_j-d_j\gamma_j)^2\).
2
Write the objective by direction.
Orthogonality gives a sum of terms \((z_j-d_j\gamma_j)^2+\lambda\gamma_j^2\), plus a residual component orthogonal to the columns of \(X\) that does not depend on \(\gamma\).
3
Minimize each term.
Differentiating with respect to \(\gamma_j\) gives \(-2d_j(z_j-d_j\gamma_j)+2\lambda\gamma_j=0\), so \((d_j^2+\lambda)\gamma_j=d_jz_j\).

Because \(d_j^2+\lambda>0\), each direction has a unique minimizing coefficient. Putting the directions back together gives the ridge coefficients and fitted response:

$$ \widehat{\gamma}_{\lambda,j} =\frac{d_j}{d_j^2+\lambda}u_j^\mathsf{T}y, \qquad \widehat{\beta}_{\lambda} =\sum_{j=1}^{r}\frac{d_j}{d_j^2+\lambda}(u_j^\mathsf{T}y)v_j, $$
$$ X\widehat{\beta}_{\lambda} =\sum_{j=1}^{r}\frac{d_j^2}{d_j^2+\lambda}(u_j^\mathsf{T}y)u_j. $$

To interpret these equations, first consider ordinary least squares in a direction with \(d_j>0\). Its coefficient coordinate is \(\widehat{\gamma}_{\mathrm{OLS},j}=u_j^\mathsf{T}y/d_j\), so ridge multiplies that coefficient coordinate by

$$ s_j(\lambda)=\frac{d_j^2}{d_j^2+\lambda}. $$

The fitted-response component along \(u_j\), whose unpenalized value is \(u_j^\mathsf{T}y\), is multiplied by the same factor. This is why \(s_j(\lambda)\) is called the direction-specific shrinkage factor. For \(\lambda>0\), it lies strictly between 0 and 1.

The ordering of shrinkage follows directly from the formula. Holding \(\lambda>0\) fixed, \(s_j\) increases as \(d_j^2\) increases; equivalently, differentiating \(s\) with respect to \(d^2\) gives \(\lambda/(d^2+\lambda)^2>0\). Thus larger-singular-value directions retain a larger fraction of their unpenalized coefficient and fitted-response components. If \(d_j=0\), the data contain no information about that coefficient direction. Ridge sets its coefficient to zero, while the direction contributes nothing to \(X\widehat{\beta}_{\lambda}\).

Worked Example: Shrinkage in Two SVD Directions

Suppose a centered design matrix has two positive singular values \(d_1=4\) and \(d_2=1\). The response projections are \(u_1^\mathsf{T}y=8\) and \(u_2^\mathsf{T}y=3\). For the ridge objective above, use \(\lambda=4\). Find the shrinkage factors, the ridge coefficient coordinates, and the fitted-response coordinates. State what the comparison shows.

Method. Apply the SVD formulas separately in each direction. For direction 1, \(d_1^2=16\), so \(s_1=16/(16+4)=16/20=0.8\). Its OLS coefficient coordinate is \(8/4=2\); its ridge coordinate is \(0.8(2)=1.6\). Equivalently, the ridge formula gives \(4(8)/(16+4)=32/20=1.6\). Its fitted-response coordinate is \(4(1.6)=6.4\), also \(0.8(8)=6.4\).

For direction 2, \(d_2^2=1\), so \(s_2=1/(1+4)=0.2\). Its OLS coefficient coordinate is \(3/1=3\); its ridge coordinate is \(0.2(3)=0.6\), or \(1(3)/(1+4)=0.6\) from the ridge formula. Its fitted-response coordinate is \(1(0.6)=0.6\), also \(0.2(3)=0.6\).

The ridge coefficient vector is therefore \(1.6v_1+0.6v_2\), and its fitted response has components \(6.4u_1+0.6u_2\) in these two directions. The smaller singular value has the smaller shrinkage factor, so its components are reduced more strongly. Check: the coefficient coordinates are respectively \(0.8\) and \(0.2\) times their OLS values, as required.

What the Factor Says About Stability

The denominator \(d_j^2+\lambda\) explains why a positive penalty stabilizes the calculation. When a singular value is small, the corresponding OLS coefficient \(u_j^\mathsf{T}y/d_j\) can be large or sensitive to changes in the response. Ridge replaces division by \(d_j\) with the coefficient multiplier \(d_j/(d_j^2+\lambda)\). For fixed \(\lambda>0\), this multiplier remains finite, including as \(d_j\) approaches zero. At exactly zero, it is zero.

The amount of shrinkage depends on the ratio of \(d_j^2\) to \(\lambda\), not on a universal cutoff. If \(d_j^2\) is much larger than \(\lambda\), then \(s_j\) is near 1. If \(d_j^2\) is much smaller than \(\lambda\), then \(s_j\) is near 0. If \(d_j^2=\lambda\), then \(s_j=1/2\). These comparisons describe the chosen design matrix and penalty; changing predictor units can change the singular values and therefore the shrinkage pattern.

Worked Example: Comparing Strong and Weak Directions

For a second hypothetical design, suppose the positive singular values are \(d_1=10\) and \(d_2=2\), the response projections are \(z_1=5\) and \(z_2=6\), and \(\lambda=4\). Calculate each shrinkage factor, coefficient coordinate, and fitted-response coordinate. Round only final decimal answers to four decimal places.

Method. For direction 1, \(d_1^2=100\), so \(s_1=100/(100+4)=25/26\). The OLS coefficient coordinate is \(5/10=0.5\). Thus the ridge coefficient coordinate is \((25/26)(0.5)=25/52\), which is \(0.4808\) to four decimals. The fitted-response coordinate is \(10(25/52)=125/26\), which is \(4.8077\); it also equals \((25/26)(5)\).

For direction 2, \(d_2^2=4\), so \(s_2=4/(4+4)=1/2\). The OLS coefficient coordinate is \(6/2=3\). The ridge coordinate is \((1/2)(3)=1.5000\). Its fitted-response coordinate is \(2(1.5)=3.0000\), or \((1/2)(6)=3.0000\).

The first direction retains \(25/26\) of its unpenalized component, while the second retains one-half. The comparison is consistent with \(d_1>d_2\): for the same penalty, the larger singular value means a larger shrinkage factor. Check: multiplying each ridge coefficient coordinate by its singular value recovers the fitted-response coordinate.

Worked Example: A Weak Direction and a Nearly Unidentified Coefficient

Suppose \(d_1=3\), \(d_2=0.2\), \(u_1^\mathsf{T}y=6\), \(u_2^\mathsf{T}y=1\), and \(\lambda=0.16\). Find the OLS and ridge coefficient coordinates in both directions, then compare their fitted-response coordinates. Give final decimal answers to four places.

Method. In direction 1, \(d_1^2=9\) and \(s_1=9/(9+0.16)=9/9.16=225/229\). The OLS coefficient coordinate is \(6/3=2\). The ridge coefficient coordinate is \(2(225/229)=450/229=1.9651\). The fitted-response coordinate is \(3(450/229)=1350/229=5.8952\); the unpenalized coordinate was 6.

In direction 2, \(d_2^2=0.04\), and \(s_2=0.04/(0.04+0.16)=0.04/0.20=0.2\). The OLS coefficient coordinate is \(1/0.2=5\), whereas the ridge coordinate is \(0.2(5)=1\). Directly, the ridge formula gives \(0.2(1)/(0.04+0.16)=1\). The fitted-response coordinate is \(0.2(1)=0.2\), compared with the unpenalized coordinate 1.

The ridge estimate substantially reduces the coefficient in the weak direction, from 5 to 1, while reducing the strong-direction coefficient only slightly. This is precisely the intended direction-specific effect. Check: the fitted-response coordinates are \(5.8952\) and \(0.2000\), equal to their original response projections multiplied by \(225/229\) and \(0.2\), respectively.

Common Exam Trap: Reversing the Shrinkage

A frequent error is to treat \(d_j^2/(d_j^2+\lambda)\) as though it were the amount removed, rather than the fraction retained. For example, with \(d=2\) and \(\lambda=4\), the factor is \(4/(4+4)=0.5\). Calling this “50% removed” and retaining only half the response component happens to give the right number in this special case, which can conceal the mistaken interpretation.

For \(d=4\) and the same \(\lambda=4\), the factor is \(16/20=0.8\). The component retained is \(80\%\), and the component removed is \(1-0.8=0.2\), or \(20\%\). Incorrectly treating \(0.8\) as the amount removed would instead retain \(20\%\), reversing the conclusion. Always label \(s_j\) as the retained fraction; the removed fraction is \(1-s_j=\lambda/(d_j^2+\lambda)\).

Key takeaway: In SVD coordinates, ridge multiplies the coefficient and fitted-response components in direction \(j\) by \(d_j^2/(d_j^2+\lambda)\). A smaller singular value gives a smaller retained fraction for the same positive penalty; a zero singular value gives a zero ridge coefficient in that unidentifiable direction.

Check Your Understanding

For each problem, use the stated ridge objective with \(\lambda>0\). Give final decimal answers to four places where needed.

  1. A design has singular value \(d=5\), response projection \(u^\mathsf{T}y=10\), and \(\lambda=5\). Find the shrinkage factor, OLS coefficient coordinate, ridge coefficient coordinate, and ridge fitted-response coordinate. (Answer: \(s=25/30=0.8333\); OLS coordinate \(=2\); ridge coordinate \(=1.6667\); fitted-response coordinate \(=8.3333\).)
  2. Two directions have singular values 3 and 1, and \(\lambda=2\). Which direction is shrunk more, and what is each retained fraction? (Answer: The \(d=1\) direction is shrunk more; the retained fractions are \(9/11=0.8182\) and \(1/3=0.3333\), respectively.)
  3. In one direction, \(d=0.5\), \(u^\mathsf{T}y=2\), and \(\lambda=0.75\). Calculate the OLS and ridge coefficient coordinates and the ridge fitted-response coordinate. (Answer: OLS coordinate \(=4\); \(s=0.25/(0.25+0.75)=0.25\); ridge coordinate \(=1\); fitted-response coordinate \(=0.5\).)
  4. A direction has \(d^2=6\). What penalty gives a retained fraction of \(0.75\)? Then find the retained fraction if the penalty is doubled. (Answer: Solving \(6/(6+\lambda)=0.75\) gives \(\lambda=2\); doubling it to 4 gives \(6/(6+4)=0.6\).)
  5. A design has a zero singular value in one coefficient direction. Explain the effect of ridge with \(\lambda>0\) on that coefficient and on fitted values in that direction. (Answer: Ridge sets the coefficient component to zero because it increases the penalty without reducing residual error; the direction contributes zero to the fitted values.)

Previous

Next

Back to Tutorials
Clinical Biostats