Ridge Shrinkage Along Singular Vectors

Why Look at Singular Directions?

Ridge regression shrinks coefficients toward zero, but the amount of shrinkage is not the same in every direction through the predictor space. The singular value decomposition (SVD) makes those directions visible. It shows how much variation in the predictors is available along each direction, and therefore how ridge penalization affects that direction.

This view is useful because a coefficient vector combines many directions. A ridge estimate may preserve components well supported by the predictors while reducing components supported by comparatively little predictor information. The shrinkage factors describe that effect precisely.

Model and Notation

Let \(X\) be a centered \(n \times p\) design matrix and \(y\) a centered response vector. Centering means that each predictor column and the response have had their sample mean subtracted. The intercept is therefore handled separately and is not penalized. For the direction-specific calculations below, use the linear model \(y=X\beta+\varepsilon\), with the design matrix treated as fixed and errors satisfying \(E(\varepsilon)=0\) and \(\operatorname{Var}(\varepsilon)=\sigma^2 I\). The ridge estimates themselves are defined by minimizing the penalized sum of squares; normal errors are not needed for the algebra or the variance results below.

A singular value measures the size of the design matrix’s action in its corresponding right-singular-vector direction: \(Xv_j=d_j u_j\). Equivalently, \(d_j^2\) is an eigenvalue of \(X^\mathsf{T}X\). A small \(d_j^2\) means that changes in coefficients along \(v_j\) produce relatively little variation in the fitted values. The centered design has no information in any null-space direction \(v\) for which \(Xv=0\).

The penalty depends on predictor scaling. As established in the earlier material on scaling predictors before penalization, comparisons of shrinkage across predictors are most useful when scaling is appropriate to the analysis. Here the SVD describes shrinkage for the centered design matrix as actually entered into the ridge fit.

Deriving the Shrinkage Factors

Use the SVD to express the coefficient vector in right-singular-vector coordinates, \(\beta=\sum_{j=1}^r a_jv_j+\beta_0\), where \(\beta_0\) lies in the null space of \(X\). Also define \(q_j=u_j^\mathsf{T}y\), the response’s coordinate along \(u_j\). Because the singular vectors are orthonormal, the penalized objective separates into a sum of terms, one for each singular direction, plus terms unaffected by the coefficients.

For each \(a_j\), differentiate its quadratic term and set the derivative equal to zero. Since \(d_j^2+\lambda>0\), this gives a unique minimizing coordinate. The null-space term is minimized at \(\beta_0=0\).

When \(\lambda=0\) and \(X\) has full column rank, the least-squares coordinate is \(\widehat a_{j,0}=q_j/d_j\). Thus the ridge coefficient in direction \(v_j\) is the least-squares coefficient multiplied by a direction-specific shrinkage factor.

The fitted-value statement follows because \(Xv_j=d_ju_j\): the ridge fit’s coordinate along \(u_j\) is \(d_j\widehat a_{j,\lambda}=d_j^2q_j/(d_j^2+\lambda)\). This distinction matters: the coefficient and fitted value are different quantities, but their shrinkage factors match when each is compared with its least-squares counterpart in the same direction.

For a fixed positive \(\lambda\), the factor approaches 1 as \(d_j^2\) becomes large, and approaches 0 as \(d_j^2\) becomes small. Increasing \(\lambda\) reduces the factor in every direction. If a singular value is exactly zero, the data cannot identify that coefficient direction; ridge sets its null-space coefficient to zero rather than estimating it from the response.

Reading the Effect of Predictor Information

The shrinkage factor depends on the squared singular value relative to \(\lambda\), not on the singular value alone. In particular, a direction is substantially shrunk when \(d_j^2\) is small compared with \(\lambda\). This comparison gives a practical way to understand the role of the tuning parameter without assigning it an interpretation as a universal amount of shrinkage.

Why Shrinkage Can Stabilize Estimates

The small singular values also explain why least-squares coefficient estimates can be variable in weak directions. Under the stated fixed-design model, the response coordinate satisfies \(q_j=d_ja_j+u_j^\mathsf{T}\varepsilon\). The error term has variance \(\sigma^2\), because \(u_j\) is a unit vector and \(\operatorname{Var}(\varepsilon)=\sigma^2I\). The least-squares coefficient divides that error by \(d_j\), so its variance is \(\sigma^2/d_j^2\). Ridge instead multiplies the response coordinate by \(d_j/(d_j^2+\lambda)\).

The expectation shows the cost: ridge generally biases the coefficient toward zero. The variance expression shows a potential benefit: compared with least squares, ridge multiplies the variance in that direction by the square of the shrinkage factor. This is a direction-by-direction bias–variance tradeoff, not a guarantee that every ridge fit predicts better.

Common Mistake and Practical Meaning

A common mistake is to say, “Ridge shrinks every predictor coefficient by the same percentage.” That is generally wrong. Ridge applies a common penalty to the coefficient vector, but in singular-vector coordinates its shrinkage factors are \(d_j^2/(d_j^2+\lambda)\), which vary with the design. These are factors for directions \(v_j\), not necessarily for individual predictors. Each original predictor coefficient can combine contributions from several singular directions.

Another useful caution is that a low singular value identifies a direction of limited predictor information, not automatically a particular variable. The SVD describes combinations of columns of \(X\). The direction-specific result explains how ridge acts on those combinations; it does not by itself identify which named predictor should be removed or establish a causal interpretation.