Ridge Shrinkage Along Design-Matrix Directions

Why shrinkage depends on direction

Ridge regression does not shrink every feature coefficient by the same amount when viewed in the original predictor coordinates. Its action is simpler in directions formed from the design matrix’s singular vectors: each direction in the fitted signal is multiplied by a factor determined by its singular value and the penalty. Directions strongly represented in the design matrix are shrunk less; weak directions are shrunk more.

This perspective matters when \(p\), the number of predictors, is large relative to \(n\), the number of observations. The design matrix may have directions with very different levels of information, or even directions in which its columns provide no identifiable signal at all. Singular-vector coordinates reveal which directions ridge retains, how the penalty controls that retention, and why the effective degrees of freedom need not equal the number of predictors.

Model and directional representation

Assume the response vector \(y\) and the columns of the design matrix \(X\) have been centered, so an intercept can be handled separately and is not penalized. Use the ridge objective

This convention has no factor of one-half in either term. Consequently, the penalty \(\lambda\) is compared with squared singular values of \(X\). Changing the scaling of the objective changes the numerical value assigned to the penalty parameter.

Let \(z_j=u_j^\mathsf{T}y\) be the observed response component along \(u_j\). The response may also have a component orthogonal to every column of \(U\), but that component cannot be fitted by \(X\beta\). If \(b_j=v_j^\mathsf{T}\hat{\beta}_{\lambda}\), then the fitted signal’s component along \(u_j\) is \(d_j b_j\).

Deriving the amount of shrinkage

In these coordinates, the part of the squared error that depends on \(b_j\) is \((z_j-d_jb_j)^2\), and the penalty contribution is \(\lambda b_j^2\). The objective separates across singular directions. Differentiating the contribution for direction \(j\) gives

Setting this derivative to zero yields \((d_j^2+\lambda)b_j=d_jz_j\), and hence \(b_j=d_jz_j/(d_j^2+\lambda)\). Multiplying by \(d_j\) converts the coefficient coordinate into the fitted signal coordinate:

The coefficient coordinate has a different formula from the fitted signal coordinate. When \(d_j\) is small, its coefficient multiplier \(d_j/(d_j^2+\lambda)\) can be small, even though that does not directly describe the fraction of the response component retained. For comparing fitted signal across directions, use \(s_j(\lambda)\).

Effective degrees of freedom

For a fixed \(X\) and \(\lambda\), ridge fitted values are a linear transformation of \(y\): \(\hat{y}_{\lambda}=S_{\lambda}y\), where \(S_{\lambda}\) is called the smoother matrix. The effective degrees of freedom for this linear smoother are defined as \(\operatorname{tr}(S_{\lambda})\), the trace of that matrix.

Because \(X=UDV^\mathsf{T}\), the ridge fitted values can be written as \(U\operatorname{diag}(s_1(\lambda),\ldots,s_r(\lambda))U^\mathsf{T}y\). Therefore the smoother matrix is \(S_{\lambda}=U\operatorname{diag}(s_1(\lambda),\ldots,s_r(\lambda))U^\mathsf{T}\). Taking its trace and using \(U^\mathsf{T}U=I\) gives

This is a derivation of the effective degrees of freedom for the stated ridge objective and fixed design. Each direction contributes between zero and one, rather than counting as either a fully included or excluded parameter. If an unpenalized intercept is fitted in addition to centered predictors, its contribution is one; the displayed sum describes the centered predictor part.

What changes as the penalty increases?

For each nonzero singular direction, the shrinkage factor decreases as \(\lambda\) increases. Differentiating with respect to \(\lambda\) gives

The same monotonicity holds for effective degrees of freedom, since it is the sum of these factors. As \(\lambda\) approaches zero from above, each nonzero direction approaches a factor of one and \(df_{\mathrm{eff}}\) approaches \(r\). As \(\lambda\) grows without bound, the factors and the predictor contribution to \(df_{\mathrm{eff}}\) approach zero. Ridge therefore moves continuously between a fit that uses the full column space and a fit with predictor signal shrunk nearly to zero.

A direction’s factor depends on the ratio of \(\lambda\) to \(d_j^2\). If \(d_j^2\) is much larger than \(\lambda\), then \(s_j(\lambda)\) is close to one. If \(d_j^2\) is much smaller than \(\lambda\), the factor is close to zero. The penalty has its strongest relative effect on directions with small singular values, which often correspond to combinations of predictors that are difficult to distinguish from one another.

Assumptions, diagnostics, and comparisons

The directional algebra requires a fixed design matrix, the stated squared-error ridge objective, and a consistent penalty convention. It does not require normally distributed errors: the shrinkage factors follow directly from the optimization problem. A probabilistic interpretation of sampling variability or prediction uncertainty would require additional assumptions about how \(y\) varies around its mean. Predictor scaling also matters: rescaling columns changes the singular values and thus changes which directions are penalized most. The interpretation here assumes that scaling and centering have been chosen deliberately.

If the model includes an intercept, it is usually left unpenalized; centering makes the singular-direction expressions for the predictors easy to state separately. If the intercept is instead penalized, it participates in the penalty and the preceding formulas must be applied to the full design as specified. Likewise, a software package using \(\|y-X\beta\|_2^2/(2n)+\lambda\|\beta\|_2^2/2\) uses a different numerical scaling of \(\lambda\) from the objective used here.

The singular values and the resulting direction-wise factors provide a useful calculation check. For \(\lambda>0\), every factor must lie strictly between zero and one, and a smaller singular value must have a smaller factor when \(\lambda\) is fixed. The effective degrees of freedom must lie between zero and \(r\), and must decrease as \(\lambda\) increases. A factor above one, a negative effective degree of freedom, or a larger factor in a weaker direction signals an algebra or convention mismatch.

Unpenalized least squares retains every nonzero singular direction fully, so its fitted-value degrees of freedom are \(r\). It can be unstable in weak directions, and when \(p>n\) its coefficients need not be unique. Ridge instead shrinks all nonzero directions continuously. A different approach is truncated singular value decomposition, which keeps selected directions at full strength and discards the rest. Ridge uses gradual rather than all-or-nothing retention; neither approach guarantees that the retained directions correspond to individual genes or causal effects.

A common mistake is to say that ridge multiplies every original gene coefficient by the same shrinkage factor. That is generally false: correlated predictors mix across the singular-vector directions, and the factor \(d_j^2/(d_j^2+\lambda)\) applies to the fitted signal along \(u_j\), not to each original coefficient. Another mistake is to interpret effective degrees of freedom as the number of nonzero coefficients. Ridge coefficients are generally not exactly zero; the trace measures the aggregate sensitivity of fitted values to the response.