Ridge Penalties and Continuous Shrinkage

Why Ridge Shrinkage Depends on Direction

A likelihood can contain very different amounts of information about different combinations of regression coefficients. When predictors are strongly correlated, the data may identify their shared contribution reasonably well while providing little information about how that contribution should be divided between them. A ridge penalty stabilizes estimation by shrinking coefficients toward zero. Its key feature is that it shrinks coefficient directions by different amounts: directions weakly supported by the likelihood receive the strongest contraction.

The derivation is especially transparent when the log-likelihood is quadratic in the coefficient vector. This includes Gaussian linear-model likelihoods with fixed variance and also provides the form of a local quadratic approximation to a nonquadratic log-likelihood. The exact quadratic result should not be mistaken for a claim that every GLM log-likelihood is globally quadratic.

Quadratic Likelihood and Ridge Objective

Let \(\beta\in\mathbb{R}^p\) be the vector of coefficients being penalized. Consider a quadratic log-likelihood, up to a term not depending on \(\beta\):

where \(s\in\mathbb{R}^p\), \(H\) is a symmetric positive semidefinite \(p\times p\) matrix, and \(c\) is constant in \(\beta\). The matrix \(H\) describes curvature, or information, in the likelihood. The ridge-penalized objective, for tuning parameter \(\lambda\geq 0\), is

This convention penalizes the squared Euclidean length of \(\beta\); in this tutorial, all coefficients in \(\beta\) are penalized. The factor \(1/2\) makes the derivative of the penalty equal to \(\lambda\beta\). The intercept and scaling conventions used in a particular model must be specified separately.

The formula follows by differentiating, but its existence and uniqueness are important too: they explain why the penalty can yield a well-defined estimate even if the unpenalized likelihood has flat directions.

Deriving the Ridge Estimator

Proof. Ignoring constants, the penalized objective is \(s^\mathsf{T}\beta-\frac{1}{2}\beta^\mathsf{T}(H+\lambda I)\beta\). Its gradient with respect to \(\beta\) is \(s-(H+\lambda I)\beta\). Setting this equal to zero gives \((H+\lambda I)\beta=s\). For every nonzero vector \(v\),

because \(H\) is positive semidefinite and \(\lambda v^\mathsf{T}v>0\). Thus \(H+\lambda I\) is positive definite and invertible, so the stationary point is \((H+\lambda I)^{-1}s\). The Hessian of the objective is \(-(H+\lambda I)\), which is negative definite. The objective is therefore strictly concave, and the stationary point is its unique global maximizer. \(\square\)

When \(H\) is positive definite, the unpenalized maximizer is \(\hat{\beta}=H^{-1}s\), so \(s=H\hat{\beta}\). In this case the ridge estimate can also be written as \((H+\lambda I)^{-1}H\hat{\beta}\). This expression makes clear that ridge transforms, rather than simply rescales, the unpenalized coefficient vector.

Spectral Decomposition and Direction-Specific Shrinkage

Because \(H\) is symmetric, it has an orthonormal eigenbasis. Write its eigenvectors as \(u_1,\ldots,u_p\), with corresponding eigenvalues \(h_1,\ldots,h_p\geq 0\), and express the unpenalized estimate as \(\hat{\beta}=\sum_{j=1}^p a_j u_j\). In this basis, \(H\) multiplies the \(j\)th component by \(h_j\), while \(H+\lambda I\) multiplies it by \(h_j+\lambda\).

Proof. Since \(H\hat{\beta}=s\), the ridge estimate is \((H+\lambda I)^{-1}H\hat{\beta}\). For each eigenvector \(u_j\), \(Hu_j=h_j u_j\) and \((H+\lambda I)u_j=(h_j+\lambda)u_j\). Therefore \((H+\lambda I)^{-1}Hu_j=\frac{h_j}{h_j+\lambda}u_j\). Applying this identity to each term in the eigenvector expansion of \(\hat{\beta}\) gives the stated formula. \(\square\)

For \(\lambda>0\), the factor \(h_j/(h_j+\lambda)\) lies strictly between zero and one. It increases with \(h_j\): a direction with substantial likelihood curvature is retained more, while a direction with small curvature is contracted more. If \(h_j=0\), that direction is not identified by the likelihood; the ridge solution has zero component in that direction. The decomposition describes shrinkage of linear combinations of coefficients, which need not coincide with shrinkage of individual named predictors.

For a weighted least-squares quadratic likelihood, \(H=X^\mathsf{T}WX\) and \(s=X^\mathsf{T}Wy\), where \(X\) is the design matrix, \(W\) is a positive semidefinite weight matrix, and \(y\) is the response vector. The resulting expression is

The eigenvalues of \(X^\mathsf{T}WX\) describe the information available about coefficient directions. Near-collinearity produces a direction with small curvature, which the ridge penalty stabilizes by adding \(\lambda\) to that direction’s curvature.

Worked Examples

Interpretation and a Common Pitfall

Ridge is useful when the likelihood has unstable, weakly identified directions. It adds curvature in every penalized direction, so the quadratic objective has a unique maximum when \(\lambda>0\), even if some predictors are exactly collinear. This is stabilization, not new information from the data: the estimate depends on the chosen penalty, and the amount of shrinkage depends on both \(\lambda\) and the likelihood curvature.

A common mistake is to argue, “Ridge multiplies every coefficient by the same number, so every coefficient is shrunk equally.” That reasoning would be valid only in a special case where all relevant eigenvalues are equal. In general, the equal multipliers apply in the eigenbasis of \(H\), not in the original predictor coordinates. Collinearity is precisely a case where the eigenvalues differ greatly. Another mistake is to call the ridge estimate unbiased merely because it is uniquely defined: the formula contracts the unpenalized estimate, so it is generally biased toward zero under the corresponding sampling model.