Tutorials › Exam SRM - Statistics for Risk Modeling › Ridge Shrinkage Along Principal Directions

Home  /  Tutorials  /  Exam SRM / Ridge and Lasso Regression

Exam SRM · Ridge and Lasso Regression · Tutorial 208 of 926

Ridge Shrinkage Along Principal Directions

Learn to express ridge coefficients in principal directions, calculate their direction-specific shrinkage, and avoid confusing eigenvalues with shrinkage factors.

Advanced 11 min read

What You'll Learn

  • Express the ridge solution using the eigendecomposition of the predictor cross-product matrix
  • Calculate ridge coefficients separately along each principal direction
  • Compare shrinkage in directions with large and small eigenvalues
  • Transform principal-direction coefficients back to the original predictor coordinates
  • Identify the effects of the penalty convention, predictor scaling, and rank deficiency

Why Ridge Shrinkage Depends on Direction

Ridge regression does not generally shrink every combination of predictors by the same proportion. Its effect depends on the directions in predictor space: combinations that the data measure strongly are usually shrunk less than combinations measured weakly. The eigenvectors of the predictor cross-product matrix identify these directions, and their eigenvalues quantify the amount of information in each one.

This decomposition makes the ridge solution easier to interpret algebraically. Instead of treating a system of simultaneous equations as a single calculation, we can solve it one principal direction at a time. This tutorial uses the penalty convention \(\operatorname{RSS}+\lambda\boldsymbol{\beta}^{\mathsf T}\boldsymbol{\beta}\), where \(\lambda\geq 0\). A convention that divides RSS by the sample size would change the numerical meaning of \(\lambda\).

Definition: For centered predictors collected in the \(n\times p\) matrix \(\mathbf X\), let \(\mathbf X^{\mathsf T}\mathbf X=\mathbf Q\mathbf D\mathbf Q^{\mathsf T}\) be an eigendecomposition, where \(\mathbf Q\) is orthogonal and \(\mathbf D=\operatorname{diag}(d_1,\ldots,d_p)\) contains the nonnegative eigenvalues. The columns \(\mathbf q_j\) of \(\mathbf Q\) are the principal directions of the predictor cross-product matrix. The ridge coefficient component in direction \(\mathbf q_j\) is the coordinate \(\mathbf q_j^{\mathsf T}\widehat{\boldsymbol{\beta}}_{\lambda}\).

Assume the linear model has an intercept and centered predictors, so the intercept can be estimated separately and is not penalized. For the slope coefficients, ridge minimizes

$$ \|\mathbf y-\mathbf X\boldsymbol{\beta}\|^2+ \lambda\boldsymbol{\beta}^{\mathsf T}\boldsymbol{\beta}. $$

The eigendecomposition applies to the fixed predictor matrix; a probability distribution for the errors is not needed for the algebra below. For statistical interpretations that use a linear model, the usual mean-zero error assumption is relevant, but the direction-by-direction solution follows directly from the objective function. When \(\lambda>0\), the ridge solution is unique, including when \(\mathbf X^{\mathsf T}\mathbf X\) is singular.

Deriving Shrinkage Along the Principal Directions

The ridge normal equations are \((\mathbf X^{\mathsf T}\mathbf X+\lambda\mathbf I)\widehat{\boldsymbol{\beta}}_{\lambda}=\mathbf X^{\mathsf T}\mathbf y\). Write the coefficient vector in principal coordinates as \(\boldsymbol{\theta}=\mathbf Q^{\mathsf T}\boldsymbol{\beta}\), and write \(\mathbf c=\mathbf Q^{\mathsf T}\mathbf X^{\mathsf T}\mathbf y\). Multiplying the normal equations by \(\mathbf Q^{\mathsf T}\) gives a separate equation for each direction.

$$ (\mathbf D+\lambda\mathbf I)\widehat{\boldsymbol{\theta}}_{\lambda} =\mathbf c, \qquad \widehat{\theta}_{\lambda,j} =\frac{c_j}{d_j+\lambda}. $$

If \(d_j>0\), the ordinary least squares coordinate in that direction is \(\widehat{\theta}_{0,j}=c_j/d_j\). Therefore, the ridge coordinate is the ordinary least squares coordinate multiplied by a direction-specific factor:

$$ \widehat{\theta}_{\lambda,j} = \frac{d_j}{d_j+\lambda}\widehat{\theta}_{0,j}. $$

This equality is the key result. The factor \(d_j/(d_j+\lambda)\) is between zero and one for \(d_j>0\) and \(\lambda>0\). For a fixed positive \(\lambda\), it increases with \(d_j\): a direction with a larger eigenvalue retains a greater fraction of its least-squares coordinate. A direction with a smaller positive eigenvalue receives stronger shrinkage. As \(\lambda\) increases, every such factor decreases.

Derivation: Since \(\mathbf Q^{\mathsf T}\mathbf Q=\mathbf I\), the eigendecomposition gives \(\mathbf Q^{\mathsf T}(\mathbf X^{\mathsf T}\mathbf X+\lambda\mathbf I)\mathbf Q=\mathbf D+\lambda\mathbf I\). Thus the \(j\)th principal-coordinate equation is \((d_j+\lambda)\widehat{\theta}_{\lambda,j}=c_j\). When \(d_j>0\), ordinary least squares gives \(\widehat{\theta}_{0,j}=c_j/d_j\). Substitution yields \(\widehat{\theta}_{\lambda,j}=[d_j/(d_j+\lambda)]\widehat{\theta}_{0,j}\), as claimed.

If \(d_j=0\), the data contain no variation in direction \(\mathbf q_j\). In exact arithmetic, \(\mathbf X\mathbf q_j=\mathbf 0\), so \(c_j=\mathbf q_j^{\mathsf T}\mathbf X^{\mathsf T}\mathbf y=0\). For \(\lambda>0\), the ridge coordinate is then zero. There is no ordinary least squares coordinate to shrink in that direction, so the multiplicative comparison with OLS does not apply.

Worked Example: Transforming Shrunk Coordinates Back to Predictors

Method and setup. Suppose the centered two-predictor cross-product matrix is \(\mathbf X^{\mathsf T}\mathbf X=\begin{pmatrix}5&4\\4&5\end{pmatrix}\), and \(\mathbf X^{\mathsf T}\mathbf y=(19/\sqrt{2},17/\sqrt{2})^{\mathsf T}\). Use \(\lambda=3\). The model has a separate unpenalized intercept. The orthonormal eigenvectors are \(\mathbf q_1=(1,1)^{\mathsf T}/\sqrt{2}\) and \(\mathbf q_2=(1,-1)^{\mathsf T}/\sqrt{2}\), with eigenvalues 9 and 1, respectively.

Step 1: Find the least squares coordinates. Project \(\mathbf X^{\mathsf T}\mathbf y\) onto each eigenvector:

$$ c_1=\mathbf q_1^{\mathsf T}\mathbf X^{\mathsf T}\mathbf y =\frac{19+17}{2}=18, \qquad c_2=\mathbf q_2^{\mathsf T}\mathbf X^{\mathsf T}\mathbf y =\frac{19-17}{2}=1. $$

The OLS coordinates are \(18/9=2\) and \(1/1=1\).

Step 2: Apply the direction-specific factors. In the first direction the factor is \(9/(9+3)=0.75\); in the second it is \(1/(1+3)=0.25\). The ridge coordinates are \(2(0.75)=1.5\) and \(1(0.25)=0.25\).

Step 3: Return to the original predictor coordinates. Since \(\widehat{\boldsymbol{\beta}}_{\lambda}=1.5\mathbf q_1+0.25\mathbf q_2\),

$$ \widehat{\boldsymbol{\beta}}_{\lambda} = \begin{pmatrix} (1.5+0.25)/\sqrt{2}\\ (1.5-0.25)/\sqrt{2} \end{pmatrix} = \begin{pmatrix} 1.75/\sqrt{2}\\ 1.25/\sqrt{2} \end{pmatrix} \approx \begin{pmatrix} 1.2374\\ 0.8839 \end{pmatrix}. $$

Check and trap avoided. The eigenvalues are 9 and 1, so the first direction must retain more of its OLS coordinate; the factors 0.75 and 0.25 have that ordering. The original-coordinate vector also gives principal coordinates \((1.2374+0.8839)/\sqrt{2}\approx1.5\) and \((1.2374-0.8839)/\sqrt{2}\approx0.25\). Do not confuse the direction coordinates with the original predictor coefficients.

Comparing Strong and Weak Directions

An eigenvalue measures the squared spread of the centered predictor data along its corresponding principal direction. The eigenvalue is not itself the shrinkage factor. The factor also depends on \(\lambda\), through the ratio \(d_j/(d_j+\lambda)\). Consequently, a numerical eigenvalue has no standalone interpretation as “large” or “small” without considering the penalty and the scale of the predictors.

Worked Example: Three Directions With Different Eigenvalues

Method and setup. Suppose a centered three-predictor design has eigenvalues \(16,4,1\), and its OLS coefficients in the corresponding orthonormal directions are \(1.5,-2,3\). Let \(\lambda=3\), with the intercept unpenalized. All three eigenvalues are positive, so the OLS coordinates are defined.

Step 1: Compute each shrinkage factor. They are \(16/(16+3)=16/19\), \(4/(4+3)=4/7\), and \(1/(1+3)=1/4\).

Step 2: Multiply each OLS coordinate by its factor.

$$ \widehat{\theta}_{\lambda,1}=1.5\left(\frac{16}{19}\right)=\frac{24}{19}\approx1.2632, \qquad \widehat{\theta}_{\lambda,2}=-2\left(\frac{4}{7}\right)=-\frac{8}{7}\approx-1.1429, \qquad \widehat{\theta}_{\lambda,3}=3\left(\frac{1}{4}\right)=0.75. $$

Check and trap avoided. The retained fractions are approximately \(0.8421, 0.5714,\) and \(0.25\), respectively. Thus the coordinate associated with eigenvalue 1 is shrunk most strongly. The signs stay the same because each factor is positive. Do not claim that every original predictor coefficient is reduced by the same factor: these calculations are for orthogonal principal coordinates.

The next calculation isolates the role of \(\lambda\). It also shows why one should compare each eigenvalue with the penalty rather than memorize a fixed shrinkage percentage.

Worked Example: How the Penalty Changes Relative Shrinkage

Method and setup. Consider two principal directions with eigenvalues 2 and 18. Suppose the OLS coordinates are 4 and \(-1\). The predictor columns are centered, and only the slope coefficients are penalized. Compare \(\lambda=2\) with \(\lambda=8\).

Step 1: Use \(\lambda=2\). The factors are \(2/(2+2)=1/2\) and \(18/(18+2)=9/10\). The ridge coordinates are \(4(1/2)=2\) and \(-1(9/10)=-0.9\).

Step 2: Use \(\lambda=8\). The factors are \(2/(2+8)=1/5\) and \(18/(18+8)=9/13\). The ridge coordinates are \(4(1/5)=0.8\) and \(-1(9/13)=-9/13\approx-0.6923\).

Check and trap avoided. Increasing \(\lambda\) reduces both retained fractions: \(0.5\) becomes \(0.2\), and \(0.9\) becomes about \(0.6923\). At either penalty, the direction with eigenvalue 2 is shrunk more. Do not compare the absolute sizes of the final coordinates to decide which direction was shrunk more; compare the retained fractions, or compare each ridge coordinate with its own OLS coordinate.

Scaling, Rank, and Common Exam Traps

The eigenvalues depend on the scale of the predictor columns. If one predictor is measured in dollars and another in thousands of dollars, their cross-product matrix and its eigenvalues change. The ridge penalty is defined relative to the coefficient scale, so changing units can change the fitted ridge coefficients for a fixed \(\lambda\). Standardizing predictors before ridge regression is one way to make the penalty operate on a common scale, but it is a modeling choice, not an automatic property of the eigendecomposition. The direction-specific formula remains valid for whichever fixed design matrix is actually used.

A frequent calculation error is to use \(\lambda/(d_j+\lambda)\) as the fraction of the OLS coordinate that remains. That quantity is the amount removed, not the amount retained:

$$ 1-\frac{d_j}{d_j+\lambda} =\frac{\lambda}{d_j+\lambda}. $$

For example, with \(d_j=9\) and \(\lambda=3\), the retained fraction is \(9/12=0.75\), while the removed fraction is \(3/12=0.25\). Multiplying an OLS coordinate by 0.25 would overstate the shrinkage.

Key takeaway: In principal coordinates, ridge multiplies an OLS coefficient by \(d_j/(d_j+\lambda)\) whenever \(d_j>0\). A larger eigenvalue means a larger retained fraction at the same positive penalty. Use the eigenvectors to convert between principal coordinates and the original predictor coefficients, and keep the design scaling and penalty convention fixed throughout the calculation.

Check Your Understanding

For each problem, assume centered predictors and an unpenalized intercept. Use the penalty convention \(\operatorname{RSS}+\lambda\boldsymbol{\beta}^{\mathsf T}\boldsymbol{\beta}\).

  1. A principal direction has eigenvalue 12 and OLS coordinate 5. If \(\lambda=3\), find its ridge coordinate. (Answer: \(4\).)
  2. Two directions have eigenvalues 25 and 1, and OLS coordinates 2 and \(-4\). For \(\lambda=4\), give both ridge coordinates. (Answer: \(50/29\) and \(-0.8\).)
  3. At \(\lambda=6\), compare the retained fractions for directions with eigenvalues 6 and 24. Which direction is shrunk more? (Answer: The fractions are \(0.5\) and \(0.8\); the direction with eigenvalue 6 is shrunk more.)
  4. In principal coordinates, the cross-product eigenvalues are 10 and 2, and the coordinates of \(\mathbf Q^{\mathsf T}\mathbf X^{\mathsf T}\mathbf y\) are 30 and \(-8\). For \(\lambda=2\), calculate the ridge coordinates and check them against the OLS coordinates. (Answer: Ridge coordinates \((2.5,-2)\); OLS coordinates \((3,-4)\).)
  5. A rank-deficient design has eigenvalues 7 and 0, and \(\mathbf Q^{\mathsf T}\mathbf X^{\mathsf T}\mathbf y=(14,0)^{\mathsf T}\). For \(\lambda=1\), find the ridge coordinates and state why an OLS shrinkage ratio is unavailable in the second direction. (Answer: \((1.75,0)^{\mathsf T}\); the second direction has eigenvalue zero, so its OLS coordinate is not defined.)

Previous

Next

Back to Tutorials
Clinical Biostats