Ridge Shrinkage with Orthogonal Predictors

Why Orthogonality Makes Ridge Shrinkage Transparent

Ridge regression adds a penalty on coefficient size to the residual sum of squares. In general, the predictors interact through their correlations, so the effect of the penalty on one coefficient depends on the design as a whole. When the predictors are orthogonal and have the same scale, that interaction disappears: every least-squares coefficient is multiplied by the same shrinkage factor.

This special case is useful for understanding the mechanics of ridge regression. It makes the penalty’s effect easy to calculate and gives a clear check on computations in more complicated settings. The result depends on the exact scaling of the objective and predictors, so those conventions must be stated before using the formula.

Definition: Let \(X\) be a design matrix with \(p\) predictor columns \(x_1,\ldots,x_p\). For this tutorial, the predictors and response are centered, each predictor has sum of squares \(n\), and distinct predictors are orthogonal: \(x_j^\mathsf{T}x_k=0\) when \(j\ne k\), and \(x_j^\mathsf{T}x_j=n\). Thus \(X^\mathsf{T}X=nI\). The scale convention corresponds to each centered predictor having mean squared value 1. We consider the slope coefficients and the objective \(\|y-X\beta\|^2+\lambda\|\beta\|^2\), with \(\lambda\ge 0\).

The equal-norm condition matters as much as orthogonality. If the columns are orthogonal but have different sums of squares, each coefficient still separates from the others, but its shrinkage factor depends on its own column’s sum of squares. Here, the common value \(n\) ensures one shared factor.

Deriving the Ridge Coefficient

With the stated objective, the ridge estimate minimizes the residual sum of squares plus \(\lambda\) times the sum of squared slopes. The ordinary least-squares estimate is the minimizer when \(\lambda=0\). Under orthogonality, expand the objective one coefficient at a time.

\begin{aligned} Q(\beta) &=(y-X\beta)^\mathsf{T}(y-X\beta)+\lambda\beta^\mathsf{T}\beta\\ &=y^\mathsf{T}y-2\beta^\mathsf{T}X^\mathsf{T}y +\beta^\mathsf{T}(X^\mathsf{T}X)\beta+\lambda\beta^\mathsf{T}\beta\\ &=y^\mathsf{T}y-2\sum_{j=1}^p\beta_jx_j^\mathsf{T}y +(n+\lambda)\sum_{j=1}^p\beta_j^2. \end{aligned}

The last line follows from \(X^\mathsf{T}X=nI\): all cross-products between different coefficient terms are zero. Differentiating with respect to \(\beta_j\) and setting the derivative equal to zero gives \(-2x_j^\mathsf{T}y+2(n+\lambda)\beta_j=0\). Therefore the ridge coefficient is \(x_j^\mathsf{T}y/(n+\lambda)\). The least-squares coefficient is \(x_j^\mathsf{T}y/n\), so their ratio gives the shrinkage formula.

\widehat{\beta}^{\,\text{ridge}}_j =\frac{x_j^\mathsf{T}y}{n+\lambda} =\frac{n}{n+\lambda}\widehat{\beta}^{\,\text{OLS}}_j

Theorem: For centered predictors with \(X^\mathsf{T}X=nI\), under the objective \(\|y-X\beta\|^2+\lambda\|\beta\|^2\), every slope coefficient is multiplied by the retained fraction \(n/(n+\lambda)\). The fraction removed from each coefficient’s magnitude is \(\lambda/(n+\lambda)\). The formula applies to this stated scaling and equal-norm orthogonal design; it is not a universal formula for correlated predictors or for objectives that scale the penalty differently.

The two fractions add to 1: \(\frac{n}{n+\lambda}+\frac{\lambda}{n+\lambda}=1\). The retained fraction determines the ridge coefficient. The removed fraction describes how much of the original coefficient magnitude is eliminated. These are complementary quantities, not interchangeable labels for the same factor.

Find the design scale.
Use \(n\) when \(X^\mathsf{T}X=nI\). Confirm that the stated ridge objective adds \(\lambda\|\beta\|^2\) to the unscaled residual sum of squares.

Calculate the retained fraction.
Compute \(r=n/(n+\lambda)\). For \(\lambda=0\), \(r=1\); for positive \(\lambda\), \(0<r<1\).

Shrink every slope.
Multiply each least-squares coefficient by \(r\), preserving its sign. If needed, calculate the fraction removed as \(1-r=\lambda/(n+\lambda)\).

Worked Calculations

Worked Example: Three Penalty Values on the Same Design

A hypothetical pricing study has six centered observations and three standardized, mutually orthogonal predictors. Thus \(X^\mathsf{T}X=6I\). The cross-products with the centered response are \(X^\mathsf{T}y=(18,-9,3)^\mathsf{T}\). Find the least-squares coefficients and the ridge coefficients at \(\lambda=6\) and \(\lambda=18\). Also distinguish the signed coefficient changes from the absolute reductions.

Method. First obtain each least-squares coefficient from \(\widehat{\beta}^{\,\text{OLS}}_j=x_j^\mathsf{T}y/6\): \[ \widehat{\beta}^{\,\text{OLS}}=(18/6,-9/6,3/6)=(3,-1.5,0.5). \] The design scale in the shrinkage formula is \(n=6\).

At \(\lambda=6\), the retained fraction is \(6/(6+6)=1/2\), and the fraction removed is also \(6/12=1/2\). Therefore \[ \widehat{\beta}^{\,\text{ridge}}=(1.5,-0.75,0.25). \] At \(\lambda=18\), the retained fraction is \(6/(6+18)=6/24=1/4\), while the fraction removed is \(18/(6+18)=18/24=3/4=0.75\). The ridge coefficients are \[ \widehat{\beta}^{\,\text{ridge}}=(0.75,-0.375,0.125). \]

For \(\lambda=6\), subtract the ridge coefficients from the least-squares coefficients in their stated order: \[ (3,-1.5,0.5)-(1.5,-0.75,0.25)=(1.5,-0.75,0.25). \] The signed change from least squares to ridge is the negative of this vector, \((-1.5,0.75,-0.25)\). Its componentwise absolute values are \((1.5,0.75,0.25)\). In particular, \(-1.5-(-0.75)=-0.75\), not \(+0.75\); its absolute reduction is \(0.75\).

Check. Each ridge coefficient has the same sign as its least-squares value and one-half its magnitude at \(\lambda=6\). At \(\lambda=18\), each retains one-quarter of its magnitude, so three-quarters is removed. A common mistake is to use \(18/24=0.75\) as the coefficient multiplier; it is the removed fraction, not the retained fraction.

Worked Example: Translating Coefficient Shrinkage Into a Prediction

A hypothetical retention analysis uses eight centered observations and three orthogonal predictors standardized so \(X^\mathsf{T}X=8I\). The least-squares slope vector is \((2,-1,0.5)\). Calculate the ridge slope vector at \(\lambda=4\) and \(\lambda=12\), then calculate the slope-only fitted contribution for a new case with standardized predictor values \((1,2,-1)\).

Method. At \(\lambda=4\), the retained fraction is \(8/(8+4)=2/3\). Multiplying every slope by \(2/3\) gives \((4/3,-2/3,1/3)\). At \(\lambda=12\), the retained fraction is \(8/(8+12)=8/20=2/5\), giving \((4/5,-2/5,1/5)\).

For the new case, the slope-only contribution is the inner product of its predictor vector and the slope vector. Under least squares it is \(1(2)+2(-1)+(-1)(0.5)=2-2-0.5=-0.5\). At \(\lambda=4\), it is \(1(4/3)+2(-2/3)+(-1)(1/3)=4/3-4/3-1/3=-1/3\). At \(\lambda=12\), it is \(1(4/5)+2(-2/5)+(-1)(1/5)=4/5-4/5-1/5=-1/5=-0.2\).

Check. Because all slopes share a common factor, the slope-only contribution is also multiplied by that factor: \((-0.5)(2/3)=-1/3\) and \((-0.5)(2/5)=-0.2\). This checks both calculations. The result concerns the slope contribution; it does not include any separately handled intercept.

Common mistake avoided. Shrinking the coefficients does not mean changing their signs or applying a different factor to each one under these assumptions. The shared factor also applies to any linear combination of the slopes, including this case’s slope-only contribution.

Worked Example: Find the Penalty for a Target Shrinkage Level

A hypothetical claims-severity model has ten centered observations and two orthogonal standardized predictors, so \(X^\mathsf{T}X=10I\). Its least-squares coefficients are \((1.2,-0.8)\). Find the penalty value that retains 40% of each coefficient, calculate the ridge coefficients, and state the fraction removed.

Method. The retained fraction must satisfy \(10/(10+\lambda)=0.40\). Solving, \(10=0.40(10+\lambda)=4+0.40\lambda\), so \(6=0.40\lambda\) and \(\lambda=15\).

The ridge coefficients are \(0.40(1.2)=0.48\) and \(0.40(-0.8)=-0.32\), giving \((0.48,-0.32)\). The fraction removed is \(15/(10+15)=15/25=0.60\), or 60%. As a direct check, \(0.40+0.60=1\), and both ridge coefficients have 40% of the corresponding least-squares magnitude.

Common mistake avoided. The requested 40% is the retained fraction. Substituting it as the removed fraction would instead imply that 60% is retained and would produce different coefficients.

What the Formula Shows—and Where It Stops

At \(\lambda=0\), the penalty is absent and the ridge estimate equals least squares. As \(\lambda\) increases from zero, \(n/(n+\lambda)\) decreases from 1 toward 0, so every coefficient moves toward zero. For any finite nonnegative \(\lambda\), a nonzero least-squares coefficient is multiplied by a positive factor: its sign does not change, and it is not set exactly to zero. If a least-squares coefficient is already zero, its ridge value remains zero.

Under this equal-norm orthogonal design, the squared Euclidean length of the coefficient vector also has a simple relationship. If \(r=n/(n+\lambda)\), then \(\widehat{\beta}^{\,\text{ridge}}=r\widehat{\beta}^{\,\text{OLS}}\), so \(\|\widehat{\beta}^{\,\text{ridge}}\|^2=r^2\|\widehat{\beta}^{\,\text{OLS}}\|^2\). This follows by squaring each component, summing, and factoring out the common \(r^2\). The coefficient norm is therefore multiplied by \(r\), while its square is multiplied by \(r^2\).

The same simple component-by-component formula does not apply when predictors are correlated. Then \(X^\mathsf{T}X\) has nonzero off-diagonal entries, so the coefficient equations are coupled. Nor should the formula be transferred unchanged to a differently scaled objective, such as one that divides the residual sum of squares by \(n\), without matching the penalty convention. The penalty’s numerical value has meaning only in relation to the objective’s scaling.

Key takeaway: With centered, standardized, orthogonal columns satisfying \(X^\mathsf{T}X=nI\), and objective \(\|y-X\beta\|^2+\lambda\|\beta\|^2\), the ridge slope vector is \(n/(n+\lambda)\) times the least-squares vector. Label this as the retained fraction; \(\lambda/(n+\lambda)\) is the fraction removed.

Check Your Understanding

For each problem, use the stated objective and design scale. Keep the retained fraction separate from the fraction removed.

A design has \(X^\mathsf{T}X=5I\), least-squares coefficients \((2,-0.6)\), and \(\lambda=5\). Find the ridge coefficients and the fraction removed. (Answer: \((1,-0.3)\); fraction removed \(=0.5\).)
For \(X^\mathsf{T}X=10I\), the least-squares vector is \((1.2,-0.8)\), and \(\lambda=15\). Calculate the retained and removed fractions and the ridge vector. (Answer: retained \(=0.4\), removed \(=0.6\), ridge vector \((0.48,-0.32)\).)
A centered orthogonal design has \(X^\mathsf{T}X=4I\) and \(X^\mathsf{T}y=(12,-4,0)\). At \(\lambda=12\), find the least-squares and ridge vectors, then state the fraction removed. (Answer: least squares \((3,-1,0)\); ridge \((0.75,-0.25,0)\); fraction removed \(=0.75\).)
A coefficient’s least-squares value is \(-2\), and the equal column sum of squares is \(n=6\). If \(\lambda=18\), calculate the ridge coefficient and the signed difference \(\widehat{\beta}^{\,\text{OLS}}-\widehat{\beta}^{\,\text{ridge}}\). (Answer: ridge coefficient \(-0.5\); signed difference \(-1.5\).)
With \(X^\mathsf{T}X=8I\), a ridge coefficient is 20% of its nonzero least-squares value. Find \(\lambda\). If that least-squares coefficient is \(3\), also find its ridge value and the fraction removed. (Answer: \(\lambda=32\); ridge value \(0.6\); fraction removed \(=0.8\).)

What You'll Learn