Ridge Risk In Orthogonal Designs

Why Orthogonality Makes Ridge Risk Transparent

Least squares is unbiased in a full-rank Gaussian linear model, but its variance can be large when a design direction is weakly measured. Ridge estimation reduces that variance by shrinking estimated coefficients toward zero. The cost is bias: if a true coefficient is far from zero, shrinkage pulls the estimate away from its target. An orthogonal design lets us calculate this trade-off one coordinate at a time, then add the coordinate risks to obtain the total squared-error risk.

Throughout, the design matrix is fixed, the error variance is known, and the ridge penalty is a fixed nonnegative number. These conditions matter: if the penalty is selected from the observed data, the fixed-penalty risk calculations below do not automatically apply.

Definition: Consider the Gaussian linear model $Y=X\beta+\varepsilon$, where $Y\in\mathbb{R}^n$, $X$ is a fixed $n\times p$ matrix of full column rank, $\beta\in\mathbb{R}^p$, and $\varepsilon\sim N(0,\sigma^2 I_n)$ with $\sigma^2>0$. The design is orthogonal here if $X^\mathsf{T}X$ is diagonal. Write $x_j$ for column $j$ of $X$ and $q_j=x_j^\mathsf{T}x_j>0$. For a fixed penalty $\lambda\geq 0$, the ridge estimator is $\hat\beta_\lambda=(X^\mathsf{T}X+\lambda I_p)^{-1}X^\mathsf{T}Y$. We use squared-error loss $L(\hat\beta,\beta)=\|\hat\beta-\beta\|^2=\sum_{j=1}^p(\hat\beta_j-\beta_j)^2$, and its risk is $R(\beta,\hat\beta)=E_\beta[L(\hat\beta,\beta)]$.

Coordinatewise Distribution and Risk

Because $X^\mathsf{T}X$ is diagonal, the least-squares estimate in coordinate $j$ is $x_j^\mathsf{T}Y/q_j$, and ridge multiplies it by a scalar between zero and one. To make the calculation explicit, define $Z_j=x_j^\mathsf{T}Y/q_j$. The model gives

Z_j=\beta_j+\frac{x_j^\mathsf{T}\varepsilon}{q_j}, \qquad E_\beta[Z_j]=\beta_j, \qquad \operatorname{Var}_\beta(Z_j)=\frac{\sigma^2}{q_j}.

The last equality follows from $\operatorname{Var}(x_j^\mathsf{T}\varepsilon)=x_j^\mathsf{T}(\sigma^2 I_n)x_j=\sigma^2q_j$. Since the errors are Gaussian, the $Z_j$ are jointly Gaussian. For $j\ne k$, their covariance is $\sigma^2 x_j^\mathsf{T}x_k/(q_jq_k)=0$, so the $Z_j$ are independent.

The ridge coordinate can now be written as $\hat\beta_{\lambda,j}=a_jZ_j$, where $a_j=q_j/(q_j+\lambda)$. Its bias is its expectation minus $\beta_j$; its variance is the variance of the random estimate.

Result: Under the stated model, for every coordinate $j$, $$ \hat\beta_{\lambda,j}=\frac{q_j}{q_j+\lambda}Z_j,\qquad \operatorname{Bias}_\beta(\hat\beta_{\lambda,j})=-\frac{\lambda}{q_j+\lambda}\beta_j,\qquad \operatorname{Var}_\beta(\hat\beta_{\lambda,j})=\frac{\sigma^2q_j}{(q_j+\lambda)^2}. $$ Its mean squared error is $$ E_\beta[(\hat\beta_{\lambda,j}-\beta_j)^2] =\frac{\lambda^2\beta_j^2+\sigma^2q_j}{(q_j+\lambda)^2}. $$

To verify the mean squared error formula, write the centered error as $\hat\beta_{\lambda,j}-\beta_j=(\hat\beta_{\lambda,j}-E_\beta[\hat\beta_{\lambda,j}])+\operatorname{Bias}_\beta(\hat\beta_{\lambda,j})$. The first term has mean zero, so the cross-product has expectation zero. Thus mean squared error is variance plus squared bias, giving the displayed expression. The same decomposition applies to any estimator with finite second moment.

Worked Example: Computing Ridge Risk With Unequal Column Norms

Suppose $p=2$, $\sigma^2=1$, $q_1=4$, $q_2=1$, $\beta=(2,-1/2)$, and $\lambda=2$. The shrinkage factors are $a_1=4/(4+2)=2/3$ and $a_2=1/(1+2)=1/3$.

For coordinate 1, the bias is $-2(2)/(4+2)=-2/3$, and the variance is $1\cdot4/(4+2)^2=1/9$. Its mean squared error is $4/9+1/9=5/9$.

For coordinate 2, the bias is $-2(-1/2)/(1+2)=1/3$, and the variance is $1\cdot1/(1+2)^2=1/9$. Its mean squared error is $1/9+1/9=2/9$. Therefore the total ridge risk is $5/9+2/9=7/9$.

The least-squares risk is $1/4+1=5/4$, since its coordinate variances are $\sigma^2/q_1=1/4$ and $\sigma^2/q_2=1$, with zero bias. Here ridge reduces total risk from $5/4$ to $7/9$, even though both ridge coordinates are biased.

Comparing Ridge With Least Squares

For least squares, $\hat\beta_{\mathrm{LS},j}=Z_j$, so it is unbiased and has variance $\sigma^2/q_j$. Ridge has lower variance in every coordinate when $\lambda>0$, but its squared bias depends on the unknown signal $\beta_j$. The exact difference in coordinate risk isolates this competition.

Theorem: For $\lambda>0$, the difference between ridge and least-squares risk in coordinate $j$ is $$ R_j(\beta,\hat\beta_\lambda)-R_j(\beta,\hat\beta_{\mathrm{LS}}) =\frac{\lambda^2\beta_j^2-2\sigma^2\lambda-\sigma^2\lambda^2/q_j}{(q_j+\lambda)^2}. $$ Consequently, ridge has lower risk in that coordinate exactly when $$ \beta_j^2<\frac{2\sigma^2}{\lambda}+\frac{\sigma^2}{q_j}. $$ For $\lambda=0$, ridge equals least squares and the risks are equal.

Proof. Least squares has coordinate risk $\sigma^2/q_j$. Subtract that from the ridge mean squared error:

\begin{aligned} R_j(\beta,\hat\beta_\lambda)-R_j(\beta,\hat\beta_{\mathrm{LS}}) &=\frac{\lambda^2\beta_j^2+\sigma^2q_j}{(q_j+\lambda)^2}-\frac{\sigma^2}{q_j}\\ &=\frac{\lambda^2\beta_j^2+\sigma^2q_j-\sigma^2(q_j+\lambda)^2/q_j}{(q_j+\lambda)^2}\\ &=\frac{\lambda^2\beta_j^2-2\sigma^2\lambda-\sigma^2\lambda^2/q_j}{(q_j+\lambda)^2}. \end{aligned}

The denominator is positive. Thus the difference is negative exactly when $\lambda^2\beta_j^2<2\sigma^2\lambda+\sigma^2\lambda^2/q_j$. Dividing by $\lambda^2>0$ gives the stated threshold. At $\lambda=0$, the estimator is $Z_j$, which is the least-squares coordinate. $\square$

The threshold says that moderate or small coefficients can benefit from shrinkage, while a sufficiently large coefficient can be harmed by its bias. A larger noise variance makes shrinkage more attractive under this coordinatewise comparison. The threshold also depends on $q_j$: a weakly measured direction has smaller $q_j$ and larger least-squares variance. This is one reason a common penalty need not produce the same risk trade-off in every direction.

Worked Example: A Coordinate Where Shrinkage Helps

Take one coordinate with $q=1$, $\sigma^2=1$, $\lambda=1$, and $\beta=0$. Least squares has risk $1/q=1$. Ridge has zero bias and variance $q/(q+\lambda)^2=1/4$, so its risk is $1/4$.

The threshold for improvement is $\beta^2<2\sigma^2/\lambda+\sigma^2/q=2+1=3$. Since $\beta^2=0<3$, the exact comparison predicts improvement. Direct subtraction gives $1/4-1=-3/4$, confirming that ridge's risk is lower by $3/4$.

Total Risk and a Coordinate Where Ridge Hurts

Under total squared-error loss, the loss is a sum of coordinate losses. Linearity of expectation therefore gives total risk as the sum of the coordinate risks; independence is not needed for this addition. Applying the coordinate comparison yields

R(\beta,\hat\beta_\lambda) =\sum_{j=1}^p\frac{\lambda^2\beta_j^2+\sigma^2q_j}{(q_j+\lambda)^2}, \qquad R(\beta,\hat\beta_{\mathrm{LS}})=\sum_{j=1}^p\frac{\sigma^2}{q_j}.

The total-risk difference is the sum of the coordinate differences. Some coordinates can improve while others worsen, so a favorable comparison in one coordinate does not by itself establish that total ridge risk is lower. To establish an improvement for a particular $\beta$, the sum of the signed differences must be negative.

Worked Example: A Large Coefficient Makes Ridge Worse

Consider a single coordinate with $q=4$, $\sigma^2=1$, $\lambda=1$, and $\beta=3$. Least squares is unbiased and has risk $1/4$. Ridge has shrinkage factor $4/5$, so its bias is $-(1/5)3=-3/5$ and its variance is $4/25$. Its risk is therefore $9/25+4/25=13/25=0.52$, which exceeds $1/4=0.25$.

The threshold for improvement is $2(1)/1+1/4=9/4$, whereas $\beta^2=9>9/4$. The comparison correctly predicts that ridge is worse. Directly, the risk difference is $13/25-1/4=(52-25)/100=27/100>0$.

Interpreting the Trade-Off

The formulas also clarify the limiting cases. At $\lambda=0$, there is no shrinkage and ridge is exactly least squares. As $\lambda$ increases, each factor $q_j/(q_j+\lambda)$ decreases, and the variance in that coordinate decreases. For fixed $\beta_j$, as $\lambda$ tends to infinity the estimate tends to zero, its variance tends to zero, and its mean squared error tends to $\beta_j^2$. Strong shrinkage can therefore be poor when the signal is large, even though it removes sampling variance.

In an orthogonal design, the individual $q_j$ values make the effect of the design visible rather than hidden in a matrix expression. A small $q_j$ means the least-squares estimate in that direction has high variance. Ridge's coordinatewise factor adapts to $q_j$, but the same $\lambda$ can still impose materially different shrinkage across directions.

Common pitfall: It is incorrect to conclude that ridge always has lower risk because its variance is lower than least squares. Risk includes squared bias as well as variance. For example, with $q=4$, $\sigma^2=1$, and $\lambda=1$, ridge variance is $4/25<1/4$, but at $\beta=3$ its squared bias is $9/25$, making total risk $13/25>1/4$. Compare total mean squared errors, not variances alone.

These calculations are exact for a fixed design, a fixed penalty, and known $\sigma^2$. They do not claim that one fixed $\lambda$ improves risk for every possible coefficient vector: the coordinate comparison itself depends on $\beta_j$, and the total comparison depends on all coordinates. The practical value of the orthogonal calculation is that it shows precisely where the bias cost enters and how it must be weighed against the variance reduction.

Check Your Understanding

Use the definitions and risk formulas above to answer these questions.

In the orthogonal Gaussian model, what is the distribution of the least-squares coordinate $Z_j$, and what feature of the design determines its variance?
For fixed $\lambda>0$, explain why ridge has lower variance but nonzero bias when $\beta_j\ne0$.
For a coordinate with $q_j=2$, $\sigma^2=3$, and $\lambda=1$, calculate the signal-size threshold below which ridge has lower risk than least squares.
Can one coordinate having lower ridge risk than least-squares risk guarantee lower total risk? Explain.
What happens to the ridge estimate and its coordinate risk as $\lambda$ tends to infinity?