Ridge Bias and Variance

Why Ridge Can Improve Estimation Despite Being Biased

Ridge regression deliberately pulls coefficient estimates toward zero. That pull introduces bias when the true coefficient is nonzero, so it may seem that ridge must be less accurate than ordinary least squares (OLS). But OLS estimates can vary substantially from sample to sample, especially in directions where the predictors provide limited information. Ridge can reduce that variance enough to offset its added bias. The balance is measured by mean squared error (MSE).

This tutorial studies that balance in fixed-design linear regression, first for one orthogonal predictor direction and then across several such directions. The calculations concern the error in estimating coefficients, not the error in predicting future responses. All numerical scenarios are hypothetical.

Definition: For an estimator \(\widehat{\theta}\) of a fixed coefficient \(\theta\), its bias is \(E[\widehat{\theta}]-\theta\), its variance is \(\operatorname{Var}(\widehat{\theta})\), and its mean squared error is \(E[(\widehat{\theta}-\theta)^2]\). The identity \( \operatorname{MSE}(\widehat{\theta})=\operatorname{Var}(\widehat{\theta})+\operatorname{Bias}(\widehat{\theta})^2 \) separates random variation from squared systematic error.

Model and Ridge Estimator in One Eigen-Direction

Assume the fixed-design linear model \(y=X\beta+\varepsilon\), where \(X\) has full column rank for the directions being analyzed, \(E[\varepsilon\mid X]=0\), and \(\operatorname{Var}(\varepsilon\mid X)=\sigma^2 I\). For the numerical examples, errors will be specified as multivariate normal with this covariance; the bias and variance formulas themselves require only the stated first two moments. The intercept, if present, is not penalized; equivalently, the response and predictors can be centered before analyzing the slopes.

Ridge estimates the slope vector by minimizing the residual sum of squares plus a squared-coefficient penalty:

\widehat{\beta}_{\lambda} =\underset{b}{\operatorname{argmin}}\left\{(y-Xb)^\mathsf{T}(y-Xb)+\lambda b^\mathsf{T}b\right\},\qquad \lambda\geq 0.

Let \(q>0\) be an eigenvalue of \(X^\mathsf{T}X\), and let \(\theta\) be the true coefficient in the associated unit eigenvector direction. The OLS estimate in that direction is denoted \(\widehat{\theta}_{\text{OLS}}\). In this coordinate, ridge multiplies the OLS estimate by the shrinkage factor \(q/(q+\lambda)\):

\widehat{\theta}_{\lambda} =\frac{q}{q+\lambda}\widehat{\theta}_{\text{OLS}}.

The eigenvalue \(q\) measures the design information in that direction. A larger \(q\) means more information and, for OLS, less sampling variance. The penalty \(\lambda\) is nonnegative; when \(\lambda=0\), ridge reduces to OLS. These formulas apply to estimable directions with \(q>0\). They do not give an OLS variance for a direction with \(q=0\), since OLS cannot estimate a coefficient in that direction.

Bias, Variance, and Mean Squared Error

Under the model assumptions, the OLS estimate in this direction has expectation \(\theta\) and variance \(\sigma^2/q\). Multiplying it by the fixed ridge shrinkage factor gives the expectation and variance of the ridge estimate:

E[\widehat{\theta}_{\lambda}] =\frac{q}{q+\lambda}\theta,\qquad \operatorname{Bias}(\widehat{\theta}_{\lambda}) =-\frac{\lambda}{q+\lambda}\theta,\qquad \operatorname{Var}(\widehat{\theta}_{\lambda}) =\frac{\sigma^2 q}{(q+\lambda)^2}.

The sign of the bias is opposite the sign of \(\theta\): ridge pulls the estimate toward zero. Its squared bias is nonnegative regardless of that sign. Combining squared bias and variance gives:

\operatorname{MSE}(\widehat{\theta}_{\lambda}) =\frac{\lambda^2\theta^2+\sigma^2q}{(q+\lambda)^2}, \qquad \operatorname{MSE}(\widehat{\theta}_{\text{OLS}}) =\frac{\sigma^2}{q}.

Derivation: Since \(\widehat{\theta}_{\lambda}=[q/(q+\lambda)]\widehat{\theta}_{\text{OLS}}\) and \(E[\widehat{\theta}_{\text{OLS}}]=\theta\), its expectation is \(q\theta/(q+\lambda)\). Subtracting \(\theta\) gives bias \(-\lambda\theta/(q+\lambda)\). Multiplying a random variable by a constant multiplies its variance by the square of that constant, so the ridge variance is \([q/(q+\lambda)]^2(\sigma^2/q)=\sigma^2q/(q+\lambda)^2\). Finally, writing \(\widehat{\theta}_{\lambda}-\theta\) as its centered random part plus its bias, the cross-product has expectation zero. Thus its MSE equals variance plus squared bias, yielding the displayed formula.

For \(\lambda>0\), the exact condition for ridge to have lower coefficient MSE than OLS in this direction follows by comparing the two expressions:

\operatorname{MSE}(\widehat{\theta}_{\lambda}) < \operatorname{MSE}(\widehat{\theta}_{\text{OLS}}) \quad\Longleftrightarrow\quad \lambda\theta^2<\sigma^2\left(2+\frac{\lambda}{q}\right).

To see the comparison, multiply both MSEs by the positive quantity \(q(q+\lambda)^2\). The inequality becomes \(q\lambda^2\theta^2+\sigma^2q^2<\sigma^2(q+\lambda)^2\). Expanding the right side and cancelling \(\sigma^2q^2\) gives \(q\lambda^2\theta^2<\sigma^2(2q\lambda+\lambda^2)\). Dividing by \(q\lambda>0\) gives the stated condition. The condition depends on the true coefficient, the noise variance, the design information, and the penalty; ridge is not guaranteed to improve MSE for every coefficient and every penalty.

Worked Example: Shrinkage Adds More Bias Than It Saves in Variance

Method and assumptions. Consider a hypothetical model for an insurance portfolio’s annual cost index, with one eigen-direction having \(q=4\), true coefficient \(\theta=3\), ridge penalty \(\lambda=2\), and error variance \(\sigma^2=4\). Assume \(\varepsilon\sim N(0,4I)\), conditional on the fixed design. Calculate ridge bias, variance, and MSE, and compare them with OLS.

Step 1: Find the shrinkage factor. The denominator is \(q+\lambda=4+2=6\), so \(q/(q+\lambda)=4/6=2/3\). Therefore \(E[\widehat{\theta}_{\lambda}]=(2/3)(3)=2\).

Step 2: Calculate the bias. The ridge estimate’s expectation is 2 while the true coefficient is 3. Hence its bias is \(2-3=-1\), and its squared bias is \((-1)^2=1\). Equivalently, \(-[\lambda/(q+\lambda)]\theta=-(2/6)(3)=-1\).

Step 3: Calculate variance and MSE. Ridge variance is \(\sigma^2q/(q+\lambda)^2=(4)(4)/6^2=16/36=4/9\). Its MSE is \(1+4/9=13/9\approx1.444\). OLS is unbiased and has variance and MSE \(\sigma^2/q=4/4=1\). Thus ridge has greater MSE here: \(13/9>1\).

Check and common mistake. The ridge variance, \(4/9\), is indeed below the OLS variance, 1, but that reduction does not offset the squared bias of 1. Counting only the variance reduction would incorrectly conclude that ridge is more accurate. The computed MSE comparison confirms that, for these assumptions and this coefficient, it is not.

Worked Example: Shrinkage Reduces MSE for a Small Coefficient

Method and assumptions. In a different hypothetical cost model, take \(q=2\), \(\theta=0.5\), \(\lambda=1\), and \(\varepsilon\sim N(0,4I)\), so \(\sigma^2=4\). Compare ridge and OLS coefficient MSE.

Step 1: Find the ridge expectation and bias. The shrinkage factor is \(2/(2+1)=2/3\). Thus \(E[\widehat{\theta}_{\lambda}]=(2/3)(0.5)=1/3\). The bias is \(1/3-1/2=-1/6\), with squared bias \(1/36\).

Step 2: Find ridge variance. Using the fixed-design variance formula gives \(\sigma^2q/(q+\lambda)^2=(4)(2)/3^2=8/9\).

Step 3: Compare MSEs. Ridge MSE is \(1/36+8/9=1/36+32/36=33/36=11/12\approx0.9167\). OLS MSE is \(\sigma^2/q=4/2=2\). Therefore ridge has lower MSE, \(11/12<2\).

Check and common mistake. The ridge MSE is below the OLS MSE even though ridge is biased. Do not compare the bias itself directly with the variance; the MSE uses squared bias plus variance. Here the added squared bias, \(1/36\), is small relative to the variance reduction from 2 to \(8/9\).

Combining Orthogonal Directions

For multiple coefficient directions, the same comparison applies direction by direction. Suppose \(X^\mathsf{T}X\) has orthogonal unit eigenvectors with positive eigenvalues \(q_j\), and let \(\theta_j\) be the true coefficient along direction \(j\). With errors having covariance \(\sigma^2I\), the coefficient-estimation MSE measured as the expected squared Euclidean error is the sum of the coordinate MSEs. Therefore:

E\!\left[\|\widehat{\beta}_{\lambda}-\beta\|^2\right] =\sum_j\left\{ \frac{\lambda^2\theta_j^2}{(q_j+\lambda)^2} +\frac{\sigma^2q_j}{(q_j+\lambda)^2} \right\}.

The corresponding OLS coefficient MSE is \(\sum_j \sigma^2/q_j\). A ridge estimate can improve the total even if its MSE is worse in one direction, provided its gains in other directions are larger. This is a coefficient-error calculation: prediction error weights directions according to the design and is not automatically equal to this sum.

Worked Example: Total Coefficient MSE Across Two Directions

Method and assumptions. Consider a hypothetical two-direction model with eigenvalues \(q_1=4\) and \(q_2=1\), true coefficients \(\theta_1=1\) and \(\theta_2=0.5\), penalty \(\lambda=1\), and \(\varepsilon\sim N(0,I)\), so \(\sigma^2=1\). Calculate the ridge and OLS total coefficient MSEs by summing the direction-specific terms.

Step 1: Direction 1. The shrinkage factor is \(4/(4+1)=4/5\). The bias is \(-(1/5)(1)=-1/5\), with squared bias \(1/25=0.04\). The variance is \((1)(4)/5^2=4/25=0.16\). The direction’s ridge MSE is \(0.04+0.16=0.20\).

Step 2: Direction 2. The shrinkage factor is \(1/(1+1)=1/2\). The bias is \(-(1/2)(0.5)=-0.25\), with squared bias \(0.0625\). The variance is \((1)(1)/2^2=0.25\). The direction’s ridge MSE is \(0.0625+0.25=0.3125\).

Step 3: Add the direction-specific MSEs. The total ridge coefficient MSE is \(0.20+0.3125=0.5125\). The OLS coefficient MSE is \(1/4+1/1=0.25+1=1.25\). Thus ridge has lower total coefficient MSE, \(0.5125<1.25\).

Check and common mistake. The second direction has more ridge shrinkage because its eigenvalue is smaller, but its MSE is still computed using its own \(q_2\), \(\theta_2\), and variance. Applying the first direction’s shrinkage factor or variance to both directions would give an incorrect total.

What the Comparison Does—and Does Not—Say

The calculations establish a bias-variance tradeoff, not a universal ranking of estimators. Increasing \(\lambda\) increases shrinkage and usually reduces variance, but also increases the magnitude of the bias when the true coefficient is nonzero. The balance depends on the unknown coefficient and the model’s noise level. In practice, a single coefficient’s true value is not known, so the formulas explain the mechanism rather than supplying a direct rule for choosing \(\lambda\).

A frequent exam trap is to say “ridge is better because its variance is lower.” Lower variance alone is not enough: the relevant comparison is \(\text{variance}+\text{squared bias}\). Another trap is to compare coefficient MSE with prediction error without accounting for how the design weights directions. Keep the target of the calculation explicit: here it is squared error in the coefficient estimate.

Key takeaway: In a direction with eigenvalue \(q>0\), ridge bias is \(-\lambda\theta/(q+\lambda)\), ridge variance is \(\sigma^2q/(q+\lambda)^2\), and ridge MSE is their squared-bias-plus-variance sum. Ridge improves on OLS only when that total is smaller than \(\sigma^2/q\); the comparison may differ across directions.

Check Your Understanding

For each hypothetical fixed-design model, use coefficient MSE and show the bias-variance tradeoff in the requested direction or directions.

Let \(q=9\), \(\theta=2\), \(\lambda=3\), and \(\varepsilon\sim N(0,9I)\). Find the ridge bias, variance, and MSE, then compare with OLS. (Answer: Ridge bias \(=-0.5\), variance \(=0.5625\), MSE \(=0.8125\); OLS MSE \(=1\), so ridge is lower.)
Let \(q=4\), \(\theta=1\), \(\lambda=4\), and \(\varepsilon\sim N(0,2I)\). Calculate both MSEs and state which is smaller. (Answer: Ridge bias \(=-0.5\), variance \(=0.125\), MSE \(=0.375\); OLS MSE \(=0.5\), so ridge is lower.)
Let \(q=1\), \(\theta=2\), \(\lambda=2\), and \(\varepsilon\sim N(0,I)\). Calculate both MSEs and state which is smaller. (Answer: Ridge bias \(=-4/3\), variance \(=1/9\), MSE \(=17/9\); OLS MSE \(=1\), so OLS is lower.)
There are two orthogonal directions with \(q_1=1\), \(q_2=9\), \(\theta_1=\theta_2=1\), \(\lambda=3\), and \(\varepsilon\sim N(0,I)\). Find the total ridge coefficient MSE and the total OLS coefficient MSE. (Answer: Ridge squared biases sum to \(0.625\), variances sum to \(0.125\), and total MSE is \(0.75\); OLS MSE is \(1+1/9=10/9\), so ridge is lower.)
For \(q=4\), \(\theta=0.5\), \(\lambda=2\), and \(\varepsilon\sim N(0,I)\), verify whether the exact condition for ridge to beat OLS holds, then calculate both MSEs. (Answer: \( \lambda\theta^2=0.5 < 1(2+\lambda/q)=2.5\), so it holds; ridge MSE \(=5/36\), compared with OLS MSE \(=1/4\).)

What You'll Learn