Why Derive Ridge Prediction Variance?
Ridge regression shrinks coefficient estimates toward zero. That changes the variability of fitted values across repeated samples, but a small variance alone does not guarantee accurate predictions: shrinkage can introduce bias. Deriving the variance makes the trade-off precise and helps distinguish the variability of an estimated mean response from the uncertainty in a future observed response.
The results below use a fixed design matrix and a linear model with independent, equal-variance errors. Predictors are centered and scaled as appropriate for the chosen ridge penalty, and the intercept is handled separately. Thus the formulas describe the penalized slope component. This setup is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
The assumption \(\operatorname{Cov}(\varepsilon\mid X)=\sigma^2 I_n\) means errors are uncorrelated and have a common variance. Normal errors are not needed for these variance calculations. If errors are correlated or have unequal variances, this covariance assumption fails and the formulas must be changed.
Ridge Coefficient and Fitted-Value Variance
For \(\lambda\geq 0\), the ridge estimate and the fitted values at the training observations are
When \(\lambda>0\), \(X^\top X+\lambda I_p\) is invertible, including when \(p>n\). For \(\lambda=0\), these expressions require the ordinary least-squares inverse to exist; in particular, \(X\) must have full column rank.
The first formula describes the sampling variability of the coefficient estimates. The second describes how that variability appears among fitted values at the training cases. Neither formula is the mean squared error: ridge can be biased because \(E[\hat{\beta}_{\lambda}\mid X]\) generally differs from \(\beta\) when \(\lambda>0\).
Why the Covariance Formulas Hold
The derivation uses the fact that a fixed linear transformation of a random vector transforms its covariance by multiplication on the left and right. From the model, \(y=X\beta+\varepsilon\), so \(\operatorname{Cov}(y\mid X)=\sigma^2 I_n\). Write \(A=X^\top X+\lambda I_p\). Since \(A\) is symmetric, the ridge estimate is a linear transformation of \(y\), and therefore
The fitted vector is \(H_{\lambda}y\), so applying the same covariance rule gives \(\operatorname{Cov}(\hat{y}_{\lambda}\mid X)=H_{\lambda}(\sigma^2I_n)H_{\lambda}^{\top}\). This derivation depends on conditioning on \(X\); if the design is treated as random, an unconditional variance calculation also has to account for its variation.
Comparing Ridge with Least Squares
The effect of shrinkage is clearest in the principal directions of the design. Let \(v_j\) be a unit eigenvector of \(X^\top X\) with positive eigenvalue \(d_j\), so \(X^\top Xv_j=d_jv_j\). Along this direction, the ridge coefficient variance is \(\sigma^2d_j/(d_j+\lambda)^2\). Least squares has coefficient variance \(\sigma^2/d_j\) in the same direction. For \(\lambda>0\),
Thus ridge reduces coefficient variance in every estimable direction under this penalty and scaling. For training fitted values, the corresponding variance in the left singular direction is \(\sigma^2d_j^2/(d_j+\lambda)^2\), compared with \(\sigma^2\) under least squares. This is also smaller for \(\lambda>0\).
For a new predictor vector \(x_0\), the estimated mean response is \(x_0^\top\hat{\beta}_{\lambda}\). Its variance, conditional on the design, is
When \(X\) has full column rank, write \(x_0=\sum_j c_jv_j\) in the eigenvector basis. The ridge variance is \(\sigma^2\sum_j c_j^2d_j/(d_j+\lambda)^2\); the least-squares variance is \(\sigma^2\sum_j c_j^2/d_j\). The term-by-term comparison shows that ridge has lower variance for this estimated mean response when \(\lambda>0\). This does not establish that its total prediction error is lower, because prediction error also reflects bias.
Worked Example: One Design Direction
Suppose \(X^\top X\) has a single eigenvalue \(d_1=4\), the error variance is \(\sigma^2=9\), and ridge uses \(\lambda=2\). A new predictor vector has coordinate \(c_1=2\) in this direction. Calculate the coefficient variance and the variance of the estimated mean response, then compare both with least squares.
Method and assumptions. Use the fixed-design formulas above. There is one estimable direction, and \(\lambda>0\), so the ridge inverse exists.
Ridge coefficient variance. Substitute \(d_1=4\), \(\sigma^2=9\), and \(\lambda=2\):
Least-squares coefficient variance. It is \(9/4=2.25\).
Variance for the estimated mean at \(x_0\). Multiply the coefficient variance by \(c_1^2=2^2=4\). Ridge gives \(4(1)=4\); least squares gives \(4(2.25)=9\). Thus shrinkage reduces this variance from \(9\) to \(4\), a reduction of \(5\), or \(5/9=55.56\%\). The check is consistent with the direction-by-direction comparison: both ridge variances are below their least-squares counterparts.
Common mistake avoided: Do not treat the coefficient variance, \(1\), as the prediction variance. The new predictor’s coordinate contributes a factor of \(c_1^2\).
Worked Examples in Multiple Directions
Worked Example: Combining Two Predictor Directions
Suppose \(X^\top X\) has eigenvalues \(d_1=1\) and \(d_2=9\), the error variance is \(\sigma^2=4\), and \(\lambda=1\). A new predictor has coordinates \(c_1=1\) and \(c_2=2\). Find its estimated-mean variance under ridge and least squares, and compare the total variances of the training fitted values.
Method. Calculate each direction separately and add the contributions. The directions are orthogonal in this eigenvector basis, so the variance expression is the sum of their squared-coordinate contributions.
Ridge estimated-mean variance. In direction 1 the coefficient variance is \(4(1)/(1+1)^2=1\). Its contribution at the new predictor is \(1^2(1)=1\). In direction 2 the coefficient variance is \(4(9)/(9+1)^2=36/100=0.36\). Its contribution is \(2^2(0.36)=1.44\). The total is \(1+1.44=2.44\).
Least-squares estimated-mean variance. The coefficient variances are \(4/1=4\) and \(4/9\). The prediction variance is
Training fitted-value variance. The ridge covariance eigenvalues in the two corresponding training directions are \(4(1^2)/(1+1)^2=1\) and \(4(9^2)/(9+1)^2=3.24\). Their sum, the trace of the ridge fitted-value covariance, is \(1+3.24=4.24\). Under least squares each of these two directions has variance \(\sigma^2=4\), so the trace is \(4+4=8\). Ridge reduces the total training fitted-value variance from \(8\) to \(4.24\). These totals summarize variation across directions; the new-point calculation is a separate, coordinate-weighted quantity.
Check and common mistake avoided: The ridge prediction contributions sum to \(2.44\), less than the least-squares value \(5.7778\). Do not compare the trace \(4.24\) directly with the new-point variance \(2.44\): they summarize different quantities.
Worked Example: Ridge When Predictors Outnumber Cases
Suppose \(p>n\) and \(X^\top X\) has two positive eigenvalues, \(d_1=4\) and \(d_2=1\), plus one zero eigenvalue. Let \(\sigma^2=1\), \(\lambda=3\), and let a new predictor have coordinates \(2\), \(1\), and \(5\) in these three directions. Calculate its ridge estimated-mean variance. Compare with the minimum-norm least-squares variance for the component in the row space of \(X\).
Method and assumptions. The ridge formula applies because \(X^\top X+3I_p\) is invertible. The zero-eigenvalue direction is not estimable from the data; the stated minimum-norm least-squares comparison sets its coefficient to zero and concerns only the row-space component.
Ridge variance in direction 1. The contribution is \(1\cdot 4/(4+3)^2\cdot 2^2=16/49\).
Ridge variance in direction 2. The contribution is \(1\cdot 1/(1+3)^2\cdot 1^2=1/16\).
Ridge variance in the zero-eigenvalue direction. Its contribution is zero because the variance formula contains the eigenvalue as a factor. Adding the contributions gives
Minimum-norm least-squares comparison. In the two estimable directions, the coefficient variances are \(1/4\) and \(1\). The prediction variance for the row-space component is \(2^2(1/4)+1^2(1)=2\). The null-space coordinate does not contribute to this minimum-norm estimate’s sampling variance. Ridge’s row-space variance, \(0.3890\), is lower than \(2\).
Check and common mistake avoided: The zero contribution is a variance statement, not evidence that the true coefficient in that direction is zero. If the true mean response depends on that direction, setting its coefficient to zero can create bias. Do not interpret low variance as proof of low prediction error.
What the Variance Reduction Does Not Mean
The calculations concern the estimated mean response, \(x_0^\top\hat{\beta}_{\lambda}\). If a future observation follows \(Y_0=x_0^\top\beta+\varepsilon_0\), where its error is independent of the training errors and has variance \(\sigma^2\), then
The first term is irreducible future-observation noise; the second is estimation variance. This variance identity does not include squared bias. Under the linear model, ridge has conditional expectation \(E[\hat{\beta}_{\lambda}\mid X]=(X^\top X+\lambda I_p)^{-1}X^\top X\beta\), which generally differs from \(\beta\). Therefore, choosing a smaller-variance estimator is not by itself enough to establish better predictive performance.
A second trap is to compare formulas without holding the predictor scaling and penalty convention fixed. Ridge penalizes coefficient size, so changing predictor scales changes the meaning of the penalty. The variance comparison applies to the stated design and penalty; standardizing predictors before fitting is important when predictors have different units. Also, if errors have covariance matrix \(\Sigma\) rather than \(\sigma^2I_n\), the coefficient covariance becomes \(A^{-1}X^\top\Sigma X A^{-1}\), not the equal-variance expression.
Check Your Understanding
Use the fixed-design formulas in each question. Answers are rounded only where shown.
- A one-direction design has eigenvalue \(d=5\), error variance \(\sigma^2=2\), and ridge penalty \(\lambda=0.5\). A new predictor has coordinate \(c=3\). Find the ridge estimated-mean variance. (Answer: \(360/121\approx 2.9752\))
- A design has eigenvalues \(2\) and \(8\), with \(\sigma^2=3\) and \(\lambda=2\). Find the two ridge training fitted-value covariance eigenvalues and their sum, then compare the sum with least squares. (Answer: \(0.75\) and \(1.92\); ridge sum \(2.67\), versus least-squares sum \(6\))
- A new predictor has coordinates \(1\) and \(-2\) along design directions with eigenvalues \(3\) and \(12\). If \(\sigma^2=5\) and \(\lambda=3\), calculate ridge and least-squares estimated-mean variances. (Answer: ridge \(89/60\approx1.4833\); least squares \(10/3\approx3.3333\))
- In a design with two positive eigenvalues \(9\) and \(1\) and one zero eigenvalue, \(\sigma^2=2\), \(\lambda=1\), and a new predictor has coordinates \(3\), \(2\), and \(7\). Find the ridge estimated-mean variance and the minimum-norm least-squares variance for its row-space component. (Answer: ridge \(3.62\); minimum-norm least squares \(10\))