Worked Risk Calculations For Linear Regression

Why Regression Risk Depends on the Target

In regression, an estimator can be judged by how accurately it estimates the coefficient vector or by how accurately it predicts a response. These are different questions: a coefficient error in a direction that the design barely measures may count fully under coefficient loss but contribute little to prediction risk. Ridge regression adds another distinction, because shrinkage can lower variance while introducing bias.

We will calculate both kinds of risk for a fixed orthogonal design with Gaussian errors. The calculations make explicit which quantities depend on the coefficient values, the design, the ridge penalty, and the prediction point.

Model, Estimators, and Risk Definitions

Let $Y=X\beta+\varepsilon$, where $X$ is a fixed $n\times p$ matrix of full column rank, $\beta\in\mathbb{R}^p$ is fixed, and $\varepsilon\sim N_n(0,\sigma^2 I_n)$, with $\sigma^2>0$. Assume the columns of $X$ are orthogonal, so $X^\mathsf{T}X=D=\operatorname{diag}(d_1,\ldots,d_p)$ with every $d_j>0$. We use ridge regression without an unpenalized intercept: for a fixed $\lambda\geq 0$, it minimizes $\|Y-Xb\|^2+\lambda\|b\|^2$.

Definition: The least-squares estimator is $\hat{\beta}_{LS}=(X^\mathsf{T}X)^{-1}X^\mathsf{T}Y$, and the ridge estimator is $\hat{\beta}_{R}=(X^\mathsf{T}X+\lambda I_p)^{-1}X^\mathsf{T}Y$. Under Euclidean coefficient loss, their risks are $E_\beta[\|\hat{\beta}-\beta\|^2]$. For a fixed vector $x_0\in\mathbb{R}^p$, their mean-response prediction risks at $x_0$ are $E_\beta[(x_0^\mathsf{T}\hat{\beta}-x_0^\mathsf{T}\beta)^2]$.

The pointwise prediction target here is the mean response $x_0^\mathsf{T}\beta$, not a future observation that also contains error. If the future response is $Y_0=x_0^\mathsf{T}\beta+\varepsilon_0$, where $\varepsilon_0$ is independent with mean zero and variance $\sigma^2$, its prediction risk is the mean-response risk plus $\sigma^2$.

Orthogonality gives a separate scalar calculation for each coefficient. Since $X^\mathsf{T}Y=D\beta+X^\mathsf{T}\varepsilon$, and the entries of $X^\mathsf{T}\varepsilon$ are independent normal variables with variances $\sigma^2d_j$, the $j$th least-squares estimate has mean $\beta_j$ and variance $\sigma^2/d_j$. Ridge multiplies that estimate by $a_j=d_j/(d_j+\lambda)$.

Theorem: Under the stated model, the coordinate errors of both estimators are independent, and $$ \hat{\beta}_{LS,j}\sim N\!\left(\beta_j,\frac{\sigma^2}{d_j}\right), \qquad \hat{\beta}_{R,j}\sim N\!\left(a_j\beta_j,\frac{a_j^2\sigma^2}{d_j}\right), \qquad a_j=\frac{d_j}{d_j+\lambda}. $$ Consequently, the Euclidean coefficient risks are $$ R_{LS}(\beta)=\sum_{j=1}^p\frac{\sigma^2}{d_j}, \qquad R_R(\beta)=\sum_{j=1}^p\left\{(1-a_j)^2\beta_j^2+\frac{a_j^2\sigma^2}{d_j}\right\}. $$

Proof. The $j$th coordinate of $X^\mathsf{T}Y$ is $d_j\beta_j+X_j^\mathsf{T}\varepsilon$, where $X_j$ is column $j$ of $X$. The noise term is normal with variance $\sigma^2X_j^\mathsf{T}X_j=\sigma^2d_j$. For distinct coordinates $j,k$, its covariance with $X_k^\mathsf{T}\varepsilon$ is $\sigma^2X_j^\mathsf{T}X_k=0$. Joint normality therefore implies independence. Dividing coordinate $j$ by $d_j$ gives the stated least-squares distribution; multiplying by $a_j$ gives the ridge distribution. For any scalar random variable $W$ with finite second moment, $E[(W-c)^2]=\operatorname{Var}(W)+(E[W]-c)^2$. Applying this identity to each coordinate and summing proves both risk formulas. $\square$

Pointwise and Design-Average Prediction Risk

A prediction is a linear combination of coefficient estimates. Its bias is the same linear combination of the coefficient biases, so those biases must be combined before squaring. The independent coordinate variances, by contrast, add with squared prediction weights.

Theorem: For any fixed $x_0=(x_{01},\ldots,x_{0p})^\mathsf{T}$, the mean-response prediction risks are $$ R_{LS}(x_0)=\sum_{j=1}^p x_{0j}^2\frac{\sigma^2}{d_j}, \qquad R_R(x_0)= \left\{\sum_{j=1}^p x_{0j}(a_j-1)\beta_j\right\}^2 +\sum_{j=1}^p x_{0j}^2\frac{a_j^2\sigma^2}{d_j}. $$

Proof. Write the prediction error as $\sum_j x_{0j}(\hat{\beta}_j-\beta_j)$. Its expectation is $\sum_jx_{0j}E[\hat{\beta}_j-\beta_j]$, and its variance is $\sum_jx_{0j}^2\operatorname{Var}(\hat{\beta}_j)$, because the coordinate errors are independent. The squared-error risk equals this variance plus the square of this expectation. For least squares the expectation is zero and the coordinate variances are $\sigma^2/d_j$; for ridge they are $a_j^2\sigma^2/d_j$ and the coordinate biases are $(a_j-1)\beta_j$. Substitution yields the formulas. $\square$

For predictions at the observed design points, the total squared error in fitted mean responses is $\|X\hat{\beta}-X\beta\|^2$. The average risk over those $n$ points is $$ \frac{1}{n}E_\beta[\|X\hat{\beta}-X\beta\|^2] =\frac{1}{n}\sum_{j=1}^p d_j E_\beta[(\hat{\beta}_j-\beta_j)^2]. $$ This follows from $\|Xv\|^2=v^\mathsf{T}X^\mathsf{T}Xv=v^\mathsf{T}Dv=\sum_jd_jv_j^2$, applied to $v=\hat{\beta}-\beta$, and then taking expectations. Thus design-average prediction risk weights each coefficient’s mean squared error by its column length squared.

Worked Risk Calculations

Worked Example: Coefficient Risk With Unequal Column Lengths

Suppose $X^\mathsf{T}X=\operatorname{diag}(4,1)$, $\sigma^2=1$, $\lambda=2$, and the true coefficient is $\beta=(1,2)^\mathsf{T}$. The shrinkage factors are $a_1=4/(4+2)=2/3$ and $a_2=1/(1+2)=1/3$.

For least squares, the coefficient biases are zero and the variances are $1/4$ and $1$. Therefore $R_{LS}=1/4+1=5/4$.

For ridge, the coordinate biases are $(2/3-1)1=-1/3$ and $(1/3-1)2=-4/3$. The coordinate variances are $(2/3)^2(1/4)=1/9$ and $(1/3)^2(1)=1/9$. Hence $R_R=(1/9+1/9)+(1/9+16/9)=19/9$, which is larger than $5/4$. Here the squared bias introduced by shrinkage outweighs its variance reduction.

Worked Example: A Prediction Error With Two Shrunk Coefficients

Now take $X^\mathsf{T}X=4I_2$, $\sigma^2=1$, $\lambda=2$, $\beta=(1,1)^\mathsf{T}$, and predict the mean response at $x_0=(1,1)^\mathsf{T}$. Both ridge factors equal $2/3$. Each ridge coefficient has bias $-1/3$ and variance $(2/3)^2/4=1/9$.

The ridge prediction bias is the sum of the two coordinate biases, $-1/3-1/3=-2/3$. Its square is $4/9$. The prediction variance is $1/9+1/9=2/9$, so the ridge prediction risk is $4/9+2/9=2/3$.

For least squares, the prediction bias is zero and the variance is $1/4+1/4=1/2$. Thus its prediction risk is $1/2$, lower than ridge risk at this target. Although the sum of the two ridge coefficient mean squared errors is $2/9+2/9=4/9$, that sum is not the prediction risk: the squared total bias includes the positive cross-term $2(-1/3)(-1/3)=2/9$.

Worked Example: When Ridge Improves Design-Average Prediction

Use $X^\mathsf{T}X=\operatorname{diag}(4,1)$, $\sigma^2=1$, $\lambda=2$, and $\beta=(1/5,1/5)^\mathsf{T}$. The two ridge factors are again $2/3$ and $1/3$. The least-squares coefficient mean squared errors are $1/4$ and $1$, so the total squared error risk across fitted means is $4(1/4)+1(1)=2$. If $n=5$, the average risk per design point is $2/5$.

For ridge, coordinate 1 has squared bias $(-(1/3)(1/5))^2=1/225$ and variance $1/9=25/225$, giving mean squared error $26/225$. Coordinate 2 has squared bias $(-(2/3)(1/5))^2=4/225$ and variance $1/9=25/225$, giving $29/225$. The total risk across fitted means is therefore $4(26/225)+29/225=133/225$. Dividing by five gives average risk $133/1125$, about $0.118$, less than the least-squares value $2/5=0.4$. In this example the coefficients are small enough that the reduction in variance more than offsets ridge bias.

How to Read the Comparison

These calculations do not imply that ridge is always better or always worse. For coefficient risk, its coordinatewise contribution is $(1-a_j)^2\beta_j^2+a_j^2\sigma^2/d_j$, while least squares contributes $\sigma^2/d_j$. Ridge helps in coordinate $j$ precisely when $$ (1-a_j)^2\beta_j^2<(1-a_j^2)\frac{\sigma^2}{d_j}. $$ For $0<a_j<1$, dividing by $1-a_j$ gives the equivalent condition $(1-a_j)\beta_j^2<(1+a_j)\sigma^2/d_j$. The condition depends on the unknown coefficient, which is why a risk calculation at a specified parameter is not by itself a universal guarantee.

For a point prediction, the bias terms from different coefficients can reinforce or offset one another, and the prediction weights $x_{0j}$ determine which coordinates matter. A penalty that improves average prediction over the observed design can still worsen prediction at a particular $x_0$. If the target is a future response rather than its mean, add $\sigma^2$ to each method’s pointwise risk under the independent-error setup; this common addition does not change their comparison.

Common Pitfall

Adding coordinate mean squared errors to get pointwise prediction risk. This omits cross-products between coordinate biases. First calculate the total prediction bias $\sum_jx_{0j}E[\hat{\beta}_j-\beta_j]$, then square it; add the prediction variance $\sum_jx_{0j}^2\operatorname{Var}(\hat{\beta}_j)$ afterward.
Confusing coefficient risk with prediction risk. Coefficient risk uses unweighted Euclidean error, while design-average prediction risk weights coordinate errors by $d_j$. State the target and loss before comparing risks.
Treating ridge bias as a variance term. Ridge variance is reduced by $a_j^2$, but its bias is $(a_j-1)\beta_j$. Both terms belong in squared-error risk.

Key takeaway: Under an orthogonal Gaussian design, least-squares and ridge risks reduce to coordinate calculations. Coefficient risk sums coordinate mean squared errors; pointwise prediction risk instead squares the combined prediction bias and adds the weighted variances.

Check Your Understanding

Use the formulas and distinguish the target of each risk calculation.

For one coordinate with $d_j=9$, $\sigma^2=2$, $\lambda=3$, and $\beta_j=1$, calculate the least-squares and ridge coefficient mean squared errors.
Why does pointwise ridge prediction risk generally not equal the sum of the coordinate ridge mean squared errors when $x_0=(1,1)^\mathsf{T}$?
If a fixed prediction point has $x_{02}=0$, how does coordinate 2 contribute to its mean-response prediction risk?
For $X^\mathsf{T}X=D$, explain why average in-sample prediction risk weights coordinate $j$'s mean squared error by $d_j/n$.
How does the risk change when the target is an independent future response rather than its mean response?