Bias-Variance Trade-Off Under Ridge Shrinkage

Why Shrinkage Changes Both Bias and Variance

In regression, correlated predictors can make coefficient estimates unstable: small changes in the data may produce large changes in the fitted coefficients. The earlier material on correlated predictors explains why this variability matters. Ridge regression addresses it by shrinking coefficient estimates toward zero. That can make an estimate systematically different from the true coefficient, but it can also reduce how much the estimate varies from sample to sample.

This is the bias-variance trade-off. It is not a guarantee that every amount of shrinkage improves a model. Its value depends on the coefficient, the estimator’s sampling variability, and—when the goal is prediction—the predictor values at which accuracy matters. The calculations below isolate the trade-off for one coefficient using a fixed shrinkage factor.

Definition: Let \(\beta\) be a fixed true coefficient, and let \(\hat{\beta}\) be an unbiased estimator with \(E[\hat{\beta}]=\beta\) and \(\operatorname{Var}(\hat{\beta})=\tau^2\). For a fixed shrinkage factor \(c\), define \(\tilde{\beta}=c\hat{\beta}\). Its bias is \(E[\tilde{\beta}]-\beta\), its variance is \(\operatorname{Var}(\tilde{\beta})\), and its mean squared error is \(E[(\tilde{\beta}-\beta)^2]\). The usual shrinkage range is \(0\leq c\leq 1\): \(c=1\) leaves the estimate unchanged, while \(c=0\) sets it to zero.

The MSE is the expected squared error in estimating \(\beta\), averaged over repeated samples under the stated model. It is not the squared error from one particular fitted data set. For an unbiased estimator, the MSE equals its variance. Once the estimator is shrunk, its bias must also be included.

Deriving the Bias-Variance Calculation

The following identity holds for any estimator with a finite second moment: its MSE is its variance plus the square of its bias. To see why, write the error as the centered random part plus the bias:

\tilde{\beta}-\beta = \bigl(\tilde{\beta}-E[\tilde{\beta}]\bigr) + \bigl(E[\tilde{\beta}]-\beta\bigr).

Squaring and taking expectations gives a variance term, a squared-bias term, and a cross term. The cross term is zero because \(E[\tilde{\beta}-E[\tilde{\beta}]]=0\). Thus:

E[(\tilde{\beta}-\beta)^2] = \operatorname{Var}(\tilde{\beta}) + \bigl(E[\tilde{\beta}]-\beta\bigr)^2.

For \(\tilde{\beta}=c\hat{\beta}\), with fixed \(c\) and an unbiased \(\hat{\beta}\), the expectation and variance are:

E[\tilde{\beta}]=c\beta, \qquad \operatorname{Bias}(\tilde{\beta})=(c-1)\beta, \qquad \operatorname{Var}(\tilde{\beta})=c^2\tau^2.

The variance scales by \(c^2\), not by \(c\), because multiplying a random variable by a constant multiplies its variance by the square of that constant. Combining the squared bias and variance gives:

R(c) = E[(c\hat{\beta}-\beta)^2] = (1-c)^2\beta^2+c^2\tau^2.

At \(c=1\), the estimator is unbiased and \(R(1)=\tau^2\). For \(c<1\), the estimate is biased unless \(\beta=0\), but its variance is smaller than \(\tau^2\) whenever \(0\leq c<1\) and \(\tau^2>0\). Whether the total risk falls depends on the balance between these two changes.

Worked Example: Checking Whether Moderate Shrinkage Helps

Suppose a hypothetical claim-cost model has true coefficient \(\beta=1\). Across repeated samples, the ordinary unbiased coefficient estimate follows \(\hat{\beta}\sim N(1,0.36)\), where the second parameter is the variance. Apply a fixed shrinkage factor \(c=0.9\), and compare the coefficient MSE with that of the unshrunk estimate.

Method. Use the bias-plus-variance decomposition. The assumptions are a fixed true coefficient, an unbiased estimate with variance \(0.36\), and a nonrandom shrinkage factor \(0.9\).

Bias. The expectation of the shrunken estimate is \(0.9(1)=0.9\), so its bias is \(0.9-1=-0.1\). Its squared bias is \((-0.1)^2=0.01\).

Variance. The variance is \(0.9^2(0.36)=0.81(0.36)=0.2916\).

MSE and comparison. The shrunken MSE is \(0.01+0.2916=0.3016\). The unshrunk estimate is unbiased, so its MSE is its variance, \(0.36\). Since \(0.3016<0.36\), this shrinkage improves expected coefficient accuracy in this example.

Check. The shrunken estimate has lower variance than the original estimate, and the added squared bias, \(0.01\), is smaller than the variance reduction, \(0.36-0.2916=0.0684\). The net MSE reduction is \(0.0684-0.01=0.0584\), matching \(0.36-0.3016=0.0584\). The common mistake is to use \(0.9\) as the bias; the bias is the difference between the shrunken estimate’s expectation and the true value, namely \(-0.1\).

The Shrinkage Level That Minimizes Scalar Risk

The risk expression also shows how to choose \(c\) when \(\beta\) and \(\tau^2\) are known. Differentiate the quadratic risk with respect to \(c\):

R'(c)=2(c-1)\beta^2+2c\tau^2 =2c(\beta^2+\tau^2)-2\beta^2.

If \(\beta^2+\tau^2>0\), setting this derivative equal to zero gives the unique minimizing factor. The second derivative is \(R''(c)=2(\beta^2+\tau^2)>0\), so the risk is strictly convex and this stationary point is a minimum:

c^*=\frac{\beta^2}{\beta^2+\tau^2}.

For a nonzero \(\beta\) and positive variance, this factor lies strictly between zero and one. Greater sampling variance relative to the squared coefficient leads to more shrinkage. The formula is an explanatory result, not usually a directly available fitting rule: in practice, the true coefficient and its sampling distribution are unknown.

There is one degenerate case to handle separately. If \(\beta=0\) and \(\tau^2=0\), the estimate equals zero with probability one, and \(R(c)=0\) for every \(c\). The denominator in the formula for \(c^*\) is then zero, so the formula does not apply. If \(\beta=0\) but \(\tau^2>0\), the formula does apply and gives \(c^*=0\).

Worked Example: Finding the Minimum-Risk Shrinkage Factor

For a hypothetical severity model, suppose \(\beta=2\) and \(\hat{\beta}\sim N(2,1)\). Find the fixed shrinkage factor that minimizes coefficient MSE, calculate the resulting MSE, and compare it with the unshrunk estimate.

Method. Substitute the known values into the scalar risk formula and minimize the resulting quadratic. Here \(\beta^2=4\), \(\tau^2=1\), and \(\beta^2+\tau^2=5>0\), so the minimizing-factor formula applies.

Minimizing factor. \(c^*=4/(4+1)=0.8\).

Squared bias at \(c=0.8\). The bias is \((0.8-1)(2)=-0.4\), so its square is \(0.16\).

Variance at \(c=0.8\). The variance is \(0.8^2(1)=0.64\).

MSE and comparison. The shrunken MSE is \(0.16+0.64=0.80\). The unshrunk estimator has MSE \(1\), since it is unbiased with variance \(1\). Thus the minimum-risk shrinkage lowers MSE by \(1-0.80=0.20\).

Check. Differentiating \(R(c)=4(1-c)^2+c^2\) gives \(R'(c)=10c-8\), which is zero at \(c=0.8\). The common mistake is to choose the factor by looking only at the variance reduction; the selected factor minimizes the sum of squared bias and variance.

From Coefficient Risk to Prediction Risk

The quantity of interest in predictive analytics is often a prediction, not the coefficient by itself. Consider a one-predictor model for the conditional mean, \(E[Y\mid x]=\alpha+x\beta\), where the intercept \(\alpha\) is known and held fixed for this calculation. At a specified predictor value \(x_0\), the estimated conditional mean is \(\alpha+x_0\tilde{\beta}\). Its estimation error is \(x_0(\tilde{\beta}-\beta)\), so its expected squared error is \(x_0^2R(c)\). If \(x_0=0\), coefficient error has no effect on this particular conditional-mean prediction.

For a future response \(Y_0=\alpha+x_0\beta+\varepsilon_0\), assume the future error is independent of the coefficient estimate, has mean zero, and has variance \(\sigma^2\). Then the expected squared error for predicting the future response includes this irreducible variation:

E[(Y_0-(\alpha+x_0\tilde{\beta}))^2] = \sigma^2+x_0^2R(c).

To justify the addition, write the prediction error as \(\varepsilon_0-x_0(\tilde{\beta}-\beta)\). Its squared expectation is the sum of the two variances and squared biases; the cross term is zero because the future error is independent and mean zero. The same future-response variance \(\sigma^2\) is present for the shrunken and unshrunk predictions, so it does not change which has lower expected squared error under these assumptions.

Worked Example: Comparing Expected Accuracy at a Particular Risk Profile

A hypothetical model predicts a future claim-cost measure using one predictor. Let the true slope be \(\beta=0.2\), and suppose \(\hat{\beta}\sim N(0.2,0.01)\). Use shrinkage \(c=0.5\). At \(x_0=3\), compare expected squared error for the conditional-mean prediction with the unshrunk fit. Also suppose a future response has an independent error \(\varepsilon_0\sim N(0,0.04)\), and compare future-response prediction error.

Method. First compute coefficient MSE, then multiply it by \(x_0^2\). Add the future-response variance only for prediction of an individual future response. The intercept is treated as known and identical under both methods.

Coefficient bias and variance. The shrunken estimate has bias \((0.5-1)(0.2)=-0.1\), with squared bias \(0.01\). Its variance is \(0.5^2(0.01)=0.0025\). Therefore its coefficient MSE is \(0.01+0.0025=0.0125\). The unshrunk coefficient MSE is \(0.01\).

Conditional-mean prediction. Since \(x_0^2=3^2=9\), the shrunken prediction’s expected squared error is \(9(0.0125)=0.1125\). The unshrunk prediction’s expected squared error is \(9(0.01)=0.09\). Shrinkage is worse at this predictor value.

Future-response prediction. Add \(\sigma^2=0.04\) to each result. The shrunken method has expected squared error \(0.1125+0.04=0.1525\); the unshrunk method has \(0.09+0.04=0.13\).

Check. In both comparisons the shrunken method is worse by \(0.1525-0.13=0.0225\), the same as \(0.1125-0.09\). The future-response noise raises both errors equally. The common mistake is to claim that reducing variance must improve prediction; here the added squared bias exceeds the variance reduction.

What the Scalar Illustration Does—and Does Not—Show

The calculations establish a possibility, not a universal outcome. Shrinkage helps when the reduction in variance is larger than the squared bias it introduces. In the scalar model, the exact comparison with no shrinkage is:

R(c)-R(1) = (1-c)^2\beta^2-(1-c^2)\tau^2.

For \(0\leq c<1\), the first term is the added squared bias and the second term is the variance reduction. The risk falls precisely when \((1-c)^2\beta^2<(1-c^2)\tau^2\). At \(c=1\), neither changes. This comparison also explains why a shrinkage factor that works well for one coefficient or predictor value need not work well for another.

In a regression with several predictors, prediction risk depends on the full coefficient-error pattern and the predictor distribution. For a fixed predictor vector \(x_0\), if the coefficient estimator has bias vector \(b=E[\tilde{\boldsymbol{\beta}}]-\boldsymbol{\beta}\) and covariance matrix \(V\), the conditional-mean prediction error is:

E\left[ \left(x_0^\mathsf{T}\tilde{\boldsymbol{\beta}} -x_0^\mathsf{T}\boldsymbol{\beta}\right)^2 \right] = x_0^\mathsf{T}Vx_0+(x_0^\mathsf{T}b)^2.

Thus, coefficient bias can matter differently across prediction profiles: the relevant mean-prediction bias is \(x_0^\mathsf{T}b\). Across a population of prediction cases, the average risk depends on how common and how important those profiles are. A lower coefficient variance alone does not establish lower average prediction error.

The scalar factor \(c\) is a transparent illustration of the trade-off; it should not be mistaken for a claim that ridge regression multiplies every fitted coefficient by the same number. In actual ridge regression, the effect of the penalty depends on the design and the direction in coefficient space. The practical question is therefore whether the fitted procedure predicts better for the intended use, not merely whether its coefficients are smaller or less variable.

Common Exam Traps

Using the shrinkage factor as the bias. The bias is \(E[c\hat{\beta}]-\beta=(c-1)\beta\), not \(c\), \(1-c\), or \(c\beta\) by itself.
Forgetting to square the bias. MSE uses \(\operatorname{Bias}^2\). A negative bias still contributes a nonnegative amount.
Scaling variance by \(c\). The correct variance is \(c^2\tau^2\).
Assuming shrinkage always improves accuracy. It lowers variance for \(c<1\), but may add more squared bias than the variance reduction saves.
Confusing conditional-mean and future-response prediction. Under independent mean-zero future error, future-response error includes the additional variance \(\sigma^2\); conditional-mean estimation error does not.
Applying the minimizing formula in the degenerate case. When \(\beta=\tau^2=0\), the denominator is zero and every shrinkage factor has zero risk.

Key takeaway: Under the scalar assumptions, shrinkage changes coefficient MSE from \(\tau^2\) to \((1-c)^2\beta^2+c^2\tau^2\). It can improve expected accuracy when its variance reduction outweighs its added squared bias. Prediction risk then depends on how coefficient errors affect the predictor values that matter.

Check Your Understanding

For each problem, use the stated distribution and assumptions. Answers are given so you can check both the calculation and the interpretation.

A hypothetical pricing model has \(\beta=3\), with \(\hat{\beta}\sim N(3,4)\). If \(c=0.6\), calculate the shrunken estimate’s bias, variance, and MSE, and compare its MSE with the unshrunk estimate. (Answer: bias \(=-1.2\), variance \(=1.44\), MSE \(=2.88\), less than the unshrunk MSE \(4\).)
For a hypothetical retention model, \(\beta=-2\) and \(\hat{\beta}\sim N(-2,3)\). With \(c=0.75\), calculate the signed bias, squared bias, variance, and MSE. Does shrinkage improve on the unshrunk MSE? (Answer: bias \(=0.5\), squared bias \(=0.25\), variance \(=1.6875\), MSE \(=1.9375\); yes, it is below \(3\).)
Suppose \(\beta=1\) and \(\hat{\beta}\sim N(1,3)\). Find the MSE-minimizing fixed shrinkage factor and its MSE, and compare with \(c=1\). (Answer: \(c^*=1/(1+3)=0.25\); squared bias \(=0.5625\), variance \(=0.1875\), MSE \(=0.75\), compared with unshrunk MSE \(3\).)
A hypothetical one-predictor claim model has \(\beta=1\), \(\hat{\beta}\sim N(1,0.25)\), and uses \(c=0.8\). At \(x_0=-2\), calculate the conditional-mean prediction MSE. If the future response has independent error \(N(0,0.3)\), calculate its prediction MSE too, and compare both with the unshrunk values. (Answer: coefficient MSE \(=0.04+0.16=0.20\); conditional-mean prediction MSE \(=4(0.20)=0.80\), versus \(1.00\) unshrunk; future-response MSE \(=1.10\), versus \(1.30\) unshrunk.)
Let \(\beta=0\) and \(\tau^2=0\), so the unbiased estimator equals zero with probability one. State the risk for any fixed \(c\), and explain whether the formula for \(c^*\) can be used. (Answer: \(R(c)=0\) for every \(c\); the formula cannot be used because its denominator \(\beta^2+\tau^2\) is zero.)

What You'll Learn