Shrinkage Bias in Lasso Predictions

Why Shrinkage Changes Prediction Error

Lasso does more than choose which coefficients to retain. For coefficients that remain nonzero, it generally pulls their estimates toward zero. That shrinkage can make estimates less variable across repeated training samples, while also moving their average away from the true coefficient. A lower-variance estimate is therefore not automatically a more accurate predictor: the relevant comparison includes both variance and squared bias.

The distinction matters because a model is often used to predict outcomes, not to recover the exact data-generating coefficients. Prediction quality depends on which covariate values will be encountered and how errors in the estimated coefficients combine at those values. This tutorial is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Definition: For a fixed test covariate vector \(x_0\), let the true conditional mean be \(f_0=x_0^\top\beta\), and let the fitted prediction from a random training sample be \(\hat f_0=x_0^\top\hat\beta\). The prediction bias is \(E[\hat f_0]-f_0\), the prediction variance is \(\operatorname{Var}(\hat f_0)\), and the mean squared prediction error for the conditional mean is \(E[(\hat f_0-f_0)^2]\).

These expectations are over repeated training samples under the stated model, with the test covariates held fixed. If the quantity to be predicted is a future response \(Y_0=f_0+\varepsilon_0\), and \(\varepsilon_0\) is independent of the training data with mean zero and variance \(\sigma^2\), then the expected squared error for predicting \(Y_0\) adds \(\sigma^2\), the irreducible response variance. That added term is common to competing models under the same data-generating process.

Bias and Variance in an Orthogonal Lasso Model

To see how shrinkage acts, consider a fixed design with centered response and predictors, and take the intercept to be absent or known to be zero. Suppose the columns are orthogonal and scaled so that \(X^\top X/n=I\). The lasso minimizes \((1/(2n))\|y-Xb\|^2+\lambda\|b\|_1\), where \(\lambda\geq 0\). For predictor \(j\), write \(z_j=x_j^\top y/n\), the corresponding unpenalized least-squares estimate in this setting.

Under these assumptions, the lasso estimate is the soft-thresholded value of \(z_j\). This coordinatewise expression makes the source of shrinkage explicit.

\hat\beta_j=S_\lambda(z_j) =\operatorname{sign}(z_j)(|z_j|-\lambda)_+ = \begin{cases} z_j-\lambda, & z_j>\lambda,\\ 0, & |z_j|\leq\lambda,\\ z_j+\lambda, & z_j<-\lambda. \end{cases}

Here \((a)_+=\max(a,0)\). The expression applies to this orthogonal, equally scaled design and the stated penalty convention; it is not a general coordinate-by-coordinate solution when predictors are correlated. The orthogonal setting is useful for isolating the bias-variance effect without complications from competing predictors.

Derivation: With orthogonal columns, the part of the objective involving one coefficient \(b_j\), after dropping terms that do not depend on it, is \(\tfrac12(b_j-z_j)^2+\lambda|b_j|\). For \(b_j>0\), differentiating gives \(b_j-z_j+\lambda=0\), so the candidate is \(z_j-\lambda\), valid when \(z_j>\lambda\). For \(b_j<0\), the derivative is \(b_j-z_j-\lambda=0\), giving \(z_j+\lambda\), valid when \(z_j<-\lambda\). At \(b_j=0\), the subgradient condition is \(0\in-z_j+\lambda[-1,1]\), equivalent to \(|z_j|\leq\lambda\). These three cases give the soft-thresholding formula.

For a fixed test vector \(x_0\), prediction error depends on the linear combination \(x_0^\top\hat\beta\), not on each coefficient considered in isolation. Its decomposition follows by adding and subtracting the mean prediction:

\begin{aligned} E[(\hat f_0-f_0)^2] &=E\left[(\hat f_0-E[\hat f_0]+E[\hat f_0]-f_0)^2\right]\\ &=\operatorname{Var}(\hat f_0)+(E[\hat f_0]-f_0)^2. \end{aligned}

The cross-product in the expansion has expectation zero because \(E[\hat f_0-E[\hat f_0]]=0\). This identity is exact when the expectations exist. It explains why variance reduction alone cannot determine which penalty gives lower prediction error: the squared bias must also be included.

Worked Examples: Variance Reduction and Bias Costs

Worked Example: Shrinkage Helps for a Null Coefficient

Suppose a one-predictor orthogonal model has true coefficient \(\beta=0\). Across repeated training samples, the unpenalized estimate \(Z\) equals \(-2\) or \(2\), each with probability \(1/2\). Apply lasso with \(\lambda=1\). Compare coefficient mean squared error and prediction variance at \(x_0=3\). This finite sampling distribution is specified to make the calculations exact.

Method. Apply soft thresholding to each possible value of \(Z\), then calculate the estimate's mean and variance. Since the true coefficient is zero, coefficient mean squared error equals the estimate's variance when the estimate is unbiased.

For \(Z=-2\), \(\hat\beta=S_1(-2)=-1\). For \(Z=2\), \(\hat\beta=S_1(2)=1\). Thus the lasso estimate takes \(-1\) and \(1\), each with probability \(1/2\), so \(E[\hat\beta]=0\), \(\operatorname{Var}(\hat\beta)=\tfrac12(-1)^2+\tfrac12(1)^2=1\), and its squared bias is zero. Its coefficient mean squared error is \(1\).

The unpenalized estimate takes \(-2\) and \(2\), each with probability \(1/2\). Its mean is zero, and its variance and coefficient mean squared error are both \(\tfrac12(-2)^2+\tfrac12(2)^2=4\). At \(x_0=3\), prediction is \(3\hat\beta\), so the lasso prediction variance is \(3^2(1)=9\), compared with \(3^2(4)=36\) for the unpenalized estimate. The lasso reduces prediction variance by a factor of four in this example and introduces no bias because the true coefficient is zero.

Check. The lasso prediction is equally likely to be \(-3\) or \(3\); its mean is zero, and its mean squared error for predicting the true mean \(3\beta=0\) is \(\tfrac12(9)+\tfrac12(9)=9\), matching its variance. The common mistake to avoid is to assume that shrinkage always creates bias: here the coefficient is truly zero, so the estimate remains unbiased.

Worked Example: Lower Variance but Higher Prediction Error

Now suppose the true coefficient is \(\beta=2\). Across repeated training samples, the unpenalized estimate \(Z\) equals \(1\) or \(3\), each with probability \(1/2\). Use \(\lambda=1.5\), and compare the lasso and unpenalized estimates for prediction at \(x_0=1\).

Method. Calculate each estimator's mean, variance, and squared bias. Add squared bias and variance to obtain its mean squared error for the conditional mean.

Soft thresholding gives \(S_{1.5}(1)=0\), since \(|1|\leq1.5\), and \(S_{1.5}(3)=3-1.5=1.5\). The lasso estimate therefore has mean \((0+1.5)/2=0.75\). Its bias is \(0.75-2=-1.25\), and its variance is \(\tfrac12(0-0.75)^2+\tfrac12(1.5-0.75)^2=0.5625\). Its mean squared error is

$$ (-1.25)^2+0.5625=1.5625+0.5625=2.125. $$

The unpenalized estimate has mean \((1+3)/2=2\), so its bias is zero. Its variance is \(\tfrac12(1-2)^2+\tfrac12(3-2)^2=1\), and its mean squared error is \(1\). Because \(x_0=1\), coefficient and prediction errors have the same numerical values here. Lasso reduces variance from \(1\) to \(0.5625\), but its squared bias of \(1.5625\) more than offsets that reduction: its prediction mean squared error is \(2.125\), higher than \(1\).

Check. The lasso errors relative to the true coefficient are \(-2\) and \(-0.5\), so their squared values average to \((4+0.25)/2=2.125\), in agreement with the bias-variance calculation. The common mistake to avoid is to compare the two estimates using variance alone and declare lasso better.

Prediction Accuracy Is Not Coefficient-Recovery Accuracy

Coefficient recovery and prediction are different objectives. A typical coefficient-recovery measure is the squared Euclidean error \(\|\hat\beta-\beta\|^2\), which weights coefficient errors without regard to the test covariate values. At a particular \(x_0\), prediction error is \((x_0^\top(\hat\beta-\beta))^2\). A coefficient error can have little effect on prediction when its corresponding covariate is small or absent in the prediction population. Conversely, a small error in a coefficient attached to a frequently large covariate can matter substantially.

Worked Example: Better Prediction Despite Worse Coefficient Recovery

A two-predictor model has true coefficients \(\beta=(1,2)\). For a particular fitted data set, suppose the unpenalized estimates are \(\hat\beta_{\text{OLS}}=(1.2,2.1)\), while the lasso estimates are \(\hat\beta_{\text{Lasso}}=(0.95,1.4)\). The target population has test covariate vector \(x_0=(1,0)\). Compare coefficient squared error with squared prediction error for this test population.

Method. First calculate each coefficient vector's squared Euclidean distance from \((1,2)\). Then calculate each prediction using \(x_0^\top\hat\beta\) and compare it with the true conditional mean \(x_0^\top\beta\). These are realized errors for the stated fit, not estimates of repeated-sample expected risk.

For OLS, the coefficient errors are \(1.2-1=0.2\) and \(2.1-2=0.1\). Its coefficient squared error is \(0.2^2+0.1^2=0.04+0.01=0.05\). For lasso, the coefficient errors are \(0.95-1=-0.05\) and \(1.4-2=-0.6\). Its coefficient squared error is \((-0.05)^2+(-0.6)^2=0.0025+0.36=0.3625\). On this measure, OLS is more accurate at recovering the coefficient vector.

For the target covariates, the true mean is \(x_0^\top\beta=(1)(1)+(0)(2)=1\). The OLS prediction is \((1)(1.2)+(0)(2.1)=1.2\), with squared prediction error \((1.2-1)^2=0.04\). The lasso prediction is \((1)(0.95)+(0)(1.4)=0.95\), with squared prediction error \((0.95-1)^2=0.0025\). Thus lasso predicts better for this target population even though its coefficient squared error is larger: the second coefficient, where its error is substantial, has no effect when the second test covariate is zero.

Check. The prediction comparison follows directly from the dot products with \((1,0)\), not from the coefficient distances. The common mistake to avoid is to treat a coefficient-recovery measure as if it necessarily ranked models by prediction accuracy.

How to Read the Trade-Off

The examples show three distinct possibilities. When a true coefficient is zero, shrinkage can reduce variance without adding bias. When a nonzero coefficient is shrunk, bias can outweigh the variance reduction and increase prediction error. And even when a coefficient vector is less accurately recovered overall, a fitted model can predict better for the covariate values that matter in the intended application.

For prediction at a fixed \(x_0\), the bias and variance of the fitted mean are

\operatorname{MSE}(\hat f_0) = \left(E[\hat f_0]-f_0\right)^2 + \operatorname{Var}(\hat f_0).

This is a repeated-sample statement, not a guarantee that one realized fit will have a particular error. The amount of shrinkage is controlled by \(\lambda\), but neither larger nor smaller \(\lambda\) can be declared preferable from the bias or variance alone. The relevant criterion is prediction error for the intended target, including the target covariate distribution and, when predicting future responses, the irreducible response noise.

Exam trap: Suppose an estimate has bias \(-1.25\) and variance \(0.5625\). It is incorrect to report its mean squared error as \(0.5625\) and conclude it beats an unbiased estimate with variance \(1\). The correct calculation is \((-1.25)^2+0.5625=1.5625+0.5625=2.125\), which is greater than \(1\). Always include squared bias when comparing mean squared errors.

A second trap is to equate the best coefficient estimate with the best prediction. A model's coefficient errors are not automatically weighted according to how its predictors appear in future cases. State the prediction target and its covariates before interpreting a coefficient-based accuracy measure. For a future response, also distinguish uncertainty in the fitted mean from the additional variation of the response around that mean.

Key takeaway: Lasso shrinkage can trade variance for squared bias. Compare models using prediction error for the intended target, and do not infer prediction quality solely from coefficient recovery or from variance reduction.

Check Your Understanding

For each question, distinguish bias from variance where required, and use the stated test covariates when calculating prediction error.

For a true coefficient \(\beta=0\), suppose \(Z=-1.5\) or \(1.5\), each with probability \(1/2\), and use \(\lambda=0.5\). Find the lasso coefficient variance and the lasso and unpenalized prediction variances at \(x_0=2\). (Answer: lasso coefficient variance \(1\); prediction variances \(4\) for lasso and \(9\) for unpenalized least squares.)
Let \(\beta=1.5\), \(Z=1\) or \(2\) with equal probability, and \(\lambda=1\). Calculate the lasso estimate's bias, variance, and mean squared error for the conditional mean at \(x_0=1\). (Answer: bias \(-1\), variance \(0.25\), mean squared error \(1.25\).)
Let \(\beta=0.8\), \(Z=0.3\) or \(1.3\) with equal probability, and \(\lambda=0.6\). Find the lasso estimate's mean, bias, variance, and mean squared error for \(x_0=1\), and compare its mean squared error with that of \(Z\). (Answer: lasso mean \(0.35\), bias \(-0.45\), variance \(0.1225\), mean squared error \(0.325\); the unpenalized mean squared error is \(0.25\).)
The true coefficient vector is \((1,2)\). Compare estimates \((1.2,2.1)\) and \((0.95,1.4)\) by coefficient squared error and by squared prediction error at \(x_0=(1,0)\). (Answer: coefficient squared errors are \(0.05\) and \(0.3625\), respectively; prediction squared errors are \(0.04\) and \(0.0025\), respectively.)

What You'll Learn