Shrinkage Bias and Predictive Risk

Why a Biased Coefficient Can Predict Better

Penalization deliberately pulls fitted coefficients toward a target, usually zero. That makes coefficient estimates biased when the target is not the true coefficient. Yet the aim of a predictive model is not necessarily to recover every coefficient without bias: it is to predict well on new data. A small, systematic displacement can be worthwhile if it reduces the variability of predictions enough to lower their average out-of-sample error.

The relevant trade-off depends on the prediction target and loss. Squared error for a predicted mean leads to an exact bias–variance decomposition. For a GLM's predictive distribution, expected negative log-likelihood has a different but closely related interpretation: its excess over the true model's risk is a Kullback–Leibler divergence. These provide precise ways to discuss when shrinkage helps, and why coefficient bias by itself does not answer the question.

Prediction Risk and Its Decomposition

Let \(X\) denote the training data, and let \(\hat\beta\) be a coefficient estimator computed from \(X\). At a fixed covariate vector \(x\), write \(\hat\mu(x)\) for its predicted conditional mean, and let \(\mu_0(x)=E[Y\mid x]\) be the true conditional mean. The training data are random, so \(\hat\mu(x)\) is random even when the prediction point \(x\) is fixed.

For any estimator with finite second moment, the risk decomposes exactly. This statement concerns the predicted quantity, not automatically the coefficient vector.

If the target is a future response \(Y_{\mathrm{new}}\) at \(x\), and it is conditionally independent of the training data given \(x\), then \(E[(Y_{\mathrm{new}}-\hat\mu(x))^2]=\operatorname{Var}(Y_{\mathrm{new}}\mid x)+R_\mu(x)\). The first term is irreducible response variation; an estimator cannot remove it. Comparing two fits under squared prediction error therefore reduces to comparing their risks for the conditional mean.

For a linear predictor \(\hat\beta\) and a fixed vector \(c\), the same algebra gives \(E[(c^\mathsf{T}\hat\beta-c^\mathsf{T}\beta_0)^2]=(c^\mathsf{T}\operatorname{Bias}(\hat\beta))^2+c^\mathsf{T}\operatorname{Cov}(\hat\beta)c\). A GLM's mean prediction is generally a nonlinear function of \(x^\mathsf{T}\hat\beta\), so this coefficient identity is not, by itself, an exact decomposition for that mean. For a smooth inverse link \(h=g^{-1}\), a local approximation around \(\beta_0\) uses the gradient \(a(x)=h'(x^\mathsf{T}\beta_0)x\): the prediction error is approximately \(a(x)^\mathsf{T}(\hat\beta-\beta_0)\). Its approximate risk is therefore the squared projected bias plus projected covariance. This is a local approximation, not a universal exact formula.

GLM Prediction Loss and Kullback–Leibler Risk

Squared error is natural for a conditional mean, but GLMs also provide a predictive distribution. Suppose the response has a canonical exponential-family distribution \(f(y\mid\theta,\phi)=\exp\{[y\theta-b(\theta)]/a(\phi)+c(y,\phi)\}\), with known \(a(\phi)>0\), and the true conditional canonical parameter at \(x\) is \(\theta_0\). The mean is \(b'(\theta_0)\). Compare the expected negative log-likelihood of a fitted canonical parameter \(\theta\) with that of the truth, averaging the response under the true distribution.

To derive the equality, subtract the two negative log densities. The terms \(c(Y,\phi)\) cancel, leaving \(\{b(\theta)-b(\theta_0)-Y(\theta-\theta_0)\}/a(\phi)\). Taking expectation and using \(E_{\theta_0}[Y]=b'(\theta_0)\) gives the displayed expression. By definition, the expected log density ratio \(E_{\theta_0}[\log\{f(Y\mid\theta_0,\phi)/f(Y\mid\theta,\phi)\}]\) is the Kullback–Leibler divergence. Convexity of \(b\) gives \(b(\theta)\ge b(\theta_0)+b'(\theta_0)(\theta-\theta_0)\), proving nonnegativity. \(\square\)

For a random fitted parameter, predictive log-loss risk averages this divergence over training samples. If \(b\) has a continuous third derivative near \(\theta_0\), Taylor expansion gives excess risk \(\frac{b''(\theta_0)}{2a(\phi)}(\theta-\theta_0)^2+O(|\theta-\theta_0|^3)\). Thus, for estimates close enough to the truth and with controlled remainder, the leading term again rewards a trade-off: squared bias in the prediction-relevant parameter versus its variance. It does not say that all penalties reduce risk, nor that minimizing coefficient error and minimizing log loss are identical objectives.

What the Trade-Off Does—and Does Not—Tell Us

In applications, prediction risk is usually averaged over a distribution of future covariates as well as repeated training samples. A penalty can help at some covariate values and hurt at others; the overall answer depends on how those values are weighted and on the chosen loss. Cross-validation, covered in the earlier material on penalty selection, estimates predictive performance under a validation design. It is useful precisely because the unknown risk cannot usually be computed from the fitted coefficients alone.

• Bias is not automatically a defect for prediction. A biased prediction can have lower risk if its variance falls enough.
• Coefficient bias is not the prediction target. In a GLM, a coefficient change affects predictions through covariates and the inverse link; some directions matter more for the prediction region than others.
• No penalty wins universally. A strong penalty can overshrink important signal and increase prediction error. The loss, data-generating process, and target covariate distribution all matter.

A common mistake is to conclude, “The penalized coefficients are biased, so their predictions must be worse.” That confuses a parameter property with an expected prediction loss. The correction is to state the target and loss, then compare the bias and variance of the relevant predictions—or compare expected out-of-sample loss directly. The reverse mistake is also possible: observing that shrinkage often stabilizes predictions does not establish that any particular penalty improves risk.