Prediction Accuracy Versus Coefficient Recovery

Two Different Goals for a Regression Model

A regression model can be useful for forecasting an outcome without giving accurate estimates of the underlying effects of each predictor. This distinction matters in ridge and lasso regression because both methods deliberately constrain coefficient estimates: ridge shrinks them toward zero, while lasso can shrink some to exactly zero. Those changes can improve predictions, but they also mean that the fitted coefficients need not equal the coefficients that generated the data.

The key question is therefore not simply whether a model is “good.” It is: good for what purpose? If the purpose is to predict outcomes for new cases, performance should be assessed on new cases. If the purpose is to recover scientifically meaningful effects, coefficient accuracy and the scientific design matter. A small prediction error alone does not establish that the fitted coefficients recover those effects.

The Model and the Two Error Measures

For a fixed design matrix \(X\), consider the linear model \(y=X\beta+\varepsilon\). Here \(y\) is the vector of observed responses, \(X\) contains the predictor values, and \(\beta\) is the coefficient vector. Assume the errors have mean zero, are independent, and have common variance \(\sigma^2\). When discussing a new response at predictor vector \(x_0\), assume it follows the same linear mean model, \(Y_0=x_0^\mathsf{T}\beta+\varepsilon_0\), with an independent error of mean zero and variance \(\sigma^2\). The intercept, if included, is not penalized by ridge or lasso.

A coefficient-recovery measure is the squared Euclidean error \(\|\hat{\beta}-\beta\|^2\). Its size depends on the scale of the predictors, so coefficients should be compared using a stated, consistent scaling. For prediction, the error in the fitted conditional mean at \(x_0\) is \(x_0^\mathsf{T}(\hat{\beta}-\beta)\). The new response also contains the irreducible error \(\varepsilon_0\).

When new predictor vectors have mean zero and covariance matrix \(\Sigma\), the expected squared difference between the fitted conditional mean and the true conditional mean is

This quantity weights coefficient errors according to how the predictors vary together. A large coefficient error in a direction where the predictors have little variation may have a small effect on predictions. Conversely, errors in a direction with substantial predictor variation can matter greatly. For a new response, the expected squared prediction error conditional on the fitted coefficients is this quantity plus \(\sigma^2\), provided the new error is independent of the predictors and fitted model.

Why Shrinkage Can Help Prediction

Ridge regression chooses coefficients by minimizing the residual sum of squares plus a penalty on squared coefficients. With tuning parameter \(\lambda>0\), the objective for penalized slopes is \(\|y-Xb\|^2+\lambda\|b\|^2\). Lasso instead adds a penalty on the sum of absolute coefficient values. In both cases, tuning controls the strength of the constraint. These definitions describe the criteria; selecting a tuning value for prediction requires evaluating candidate values using a suitable validation procedure, not judging the training fit alone.

A short calculation shows how ridge can reduce prediction error even though it introduces bias. For one predictor \(x\), the ridge estimate minimizes \(\|y-xb\|^2+\lambda b^2\). Differentiating with respect to \(b\) and setting the derivative to zero gives \(-2x^\mathsf{T}y+2x^\mathsf{T}xb+2\lambda b=0\). Thus

Under the stated linear model, \(E(\hat{\beta}_{\lambda})=\frac{x^\mathsf{T}x}{x^\mathsf{T}x+\lambda}\beta\): the estimate is pulled toward zero, so it is biased when \(\beta\ne 0\). Its variance is \(\frac{\sigma^2 x^\mathsf{T}x}{(x^\mathsf{T}x+\lambda)^2}\), which is smaller than the OLS variance \(\frac{\sigma^2}{x^\mathsf{T}x}\) for \(\lambda>0\). Whether the reduction in variance outweighs the added squared bias depends on the signal, noise, design, and prediction target. Bias by itself does not determine prediction quality.

Correlated Predictors: Similar Predictions, Different Coefficients

With correlated predictors, substantially different coefficient vectors can produce similar predictions. If two predictors tend to move together, changes to their coefficients in opposite directions may have little effect on the fitted values. Consequently, a coefficient vector that is close to the truth in Euclidean distance is not the only route to accurate prediction; nor does accurate prediction ensure that each coefficient has been recovered.

Lasso and the Limits of Variable Selection

Lasso can set some fitted coefficients to zero, which is useful when a sparse predictive model is desired. But a selected predictor is not automatically a uniquely identified scientific cause or effect, and a zero coefficient does not prove that the predictor has no relationship with the outcome. When predictors are correlated, different choices among them can represent much the same predictive information. The lasso selection behavior covered in the earlier material on lasso selection among correlated predictors is therefore relevant to interpretation as well as to fitting.

Evaluate the Goal You Actually Have

A model’s training residuals measure fit to the data used to estimate it; they do not by themselves measure accuracy for new cases. For a prediction goal, assess performance on data not used to fit or tune the model, using a loss aligned with the intended outcome and prediction task. The earlier material on cross-validation for penalized regression and leakage-free tuning of penalized models provides relevant procedures. A validation score estimates predictive performance for a target setting; it does not measure whether the fitted coefficients equal the scientific effects.

For coefficient recovery, first define what the scientific coefficients mean and how predictors are scaled. Then consider whether the data and design can distinguish their contributions. Correlation can make the prediction-relevant combination of coefficients better determined than the individual coefficients. Penalization can stabilize predictions by favoring particular coefficient patterns, but that stability does not, on its own, establish that the favored pattern is scientifically correct. Ordinary OLS inference also cannot simply be transferred to penalized estimates, as discussed in the earlier material on why penalized estimates lack ordinary OLS inference.

Common Mistake: Treating Prediction as Proof of Effects

• Wrong reasoning: “The lasso model predicts accurately and assigns a zero to this predictor, so the predictor has no effect.”
• Correction: Predictive accuracy assesses how well the model forecasts outcomes under a particular predictor distribution and loss. A zero penalized coefficient, especially with correlated predictors, does not establish that the underlying scientific effect is zero.
• Wrong reasoning: “Ridge coefficients are biased, so ridge must predict poorly.”
• Correction: Bias is one component of estimation error. Shrinkage can reduce variance enough to improve prediction, as the worked example illustrates. Performance must be evaluated for the intended prediction target.

The distinction is practical, not merely technical. If the goal is prediction, choose and assess a model for its performance on appropriate new cases. If the goal is scientific explanation or effect estimation, predictive performance is not a substitute for a design and analysis that support that interpretation. A model can be useful for one goal and inadequate for the other.