Why Penalization Changes Both Estimates and Their Interpretation
Ridge and lasso regression choose coefficients by balancing fit to the observed responses against a penalty on coefficient size. That tradeoff can improve predictions for new observations, but the resulting coefficients are not ordinary least-squares estimates. In particular, shrinkage generally introduces bias, and lasso can select variables using the same data later used to assess their coefficients. As a result, familiar least-squares standard errors and significance tests cannot simply be attached to penalized estimates.
The distinction matters in actuarial modeling. A hypothetical insurer might use claim-level predictors to estimate future claim cost. For prediction, a slightly biased model can be useful if it reduces variation enough to improve performance on new claims. For inference, however, a coefficient is often interpreted as evidence about an underlying relationship. The meaning and uncertainty of a penalized coefficient require different care.
Often the responses and predictors are centered, so the intercept can be omitted during the coefficient calculations; it is then recovered as the response mean. Predictor scaling also matters: the penalty acts on coefficient magnitudes, so variables on different scales can be penalized differently. The calculations below use the stated design-matrix summaries and penalty convention rather than assuming that all predictors have the same scale.
With centered data, ridge estimates have the closed form \(\hat{\beta}_{\lambda}=(X^\mathsf{T}X+\lambda I)^{-1}X^\mathsf{T}y\), when the inverse exists. Lasso estimates minimize the absolute-value penalized objective and can set some coefficients exactly to zero. For either method, the prediction at a new predictor vector \(x_0\) is \(\hat{y}_0=\hat{\beta}_0+x_0^\mathsf{T}\hat{\beta}_{\lambda}\). This is a predicted response, not a standard error or a confidence statement.
Worked Example: Ridge Shrinkage and a New Prediction
A hypothetical portfolio model uses two centered predictors. Suppose the design summaries are \(X^\mathsf{T}X=\begin{pmatrix}4&0\\0&9\end{pmatrix}\) and \(X^\mathsf{T}y=\begin{pmatrix}12\\18\end{pmatrix}\). Use ridge regression with \(\lambda=3\) to calculate the coefficients and predict the centered response for \(x_0=(1,2)^\mathsf{T}\). Compare the result with least squares.
Method. Apply the ridge closed form using the stated objective. Because the predictors are centered, the intercept is zero in this calculation.
Penalized coefficients. Add \(\lambda\) to each diagonal entry of \(X^\mathsf{T}X\), giving \(\begin{pmatrix}7&0\\0&12\end{pmatrix}\). Therefore,
The new prediction is \(x_0^\mathsf{T}\hat{\beta}_{\lambda}=12/7+2(3/2)=12/7+3=33/7\), which is approximately \(4.714\).
Least-squares comparison. With no penalty, the coefficients are \(12/4=3\) and \(18/9=2\). The least-squares prediction is \(1(3)+2(2)=7\). Here ridge moves both coefficients toward zero and changes the prediction from \(7\) to approximately \(4.714\). That change alone does not establish which prediction is better; predictive performance must be assessed on data not used to fit the coefficients.
Check and mistake avoided. Substitution into the ridge formula gives \((4+3)(12/7)=12\) and \((9+3)(3/2)=18\), matching the two entries of \(X^\mathsf{T}y\). A common error is to divide by the unpenalized diagonal values and report the least-squares coefficients as ridge estimates.
Why Ridge Estimates Are Biased
The shrinkage in the example is not just a numerical feature of one data set. It follows directly from the fixed-design linear model. Suppose \(y=X\beta+\varepsilon\), where \(X\) is treated as fixed, \(E[\varepsilon\mid X]=0\), and \(\operatorname{Var}(\varepsilon\mid X)=\sigma^2 I\). Assume centered predictors and an unpenalized intercept handled separately. If \(X^\mathsf{T}X+\lambda I\) is invertible, then ridge has a conditional expectation and variance that can be derived from its matrix formula.
Using \(y=X\beta+\varepsilon\), write \(\hat{\beta}_{\lambda}=(X^\mathsf{T}X+\lambda I)^{-1}X^\mathsf{T}X\beta+(X^\mathsf{T}X+\lambda I)^{-1}X^\mathsf{T}\varepsilon\).
Since \(E[\varepsilon\mid X]=0\), the noise term has expectation zero. Thus \(E[\hat{\beta}_{\lambda}\mid X]=(X^\mathsf{T}X+\lambda I)^{-1}X^\mathsf{T}X\beta\), which is generally not \(\beta\) when \(\lambda>0\).
For a fixed matrix \(A\), \(\operatorname{Var}(A\varepsilon\mid X)=A(\sigma^2 I)A^\mathsf{T}\). Set \(A=(X^\mathsf{T}X+\lambda I)^{-1}X^\mathsf{T}\) to obtain \(\operatorname{Var}(\hat{\beta}_{\lambda}\mid X)=\sigma^2(X^\mathsf{T}X+\lambda I)^{-1}X^\mathsf{T}X(X^\mathsf{T}X+\lambda I)^{-1}\).
The expectation formula displays the bias, while the variance formula describes sampling variation around that expectation. Ridge can reduce variance relative to least squares, but its coefficients are generally biased for \(\beta\). These are properties of the estimator under the specified model; they do not by themselves give a complete interval or test for a penalized coefficient.
Worked Example: Why an OLS Standard Error Does Not Transfer
In a hypothetical one-predictor model, suppose the centered design satisfies \(X^\mathsf{T}X=10\) and \(X^\mathsf{T}y=20\). Use ridge with \(\lambda=10\). Assume the fixed-design model above and that \(\sigma^2=4\) is known. Calculate the ridge estimate and its conditional mean and variance when the true coefficient is \(\beta=2\). Then show why treating the ordinary least-squares standard error as a ridge standard error is misleading.
Ridge estimate. The coefficient is \(20/(10+10)=1\). The corresponding least-squares estimate would be \(20/10=2\).
Conditional distribution properties. The ridge expectation is \(10/(10+10)\) times the true coefficient, or \((1/2)(2)=1\). Its conditional bias is therefore \(1-2=-1\). Its variance is \(4(10)/(10+10)^2=40/400=0.1\), so its standard deviation around its own expectation is \(\sqrt{0.1}\), approximately \(0.3162\).
Incorrect transfer. The ordinary least-squares standard error, using the known variance, is \(\sqrt{4/10}=\sqrt{0.4}\), approximately \(0.6325\). Dividing the ridge estimate \(1\) by this OLS standard error gives approximately \(1.581\). That calculation is not a valid ordinary least-squares test statistic for the ridge coefficient: the estimate has a different sampling variance and, more importantly, is not centered on the true coefficient.
Check and mistake avoided. The conditional mean equals the computed ridge estimate here because the supplied true coefficient is \(2\), making the shrinkage factor \(1/2\). The variance calculation uses the ridge variance formula, not the OLS formula. A smaller standard deviation around a biased expectation does not justify ignoring the bias or reusing an OLS test.
Lasso Predictions and Selection
Lasso also changes predictions by changing coefficient estimates, but its absolute-value penalty can make some estimates exactly zero. A zero coefficient means the fitted lasso model does not use that predictor at the selected penalty value. It does not establish that the predictor has no relationship with the response in the population.
For centered data, a lasso solution \(\hat{\beta}\) satisfies the following coordinate conditions under the objective defined above. For a nonzero coefficient \(j\), \(X_j^\mathsf{T}(y-X\hat{\beta})=\lambda\,\operatorname{sign}(\hat{\beta}_j)\). For a zero coefficient \(j\), \(|X_j^\mathsf{T}(y-X\hat{\beta})|\leq\lambda\). These conditions state that the residual correlation of an included variable balances the penalty, while a zero coefficient can remain zero when its residual correlation is not large enough to overcome the penalty.
Worked Example: A Lasso Prediction With One Zero Coefficient
A hypothetical two-predictor model has centered design summaries \(X^\mathsf{T}X=\begin{pmatrix}4&2\\2&4\end{pmatrix}\) and \(X^\mathsf{T}y=\begin{pmatrix}6\\3.5\end{pmatrix}\). For lasso with \(\lambda=2\), verify that \(\hat{\beta}=(1,0)^\mathsf{T}\) satisfies the coordinate conditions, then calculate the prediction at \(x_0=(1,1)^\mathsf{T}\). Compare it with the least-squares prediction.
Verify the lasso solution. At the proposed coefficients, \(X^\mathsf{T}(y-X\hat{\beta})=X^\mathsf{T}y-X^\mathsf{T}X\hat{\beta}\). The first coordinate is \(6-(4(1)+2(0))=2\), equal to \(\lambda\) for the positive coefficient. The second is \(3.5-(2(1)+4(0))=1.5\), and \(|1.5|\leq2\). The coordinate conditions are satisfied, so the proposed vector is a lasso solution.
Calculate the prediction. The lasso prediction is \(1(1)+1(0)=1\).
Least-squares comparison. The determinant of \(X^\mathsf{T}X\) is \(4(4)-2(2)=12\). Its inverse is \(\frac{1}{12}\begin{pmatrix}4&-2\\-2&4\end{pmatrix}\). Thus the least-squares coefficients are \(\frac{1}{12}(24-7,\,-12+14)^\mathsf{T}=(17/12,1/6)^\mathsf{T}\), and the prediction is \(17/12+1/6=19/12\), approximately \(1.583\).
Check and mistake avoided. The lasso solution has a zero second coefficient, and its residual correlation \(1.5\) is within the allowed interval \([-2,2]\). A common mistake is to interpret that zero as proof that the predictor is irrelevant, or to report the least-squares prediction as the lasso prediction.
Prediction Uncertainty Is Not Coefficient Inference
A fitted value answers a prediction question: what response does the fitted model assign to these predictor values? Its usefulness is judged by how well predictions perform for new observations from the intended population. In contrast, coefficient inference asks what values of an underlying parameter remain plausible after accounting for sampling uncertainty and the fitting procedure. The two questions are not interchangeable.
For ordinary least squares under a correctly specified fixed-design normal linear model with independent errors of mean zero and common variance \(\sigma^2\), the least-squares coefficient estimator is unbiased and has variance \(\sigma^2(X^\mathsf{T}X)^{-1}\), when \(X^\mathsf{T}X\) is invertible. Standard least-squares tests and intervals rely on that estimator and its sampling distribution. Ridge changes the estimator and introduces bias; lasso additionally uses the data to decide which coefficients are nonzero. Applying the OLS standard error and test formula directly to the resulting estimates ignores those changes.
This does not mean uncertainty can never be assessed after penalization. It means the uncertainty method must match the question and account for the fitting procedure. For example, assessing predictive uncertainty and making claims about selected coefficients are distinct tasks and need not use the same procedure. A prediction evaluation can focus on performance for observations not used to fit the model; a coefficient claim must address penalization bias and, for lasso, the data-dependent selection step. Neither a small fitted coefficient nor a small prediction error alone supplies an ordinary least-squares significance test.
Common Traps
- Equating smaller coefficients with worse predictions. Penalization changes fitted values, but whether the change improves prediction is an empirical question, not something established by comparing coefficients alone.
- Using OLS standard errors unchanged. Ridge and lasso estimators do not generally have the OLS sampling distribution. Ridge is biased, and lasso selection adds a further complication.
- Calling a selected lasso coefficient a population conclusion. A zero estimate at one penalty value is a property of the fitted model, not proof of no underlying association.
- Forgetting the penalty convention. The displayed ridge and lasso objectives use different penalty factors. A formula for \(\lambda\) or a coefficient cannot be transferred between conventions without checking the objective.
- Confusing a prediction with an interval. Computing \(x_0^\mathsf{T}\hat{\beta}\) gives a point prediction. It does not account for uncertainty in the fitted model or variability in a future response.
Check Your Understanding
For each problem, use the stated penalty convention and distinguish a point prediction from a claim about coefficient uncertainty.
- A centered one-predictor design has \(X^\mathsf{T}X=8\) and \(X^\mathsf{T}y=24\). Under ridge with \(\lambda=4\), find the coefficient and the prediction at \(x_0=1.5\). (Answer: \(\hat{\beta}_{\lambda}=2\); prediction \(=3\).)
- A centered one-predictor lasso problem has \(X^\mathsf{T}X=5\), \(X^\mathsf{T}y=10\), and \(\lambda=3\). Find the lasso coefficient and prediction at \(x_0=2\). (Answer: \(\hat{\beta}=1.4\); prediction \(=2.8\).)
- For a one-predictor ridge model, \(X^\mathsf{T}X=16\), \(\lambda=8\), and \(\sigma^2=9\). Give the multiplicative factor relating \(E[\hat{\beta}_{\lambda}\mid X]\) to \(\beta\), and the conditional standard deviation of \(\hat{\beta}_{\lambda}\). (Answer: factor \(=2/3\); standard deviation \(=0.5\).)
- A lasso problem has \(X^\mathsf{T}X=\begin{pmatrix}5&1\\1&2\end{pmatrix}\), \(X^\mathsf{T}y=(8,2)^\mathsf{T}\), and \(\lambda=3\). Verify the candidate \((1,0)^\mathsf{T}\) using the coordinate conditions and find its prediction at \(x_0=(2,1)^\mathsf{T}\). (Answer: residual correlations are \((3,1)^\mathsf{T}\), satisfying the conditions; prediction \(=2\).)
- In a one-predictor ridge model, \(X^\mathsf{T}X=12\), \(\lambda=6\), and \(\sigma^2=4\). Find the factor relating the ridge estimate to the least-squares estimate and the conditional variance of the ridge estimate. (Answer: factor \(=2/3\); variance \(=4/27\).)