Post-Lasso Refitting and the Cost of Removing Shrinkage

Why Refit After Lasso?

The lasso can serve two purposes: it predicts a response and it selects predictors by setting some estimated coefficients exactly to zero. Its penalty also shrinks the coefficients that remain. A post-lasso refit keeps the selected predictors but estimates their coefficients again by ordinary least squares, without the lasso penalty. This can reduce shrinkage bias when the selected set is appropriate, but it removes a source of regularization. In a high-dimensional genomic setting, the resulting fit can be more sensitive to noise and less accurate on new samples.

The comparison concerns two different fitted models, not merely two ways to report the same coefficients. The lasso chooses a sparse, penalized fit using the full predictor set; the refit then treats that selected set as the model to estimate. Whether the second step improves prediction is an empirical question, not a guaranteed consequence of reducing coefficient shrinkage.

The selected set depends on the data and on the chosen penalty \(\lambda\). For a clean comparison, first suppose that \(S\) has already been chosen. The refit minimizes training residual sum of squares among all coefficient vectors using those predictors. Since the lasso coefficients restricted to \(S\) are one such vector, the refit cannot have a larger training residual sum of squares. That fact alone says nothing about prediction on new observations.

What Least Squares Requires

The refit has a unique coefficient vector exactly when \(X_S\) has full column rank. In that case, $$ \hat b^{\,OLS}=(X_S^\mathsf{T}X_S)^{-1}X_S^\mathsf{T}y. $$ Full column rank implies \(|S|\leq n\), but it does not require \(|S|<n\). If \(|S|=n\) and \(X_S\) is invertible, the coefficient solution is unique, although the fit has zero residual degrees of freedom. Positive residual degrees of freedom require \(n-\operatorname{rank}(X_S)>0\). If columns are linearly dependent, the minimizing fitted values are unique, but the coefficient vector need not be unique; software may return one solution under an additional convention or use a generalized inverse.

These conditions matter especially when \(p\) is much larger than \(n\). A lasso fit can select more than \(n\) predictors in some settings, and selected columns can also be nearly or exactly collinear. In either case, the ordinary inverse formula is unavailable. Even when the selected design has full rank, near-collinearity makes the inverse Gram matrix unstable: small changes in the data can produce large changes in the refitted coefficients.

Why Variance Can Increase

A standard calculation makes the variance cost visible. Suppose the selected set \(S\) is fixed independently of the response, the linear model \(y=X_S\beta_S+\varepsilon\) is correct, \(X_S\) has full column rank, and the errors have mean zero and covariance \(\sigma^2 I\). Then $$ \hat b^{\,OLS}=\beta_S+(X_S^\mathsf{T}X_S)^{-1}X_S^\mathsf{T}\varepsilon, \qquad \operatorname{Cov}(\hat b^{\,OLS})=\sigma^2(X_S^\mathsf{T}X_S)^{-1}. $$ For a new predictor vector \(x_0\), the variance of the estimated conditional mean \(x_0^\mathsf{T}\hat b^{\,OLS}\) is therefore $$ \operatorname{Var}(x_0^\mathsf{T}\hat b^{\,OLS}) =\sigma^2x_0^\mathsf{T}(X_S^\mathsf{T}X_S)^{-1}x_0. $$ The expression grows when the model includes directions that are poorly estimated, such as nearly collinear predictors, or when more noisy predictors contribute to the prediction. Predicting a future response adds its irreducible error variance \(\sigma^2\) to the mean-prediction error under these assumptions.

The assumptions in this calculation are important. In actual post-lasso fitting, \(S\) was selected using \(y\), so it is not fixed independently of the errors. The displayed covariance is a useful description of the variability of a fixed-model refit, but it does not by itself give the full sampling distribution after selection. Selection can add variability, and estimates conditional on having been selected need not behave like estimates from a model specified in advance.

There is another useful nuance: on a region of the data where a lasso fit has the same active set and coefficient signs, its active coefficients can be written as least squares minus a penalty-dependent offset. With the objective used here, for fixed active set and signs \(s\), this expression is $$ \hat\beta^{\,L}_S=\hat b^{\,OLS}-n\lambda(X_S^\mathsf{T}X_S)^{-1}s. $$ The offset is constant within that region, so the local covariance there is not automatically smaller than the least-squares covariance. The benefit of shrinkage is not a universal variance reduction under every conditioning scheme. Rather, regularization constrains the fit across data sets and can reduce overall prediction error by trading some bias for less sensitivity. Removing that constraint can increase variance in practice, but does not have to do so in every data set.

Assessing the Trade-Off

A post-lasso refit can help when the selected set contains the important predictors, the lasso has materially shrunk their coefficients, and the selected design is well-conditioned. It can hurt when the selected set includes noise variables, when many predictors are weakly supported, or when collinearity makes the refitted coefficients unstable. The relevant target is prediction on new data, so compare lasso and post-lasso predictions using the same held-out observations or a properly designed validation procedure. Do not choose the refit because it has lower training error: that decrease is built into the optimization.

Useful diagnostics include the number of selected columns relative to \(n\), the rank and condition of \(X_S\), the size and variability of refitted coefficients, and validation error across data splits. Large coefficient changes after refitting, unstable signs, or a rise in held-out error are warnings that the removed shrinkage was serving a useful role. A selected model with \(|S|=n\) and full rank can interpolate the training response, but has no residual degrees of freedom; a larger selected set cannot have full column rank. Near rank deficiency may be problematic even before exact non-uniqueness occurs.

The main alternatives answer different questions. The original lasso retains shrinkage and selection in one fit. Ridge regression shrinks coefficients without setting them exactly to zero, while elastic net combines an \(L_1\) and an \(L_2\) penalty and can retain groups of correlated predictors. A post-lasso refit removes shrinkage only after selection; it does not reselect predictors or automatically correct for selection-induced uncertainty. Comparing these approaches on out-of-sample predictions is more informative than comparing their training residual sums of squares.

Common Mistake

Wrong reasoning: “The post-lasso model has lower training error, so it must predict better.” Least squares is optimized to lower residual sum of squares on the selected training data; this is not evidence that its predictions generalize. Correction: assess prediction on data not used to fit or select the model, and account for the instability that can arise from both selection and refitting. A second mistake is to require \(|S|<n\) for unique least-squares coefficients. Full column rank is the condition for a unique coefficient solution and implies \(|S|\leq n\); strictly fewer independent columns than observations is needed for positive residual degrees of freedom.