Why Post-Lasso OLS Standard Errors Fail

Why Refitting Does Not Erase Selection

Lasso can be useful for prediction and for reducing a large set of candidate regressors to a smaller set. A common next step is post-lasso OLS: retain the variables selected by lasso, then estimate their coefficients by ordinary least squares without a penalty. The refit removes lasso’s direct shrinkage from the retained coefficients, but it does not make the selected model behave as if it had been specified before seeing the data.

That distinction matters for inference. The usual OLS standard error describes sampling variation under a specified regression model. When the same observations determine which regressors enter the model and estimate their coefficients, the coefficient estimates are also conditioned, implicitly or explicitly, on having passed a selection rule. That rule can change their distribution. A standard error calculated as if the selected model were fixed does not generally account for this change.

Definition: Let \(\widehat S\) be the set of regressors selected by lasso using the observed data. Post-lasso OLS is the unpenalized least-squares refit using only the columns in \(\widehat S\). Because \(\widehat S\) is computed from the data, it is a random set across repeated samples. A post-lasso OLS standard error is the usual OLS standard error computed in that refitted model, without an adjustment for the data-dependent selection step.

What the Usual OLS Standard Error Assumes

For a fixed set of regressors \(S\), write the regression model as \(y=X_S\beta_S+\varepsilon\), where \(X_S\) is the design matrix containing those regressors. Suppose \(X_S\) has full column rank and the errors have conditional mean zero and variance \(\sigma^2 I\). Then the least-squares estimate is

\widehat\beta_S=(X_S^\mathsf{T}X_S)^{-1}X_S^\mathsf{T}y

and, conditional on this fixed design and model,

\operatorname{Var}(\widehat\beta_S\mid X_S)=\sigma^2(X_S^\mathsf{T}X_S)^{-1}.

With Gaussian errors, the coefficient estimate has a normal sampling distribution under this fixed-model setup. Replacing \(\sigma\) by an estimate leads to the familiar OLS standard errors and, under the usual conditions, t-based intervals. These results are not a general guarantee for any regression that happens to be fit by least squares: they describe inference for a model treated as fixed, with the relevant assumptions satisfied.

After lasso selection, the reported coefficient is based not only on the least-squares calculation but also on the event that its variable was selected. If the variable was selected because its estimated coefficient looked sufficiently large, the estimates that survive are not a representative, unfiltered sample of all estimates. A conventional OLS standard error uses the geometry of the refitted design, but does not by itself account for this selection event.

Worked Example: A Standard Error When the Model Is Fixed

Suppose a hypothetical study has a regressor specified before examining the outcome. Its column \(x\) satisfies \(x^\mathsf{T}x=80\), and the model is \(y=\beta x+\varepsilon\), with independent errors distributed as \(N(0,2^2)\). The model is correctly specified, and the regressor is not selected using the outcome. Find the standard error for the OLS coefficient, and illustrate a 95% interval if the estimated coefficient is \(0.60\).

For this one-regressor model, the variance is \(\sigma^2/(x^\mathsf{T}x)\). Thus

SE(\widehat\beta)=\frac{2}{\sqrt{80}}=0.2236.

Using the normal critical value \(1.96\), the interval is

0.60\pm1.96(0.2236) =0.60\pm0.4383 =(0.1617,\ 1.0383).

Under the stated fixed-model assumptions, this is an ordinary model-based 95% confidence interval for the coefficient, in the units of \(y\) per unit of \(x\). Its validity in this example relies on the regressor having been specified in advance; the calculation does not establish that the same interval procedure has 95% coverage after selecting the regressor from the outcome data.

A Finite-Sample Selection Example

The effect of selection can be seen exactly in a small model. Consider a Gaussian linear regression with orthogonal, standardized candidate regressors: for each column \(x_j\), \(x_j^\mathsf{T}x_j=n\), and for distinct columns \(x_j^\mathsf{T}x_k=0\). Let the errors be independent \(N(0,\sigma^2)\), and suppose a candidate coefficient is truly zero. In this design, its unpenalized estimate has the distribution

\widehat\beta^{\,OLS}_j=\beta_j+\frac{x_j^\mathsf{T}\varepsilon}{n}, \qquad \widehat\beta^{\,OLS}_j\sim N\left(\beta_j,\frac{\sigma^2}{n}\right).

For the lasso objective \(\|y-X\beta\|^2/(2n)+\lambda\|\beta\|_1\), orthogonality gives a coordinatewise soft-thresholding rule: a coefficient is selected precisely when \(|\widehat\beta^{\,OLS}_j|>\lambda\). In other words, here the selection event is a threshold on the same noisy estimate that will be reported after the OLS refit.

Worked Example: Selection Bias and Conditional Coverage

Suppose the standard error of the unpenalized estimate is \(0.20\), the true coefficient is zero, and the lasso threshold is \(\lambda=0.20\), or one standard error. Errors are Gaussian and the design is orthogonal as described above. For clarity, take the error variance as known when forming the 95% interval. Calculate the probability that the variable is selected, the average magnitude of its estimate among selected fits, and the conditional coverage of the conventional 95% interval.

Selection probability. Under the null, write \(\widehat\beta^{\,OLS}_j=0.20Z\), where \(Z\sim N(0,1)\). The variable is selected if \(|Z|>1\). Therefore,

P(\text{selected})=P(|Z|>1)=2[1-\Phi(1)]\approx0.3173.

Magnitude bias. Among selected estimates, the expected signed estimate is zero by symmetry: positive and negative selected estimates balance. But the expected absolute estimate is positive and inflated relative to the true magnitude, which is zero. For a standard normal variable,

E(|Z|\mid |Z|>1) =\frac{2\int_1^\infty z\phi(z)\,dz}{2[1-\Phi(1)]} =\frac{\phi(1)}{1-\Phi(1)} \approx1.525.

The equality uses \(\int_1^\infty z\phi(z)\,dz=\phi(1)\). Hence

E(|\widehat\beta^{\,OLS}_j|\mid \text{selected}) =0.20(1.525)\approx0.3050.

A variable with a true coefficient of zero therefore has an average selected absolute estimate of about \(0.3050\) coefficient units in this example. This is magnitude selection bias; it does not imply a nonzero signed average in this symmetric null case.

Conditional coverage. The conventional 95% interval is \(0.20Z\pm1.96(0.20)\). It covers the true coefficient zero if \(|Z|\leq1.96\). But it is reported for this variable only when \(|Z|>1\). Its coverage among selected fits is therefore

P(|Z|\leq1.96\mid |Z|>1) =\frac{P(1<|Z|\leq1.96)}{P(|Z|>1)} =\frac{2[\Phi(1.96)-\Phi(1)]}{2[1-\Phi(1)]}.

Using the normal CDF without first rounding its values, the numerator is approximately \(0.2673\), the denominator is approximately \(0.3173\), and their ratio is approximately \(0.8424\). Thus the interval covers the zero coefficient in about 84.24% of the selected fits, not 95%. The usual interval calculation did not account for the fact that inclusion required the estimate to exceed one standard error in absolute value.

Selection Can Also Exaggerate a Nonzero Coefficient

The null example makes the truncation visible, but selection can also pull a nonzero coefficient estimate away from zero on average among selected fits. The size and direction of this effect depend on the true coefficient and the selection rule. The following calculation uses the same one-coordinate Gaussian threshold model; it is an illustration of conditional bias, not a claim that every selected coefficient is biased in the same direction by the same amount.

Worked Example: A Positive Coefficient Selected by a Threshold

Suppose a coefficient has true value \(0.10\) and standard error \(0.20\), with a Gaussian unpenalized estimate. The lasso threshold is again one standard error, so the variable is selected when the estimate’s absolute value exceeds \(0.20\). Find the expected estimate among selected fits.

In units of the standard error, the estimate is \(W=0.5+Z\), where \(Z\sim N(0,1)\), and selection is the event \(W>1\) or \(W<-1\). Equivalently, \(Z>0.5\) or \(Z<-1.5\). The selection probability is

P(\text{selected}) =1-\Phi(0.5)+\Phi(-1.5) \approx0.3753.

Using \(E[Z\,1(Z>a)]=\phi(a)\) and \(E[Z\,1(Z<-a)]=-\phi(a)\), the unstandardized first moment of \(W\) over the selection event is

E[W\,1(\text{selected})] =0.5[1-\Phi(0.5)+\Phi(-1.5)]+\phi(0.5)-\phi(1.5) \approx0.4102.

Dividing this moment by the selection probability gives

E(W\mid\text{selected}) =\frac{0.4102}{0.3753}\approx1.093.

Returning to coefficient units, the expected selected estimate is \(0.20(1.093)\approx0.2186\), rather than the true coefficient \(0.10\). The conditional upward bias is about \(0.1186\) coefficient units. Selection favors estimates in the positive tail when the true coefficient is positive, while the negative selected tail is less likely. This calculation shows why removing the lasso penalty does not remove selection bias.

What the Standard Error Leaves Out

The examples rely on an especially transparent orthogonal design and a known error variance. In an applied post-lasso regression, the usual OLS standard error is often calculated with an estimated variance and a design that is not orthogonal. Those details can change the numerical behavior, but they do not fix the underlying issue: the usual calculation treats the final set of regressors as fixed and does not account for the process that chose it.

Several complications can matter in practice:

Conditioning changes the distribution. When a coefficient is selected for having a large estimated magnitude, selected estimates are disproportionately extreme. The conventional OLS interval does not adjust for this truncation.
The selected model may omit relevant regressors. If a nonzero regressor is left out, the refitted regression may not estimate the coefficient from the intended full model. The OLS standard error does not generally account for omitted-variable bias.
The selected set can be unstable. Small changes in the sample may change which variables are included. Reporting only the standard errors from the final refit hides this model uncertainty.
Conventional variance estimates are not selective corrections. Using a heteroskedasticity-robust OLS variance estimator may address a variance-assumption problem, but it does not, by itself, account for data-dependent selection.

These points do not mean post-lasso OLS is never useful. It can provide an unpenalized refit for prediction or descriptive summaries, and there are settings in which its inference can be justified under additional assumptions or a suitable inferential design. The narrower conclusion is that the ordinary OLS standard error, calculated as though the selected regressors had been fixed in advance, does not generally warrant its usual nominal interpretation after same-sample lasso selection.

Common Mistake

Wrong reasoning: “Lasso selected the variables, but I then refit by OLS. The refit is ordinary least squares, so its usual standard errors and 95% confidence intervals are valid.”

Correction: The OLS formula is ordinary, but the input model was chosen using the same data. The coefficient’s distribution among fits that selected it can differ from its fixed-model distribution. The refit removes penalization from retained coefficients; it does not undo the selection event, account for omitted variables, or include uncertainty about which model was chosen. A nominal confidence level from a fixed-model formula is not automatically a nominal post-selection coverage rate.

Key takeaway: Post-lasso OLS standard errors quantify uncertainty as if the selected model were fixed. When lasso selects variables using the same observations, this assumption generally fails: estimates may be biased conditional on selection, and conventional intervals need not attain their stated coverage.

Check Your Understanding

Answer each question using the assumptions and distinctions developed above.

In the orthogonal Gaussian example, why does the lasso selection event depend on the same random quantity used as the post-lasso OLS estimate?
Under a zero coefficient and a symmetric two-sided threshold, why can the selected signed estimates average to zero even though their average absolute value is positive?
In the one-standard-error threshold example, what conditional event must hold for the conventional 95% interval to cover the true zero coefficient?
Why does refitting by OLS remove lasso shrinkage without removing selection bias?
Would using a heteroskedasticity-robust OLS standard error, by itself, account for the fact that lasso selected the regressors from the same data? Explain.