Why Penalized Estimates Lack Ordinary OLS Inference

Why an accurate-looking coefficient may not support an ordinary interval

A coefficient estimate is not an inferential procedure by itself. An ordinary least-squares (OLS) standard error describes the sampling variability of an OLS estimate under a specified model. Ridge and lasso estimates are constructed differently: ridge deliberately shrinks coefficients, while lasso can both shrink coefficients and set some to zero. Their behavior across repeated samples therefore does not automatically match the behavior assumed by ordinary OLS standard errors, p-values, or confidence intervals.

This matters because a routine interval or test can look familiar even when its assumptions do not fit the calculation that produced the coefficient. A small p-value from an OLS formula does not, by itself, account for having chosen predictors after inspecting the same data. And an interval centered on a shrunken estimate is not automatically an interval with its advertised coverage for the original regression coefficient.

The OLS inference being used as a reference

Consider the linear model \(Y=X\beta+\varepsilon\), where \(Y\) is the response vector, \(X\) is a fixed design matrix, and \(\beta\) is the coefficient vector. For the standard OLS formulas, assume the conditional mean is correctly specified, the columns of \(X\) are linearly independent, and the errors are independent with mean zero and common variance \(\sigma^2\). Normal errors give exact small-sample t procedures; without normality, the usual procedures may instead rely on large-sample approximations.

When the model contains an intercept and \(p\) predictors, a full-rank OLS fit has \(n-p-1\) residual degrees of freedom. The estimated residual variance is \(s^2=\mathrm{RSS}/(n-p-1)\), where \(\mathrm{RSS}\) is the residual sum of squares. The OLS standard error for a coefficient comes from this variance estimate and the design matrix. Its familiar t test and confidence interval are calibrated for the OLS estimator in this model—not for every coefficient-like quantity a regression procedure might return.

Definition: Ordinary OLS inference means using OLS standard errors, p-values, or confidence intervals derived for the OLS estimator under a specified linear model and its assumptions. Applying those formulas to a different estimator, or to a model selected using the same data, requires additional justification.

Ridge: variance changes, and shrinkage introduces bias

For clarity, suppose the response and predictors have been centered, the intercept is not penalized, and ridge minimizes half the residual sum of squares plus a quadratic penalty:

\hat{\beta}_{\mathrm{ridge}} = \arg\min_b \left\{ \frac{1}{2}(Y-Xb)^\mathsf{T}(Y-Xb) + \frac{\lambda}{2}b^\mathsf{T}b \right\}, \qquad \lambda>0.

The ridge estimate is \((X^\mathsf{T}X+\lambda I)^{-1}X^\mathsf{T}Y\). Under the model above, treating \(X\) and \(\lambda\) as fixed, its expectation and conditional variance are

E(\hat{\beta}_{\mathrm{ridge}}\mid X) = (X^\mathsf{T}X+\lambda I)^{-1}X^\mathsf{T}X\beta, $$ $$ \operatorname{Var}(\hat{\beta}_{\mathrm{ridge}}\mid X) = \sigma^2 (X^\mathsf{T}X+\lambda I)^{-1} X^\mathsf{T}X (X^\mathsf{T}X+\lambda I)^{-1}.

The expectation is generally not \(\beta\), so the estimator has shrinkage bias. Its variance also differs from the OLS variance. Thus the OLS standard error formula is not the standard error formula for a ridge estimate. More importantly, even a correctly calculated standard deviation for ridge describes variation around the ridge estimator’s expectation; it does not make that expectation equal to the original coefficient.

Worked Example: Ridge shrinkage is not an OLS interval

Suppose one centered, standardized predictor has \(X^\mathsf{T}X=100\), the error standard deviation is known to be \(4\) response units, and \(\lambda=100\). In a hypothetical sample, the OLS slope is \(2.4\) response units per predictor unit. Find the ridge estimate and compare the sampling standard deviations.

For one predictor, the ridge estimate is the OLS estimate multiplied by \(X^\mathsf{T}X/(X^\mathsf{T}X+\lambda)\). Therefore

\hat{\beta}_{\mathrm{ridge}} = \frac{100}{100+100}(2.4) = 1.2 \quad\text{response units per predictor unit.}

The OLS standard deviation in this one-predictor model is \(4/\sqrt{100}=0.4\). For ridge, the variance is \(\sigma^2(X^\mathsf{T}X)/(X^\mathsf{T}X+\lambda)^2\), giving

\operatorname{SD}(\hat{\beta}_{\mathrm{ridge}}\mid X) = 4\frac{\sqrt{100}}{100+100} = 0.2 \quad\text{response units per predictor unit.}

The ridge estimate has half the OLS standard deviation here, but that is not the whole comparison. If the true slope were \(2\), the expected ridge estimate would be \(100/(100+100)\times2=1\), a bias of \(-1\) response unit per predictor unit. An ordinary OLS interval centered at \(1.2\) with an OLS standard error of \(0.4\) would use the wrong estimator’s standard error and would not account for ridge bias. A smaller spread around a biased target does not, on its own, provide a valid interval for the original slope.

Lasso: the data help decide which coefficients are reported

Lasso adds an absolute-value penalty, which can produce exact zeros. In an orthogonal one-predictor setting with \(X^\mathsf{T}X=1\), let \(z=X^\mathsf{T}Y\) be the OLS estimate and use the objective \(\frac12(Y-Xb)^\mathsf{T}(Y-Xb)+\lambda|b|\). Minimizing this objective gives the soft-threshold estimate

\hat{\beta}_{\mathrm{lasso}} = \begin{cases} z-\lambda, & z>\lambda,\\ 0, & |z|\leq\lambda,\\ z+\lambda, & z<-\lambda. \end{cases}

A coefficient is selected precisely when \(|z|>\lambda\). The same random data therefore influence both whether the coefficient is reported as nonzero and what its value is. If a coefficient is reported because its noisy OLS estimate crossed a threshold, the selected estimate does not behave like an estimate from a model fixed independently of those data. Routine OLS p-values and intervals do not account for that selection event.

Worked Example: A lasso selection event changes a nominal p-value

Consider a hypothetical orthogonal coordinate under the null hypothesis that its true coefficient is zero. Suppose the OLS estimate \(Z\) has standard error \(1\), so \(Z\sim N(0,1)\), and lasso uses \(\lambda=2\). The observed OLS estimate is \(z=2.3\). Find the lasso estimate and compare an ordinary two-sided OLS p-value with the null probability conditional on selection.

Because \(2.3>2\), lasso selects the predictor and reports \(2.3-2=0.3\). If one instead looks at the OLS estimate and applies an unadjusted two-sided OLS test, its p-value is

P(|Z|\geq 2.3) = 2\{1-\Phi(2.3)\} \approx 0.0214.

But the predictor entered the reported set only when \(|Z|>2\). Under the null, the probability of an estimate at least as extreme as the observed one, among estimates that passed this selection rule, is

P(|Z|\geq 2.3\mid |Z|>2) = \frac{P(|Z|\geq 2.3)}{P(|Z|>2)} \approx \frac{0.0214}{0.0455} \approx 0.471.

The unadjusted p-value answers a question that does not condition on how the predictor was selected; the conditional calculation answers a different question. This example does not supply a general lasso-adjusted p-value. It shows why a routine OLS p-value cannot simply be read as if the predictor had been specified in advance.

Why refitting OLS does not automatically fix selection

A common workflow is to use lasso to select predictors and then refit OLS using only those predictors. The refitted coefficient is an OLS estimate within the reduced model, but the reduced model was chosen using the same responses. Ordinary OLS formulas for a model specified in advance do not generally account for that data-dependent choice. In repeated samples, the selected model can change; among samples where a variable is selected, its estimated association can also be unusually strong because it helped pass the selection threshold.

The distinction is between a calculation being algebraically an OLS fit and its sampling distribution matching the assumptions behind routine OLS inference. A post-selection OLS standard error can describe variability as if the reduced model had been fixed, while omitting the additional uncertainty created by selecting it. The familiar formula is not made selection-aware merely by applying it after the selection step.

Worked Example: When an OLS interval can follow model selection

Suppose, in a hypothetical analysis, a training sample was used to choose a model with two predictors. An independent sample of \(n=40\) new observations is then used to fit that model. Assume the model is correctly specified for this new sample, the design has full rank, errors are independent with constant variance and are normally distributed, and the model choice used no information from the new responses. The OLS slope for a particular predictor is \(1.5\) score points per hour, with standard error \(0.4\) score points per hour. Construct a 95% interval.

There are \(40-2-1=37\) residual degrees of freedom, including the intercept in the fitted model. The 95% t critical value is approximately \(2.026\). Thus

1.5\pm 2.026(0.4) = 1.5\pm 0.8104 = (0.6896,\ 2.3104) \quad\text{score points per hour.}

Under the stated assumptions, this ordinary OLS interval is appropriate for the coefficient in the specified two-predictor model fitted to the independent sample. In repeated new samples, intervals formed this way would cover that model’s coefficient 95% of the time. This does not establish that the coefficient is causal, nor does it make the interval valid if the same responses were used both to select the model and to fit it.

What routine coefficient intervals can and cannot say

A routine interval has meaning only after identifying its estimator, target, and assumptions. For OLS in a fixed, correctly specified model, ordinary formulas target that model’s coefficients. Ridge returns shrunken estimates, and lasso returns estimates affected by both thresholding and selection. An OLS interval copied around either penalized estimate is not automatically a confidence interval for the corresponding OLS-model coefficient. Nor does an interval that treats the selected lasso model as fixed automatically account for the path by which it was chosen.

This is not a claim that inference after penalization is impossible. It means that ordinary OLS formulas alone do not solve the problem. Procedures designed for penalized estimation or for selection-aware inference, as well as genuinely independent data for estimation after selection, can address different inferential goals. Each approach needs its own assumptions and interpretation; one should not infer validity just from seeing a standard error or interval in software output.

Common mistake and practical check

Wrong reasoning: “The software reports a coefficient and a standard error, so the usual OLS p-value and interval apply.”
Correction: First ask whether the estimate is OLS or penalized, whether predictors were selected using the same responses, and what parameter the interval is intended to cover. Shrinkage can introduce bias, and data-dependent selection changes the sampling behavior assumed by routine OLS inference.

A useful check is to state the target coefficient and the route by which the model was obtained before interpreting any interval or p-value. If the model was selected using the same data, label routine post-selection OLS output as not automatically selection-adjusted. If the estimate is ridge or lasso, do not treat an OLS standard error as though it were derived for that estimator.

Key takeaway: OLS inference is calibrated for OLS in a specified model under stated assumptions. Ridge changes the estimator’s bias and variance; lasso additionally makes selection depend on the observed data. Ordinary OLS standard errors, p-values, and intervals do not automatically account for either change.

Check Your Understanding

Try these questions before reviewing the explanations above.

Why does a ridge estimate’s smaller sampling variance not by itself justify an ordinary confidence interval for the original regression coefficient?
In the orthogonal lasso example, what event causes the predictor to be selected?
Why does refitting OLS after lasso selection not automatically make the ordinary OLS p-values valid?
What assumptions made the independent-sample OLS interval appropriate in the third worked example?
State one key question to ask before interpreting a standard error reported for a penalized regression coefficient.