Confidence Intervals After Model Selection

Why Selection Changes the Meaning of an Interval

A regression confidence interval is usually introduced for a model that has already been specified. Variable-selection procedures change that setting: the data help decide which model will be reported. If the same outcomes are used both to search and to estimate the selected model, the reported coefficient is not an ordinary estimate from a model chosen in advance. The search tends to favor estimates that look unusually large relative to their standard errors.

That tendency affects more than the point estimate. A conventional standard error, confidence interval, or p-value calculated after selection generally acts as though the reported model had been fixed before observing the data. It does not account for the fact that other candidate models were considered and not reported. The resulting intervals can have less than their advertised repeated-sampling coverage, and p-values can look more impressive than the search warrants.

What the Usual Regression Interval Assumes

For a model specified in advance, write the linear regression model as \(Y=X\beta+\varepsilon\). Here \(Y\) is the response vector, \(X\) is the fixed design matrix, and \(\beta\) is the vector of regression coefficients. For the standard finite-sample \(t\) interval, assume that \(X\) has full column rank and that the errors are independent with a common variance and a normal distribution: \(\varepsilon\sim N(0,\sigma^2 I)\). The model also assumes that the conditional mean of \(Y\) is \(X\beta\).

If the model has \(q\) coefficients, including any intercept, its residual degrees of freedom are \(df=n-q\). For coefficient \(\beta_j\), the usual interval is

Under the stated assumptions and with the model fixed in advance, this procedure has repeated-sampling coverage \(1-\alpha\): over repeated samples, the interval contains the fixed coefficient \(\beta_j\) in a proportion \(1-\alpha\) of samples. The same fixed-model framework gives a \(t\) statistic for testing a coefficient. The usual interpretation of its p-value also presumes that the model and test were specified independently of the outcome data used for the test.

After a search, the selected model is itself a function of the observed outcomes. The selected coefficient and its standard error are therefore being examined in a different sampling process. The usual formula does not include the extra variation and selection induced by that process. This does not mean every post-selection interval must miss its target, nor that every search has the same effect. It means the fixed-model guarantee no longer follows just by applying the familiar formula to the selected model.

An Apparently Strong Result From a Search

The same selection effect can be seen in a p-value. If a single test had been specified in advance, an observed statistic far from zero might provide evidence against its null hypothesis. But if many candidate tests were examined and only the most extreme result was reported, the relevant question also includes how often a search would find a result at least that extreme when all the null hypotheses are true.

How the Amount of Searching Matters

The previous calculation has a simple general form in the special case of \(k\) independent candidate statistics, all standard normal under their null hypotheses. If the selected statistic is the largest in absolute value, then a conventional 95% interval covers its zero target only if none of the \(k\) statistics exceeds 1.96 in absolute value. The coverage in this example is approximately \(0.95^k\). As the number of opportunities to select an unusually large result increases, the chance that all statistics remain within the ordinary interval’s limits decreases.

What the Examples Do—and Do Not—Establish

The examples use an especially simple model so that the selection effect can be calculated exactly. Real subset and stepwise searches may compare correlated predictors, use entry or removal rules, or select model size using a criterion other than the largest individual statistic. In those settings, the selected estimates are generally dependent, and the simple powers and family-level probabilities above need not apply. The central point remains: ordinary output from the final model does not automatically account for all the candidate models considered along the way.

There is an important contrast with a model specified before examining the outcomes. If the model truly was fixed in advance and the regression assumptions hold, ordinary coefficient intervals and tests retain their familiar fixed-model interpretation. Likewise, a selection decision made using data that are independent of the data used for inference does not create this particular reuse-of-the-same-outcomes problem. But when selection and estimation use the same outcomes, the usual standard error does not by itself measure uncertainty from the entire search.

Common Mistake: Treating the Final Model as if It Was Chosen in Advance

A common mistake is to say, “The final model has a 95% interval, so it has 95% coverage,” simply because a standard regression program printed that interval. That reasoning ignores how the model was obtained. If the data were searched to find a promising model or coefficient, the ordinary interval’s fixed-model coverage result does not automatically carry over to the complete procedure.

The correction is to describe the inference honestly: the reported interval is an ordinary fixed-model interval calculated after selection, and its nominal level does not by itself guarantee that level of repeated-sampling coverage. The p-value likewise should not be read as though only one prespecified test had been carried out. Exact post-selection procedures exist for particular selection rules and assumptions, but their validity depends on accounting for the actual selection mechanism; changing only the wording on a conventional interval does not do this.