Inference After GLM Model Selection

Why Inference Changes After Selection

A GLM standard error describes uncertainty under a specified model and estimation procedure. If the model was chosen after inspecting the same responses—for example, by trying predictors, transformations, or interactions and retaining the most promising specification—then the reported coefficient and standard error come from a data-dependent procedure. The usual confidence interval typically treats the final model as if it had been fixed in advance. That is a different sampling problem.

This does not mean that every selected-model interval is useless, or that selection always makes an interval too narrow by the same amount. It means the ordinary calculation does not, by itself, account for the selection rule. The size and direction of the resulting error depend on what was searched, how models were compared, and what parameter the interval is intended to describe.

It is useful to be precise about the target. A coefficient in a particular GLM is generally a model-specific parameter: it describes the fitted linear predictor under that model and its covariate specification. If a different model is selected, the corresponding coefficient may represent a different target. Thus, even a method that correctly accounts for selection must say what parameter it is making inference about.

A Simple Derivation of Selection Bias

The winner’s curse in GLM selection has a simple analogue. Suppose two candidate procedures estimate the same scalar quantity \(\theta\), with estimates \(Z_1\) and \(Z_2\). For illustration, assume \(Z_1\) and \(Z_2\) are independent \(N(\theta,1)\) variables, and report the larger estimate. This is not a general GLM result; it isolates the effect of choosing a winner.

Derivation. Write \(Z_1=\theta+\epsilon_1\) and \(Z_2=\theta+\epsilon_2\), where the errors are independent \(N(0,1)\). The identity \(\max(a,b)=(a+b+|a-b|)/2\) gives

Because \(\epsilon_1-\epsilon_2\sim N(0,2)\), it can be written as \(\sqrt{2}U\), where \(U\sim N(0,1)\). Direct integration of the standard normal density gives \(E[|U|]=\sqrt{2/\pi}\). Therefore \(E[|\epsilon_1-\epsilon_2|]=2/\sqrt{\pi}\), proving the result. The same selection logic can affect GLM analyses when candidate models, terms, or transformations are compared and the apparently strongest result is reported.

What the Ordinary GLM Interval Conditions On

In a regular, prespecified GLM, a Wald interval for a coefficient \(\beta_j\) commonly has the form \(\hat\beta_j\pm z_{1-\alpha/2}\widehat{\operatorname{se}}(\hat\beta_j)\). Its familiar approximate coverage relies on the model and estimation assumptions, and on the usual large-sample approximation for the estimator. If the same data were also used to choose the model, the usual calculation generally acts as though that choice had not occurred.

One way to see the issue is to examine a simplified normal calculation. Let \(Z\sim N(\theta,1)\), and suppose an analyst reports \(Z\) only when \(Z>c\). Conditional on being reported, \(Z\) no longer has its original normal distribution. Its conditional density is

where \(\phi\) is the standard normal density and \(P_\theta(Z>c)=1-\Phi(c-\theta)\). The denominator is required because the density is conditional on passing the selection rule. Treating the observed \(Z\) as an unselected normal observation omits that conditioning.

This deliberately stark threshold example is not a description of every model-selection algorithm. It demonstrates why an unconditional standard error and interval cannot simply be assumed to retain their usual interpretation after selection. A real GLM search can involve a complicated selection event rather than one threshold, and the right adjustment depends on that event.

Three Strategies for Post-Selection Inference

There is no universal correction that can be applied to every selected GLM. Three broad strategies answer different practical needs.

Sample Splitting: Separate Selection from Estimation

With sample splitting, randomly divide the observations into a selection sample and an inference sample. Use the selection sample to choose a candidate GLM. Then, without revisiting that choice in light of inference-sample results, fit the chosen model to the inference sample and calculate its usual standard errors and intervals. The key protection is independence: conditional on the selection data, the inference observations have not been used to choose the model.

Justification. Let \(S\) denote the selection data, \(I\) the inference data, and \(M=g(S)\) the model chosen by a selection rule \(g\). Independence gives the joint distribution \(P(S\in A,I\in B)=P(S\in A)P(I\in B)\) for measurable sets \(A,B\). For any selection outcome with \(g(S)=m\) having positive probability, conditioning on that outcome does not change the distribution of \(I\): \(P(I\in B\mid g(S)=m)=P(I\in B)\). Thus, the inference sample is not truncated by the selection rule. Applying a fixed-model method to \(I\) still requires its own assumptions, such as correct variance specification and sufficient regularity for a Wald approximation. \(\square\)

Splitting does not make every target automatically unambiguous. It supports inference for parameters of the selected model fitted to the inference sample. If the model’s interpretation changes with the selected specification, that should be stated. Nor should the analyst repeatedly inspect the inference results, revise the model, and then continue to treat the same data as untouched.

Selective Adjustments and Their Requirements

Selective inference tries to use the data efficiently while explicitly accounting for the selection event. In the simple normal example, selection restricted the possible observations to \(Z>c\), so conditioning led to a truncated normal density. For GLMs, the relevant event may encode which terms survived a search, which model scored best, or which path a stepwise algorithm followed. An adjustment is credible only when that event is represented accurately and the assumed sampling distribution is adequate for the problem.

In many GLM applications, especially with complex searches or small samples, deriving a reliable selection-adjusted interval can be difficult. An adjustment designed for one selection rule should not be presented as if it automatically covers a different rule. Likewise, a standard error recalculated after selection is not necessarily a selective standard error: the adjustment concerns the distribution induced by the whole selection-and-estimation procedure, not merely the final model’s variance formula.

When Formal Adjustment Is Not Available

Cautious descriptive reporting is preferable to overstating what an ordinary interval means. A report can identify the candidate models and selection criterion, describe whether transformations and interactions were searched, give the selected model and its estimates, and state clearly that conventional intervals are conditional on the final specification and are not adjusted for the search. It can also emphasize effect sizes and uncertainty rather than treating a post-selection \(p\)-value just below a threshold as decisive evidence.

Common Mistakes

• “The final model is fixed now, so its usual interval covers the whole analysis.” The final model is fixed only after a data-dependent choice. If the same observations made that choice, the usual interval generally does not account for the choice.
• “A larger standard error automatically fixes selection.” Increasing a standard error without deriving an appropriate sampling distribution does not guarantee valid coverage.
• “Sample splitting proves the model is correct.” Splitting separates model selection from inference; it does not prevent misspecification, confounding, poor measurement, or small-sample problems.
• “A selective adjustment is universal.” Adjustments depend on the selection rule, target parameter, and distributional assumptions. A method for one search cannot be assumed valid for another.