The Post-Selection Inference Trap

Why Selection Changes Inference

A regression coefficient test answers a question about a specified model and coefficient. In practice, however, an analyst may first examine many candidate predictors and retain those that appear useful. If the same responses are then used to report ordinary p-values and confidence intervals for the retained predictors, the reported inference does not reflect the selection step.

The issue is not that a coefficient test is inherently wrong. It is that its reference distribution is derived as if the model and hypothesis had been specified independently of the observed response. Selecting a predictor because its estimated effect looks unusually large makes that selected estimate more likely to be extreme, even when all candidate effects are zero. The resulting p-value can be too small, and the usual interval can be too narrow for the question being asked.

Definition: Post-selection inference concerns inference about parameters or predictions after a model, variable, or hypothesis has been chosen using data. An ordinary regression p-value or confidence interval is not automatically valid after selection on the same response data. Its validity depends on how selection was conducted and on what adjustment, if any, is used.

For a fixed linear regression model with independent normal errors, mean zero, and constant variance, the usual coefficient \(t\)-test and confidence interval have their standard properties when the model and coefficient hypothesis are fixed independently of the response data. Those properties do not generally carry over unchanged when the analyst searches across models and reports only a selected result. The same caution applies if selection uses stepwise procedures, best-subset comparisons, or repeated informal inspection of results.

This distinction matters because model-selection criteria and coefficient inference answer different questions. A criterion may be used to choose a model for prediction or parsimony. A small p-value in that selected model does not, by itself, establish that the coefficient has been tested as though it were specified in advance. This tutorial is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.

A Simple Selection Calculation

Consider \(m\) independent candidate coefficient tests under the global null hypothesis that all \(m\) true coefficients are zero. Suppose each test uses a two-sided level \(\alpha\). For any one test, the probability of a false rejection is \(\alpha\). The chance of no false rejections across all tests is \((1-\alpha)^m\), by independence. Therefore, the probability of at least one false rejection is:

P(\text{at least one false rejection})=1-(1-\alpha)^m.

This probability is called the family-wise error rate in this setting. It is larger than \(\alpha\) when \(m>1\). The formula assumes independent tests, which is often not realistic for correlated regression predictors. It is useful to expose the selection issue, but it should not be treated as a general exact adjustment for dependent tests.

Important distinction: The family-wise error rate describes the chance of at least one false rejection over a collection of tests. A p-value calculated for one coefficient as though it had been specified in advance does not answer that family-wide question. If the coefficient was selected by searching those tests, the selection must be reflected in the inference.

Worked Example: Selecting the Largest of Five Tests

A hypothetical insurer’s analyst tests five candidate rating variables in a regression under the global null. Assume the five standardized test statistics are independent standard normal variables. Their observed values are \(0.4,-1.1,2.2,1.5,-0.7\). The analyst selects the variable with the largest absolute statistic. Calculate the selected variable’s ordinary two-sided p-value and the probability, under the global null and this selection rule, of obtaining a maximum absolute statistic at least as large as the observed one.

Method. First calculate the single-test tail probability at the observed maximum, then calculate the probability that at least one of five independent null tests reaches that magnitude. This comparison avoids treating a result chosen as the most extreme as if it had been the only test performed.

The largest absolute statistic is \(2.2\). Its ordinary two-sided p-value is:

p_{\text{ordinary}}=2P(Z\geq 2.2)=2(0.0139)=0.0278, \qquad Z\sim N(0,1).

For one test, the probability of an absolute statistic at least \(2.2\) is \(0.0278\). Under the stated independence assumption, the probability that none of the five statistics reaches that magnitude is \((1-0.0278)^5=0.9722^5\approx0.8685\). Thus the probability that at least one does is:

1-0.9722^5\approx 1-0.8685=0.1315.

The selected result has an ordinary p-value of \(0.0278\), while the probability of seeing a maximum at least this extreme in the five-test search is approximately \(0.1315\). Under this global-null and independence calculation, the ordinary p-value understates the evidence threshold implied by the selection rule. The calculation checks: the search probability is greater than the single-test tail probability, as it must be when the maximum is taken over five tests.

Common mistake avoided: Reporting \(0.0278\) as though the analyst had tested only one prespecified variable ignores the search across five candidates. The family-wide calculation is not automatically an exact selective p-value for every regression selection procedure; its interpretation here depends on the explicitly stated independent-null model and maximum-absolute-statistic rule.

Why the Selected Estimate Tends to Look Too Large

Selection can affect estimates as well as p-values. If many noisy estimates are examined and the one with the largest magnitude is retained, the selected estimate is likely to include a favorable noise fluctuation. This tendency is often called the winner’s curse. It does not mean every selected estimate is biased in the same direction under every possible selection method. It describes a general consequence of choosing a result partly because it is extreme.

A short justification follows directly from the selection rule. Under the global null, let \(Z_1,\ldots,Z_m\) be independent standard normal statistics and retain \(Z_j\) when it has the largest absolute value. For any positive threshold \(c\), the event that the retained statistic has absolute value at least \(c\) is exactly the event that at least one candidate statistic does:

\left\{\max_{1\leq j\leq m}|Z_j|\geq c\right\} = \bigcup_{j=1}^{m}\{|Z_j|\geq c\}.

Independence makes the probability of the complementary event a product:

P\left(\max_j|Z_j|\geq c\right) =1-P(|Z_1|<c,\ldots,|Z_m|<c) =1-\{P(|Z|<c)\}^{m}.

Because \(P(|Z|<c)<1\), this probability exceeds the single-test probability \(P(|Z|\geq c)\) for \(m>1\). The derivation shows why the observed extremeness of a selected result cannot be assessed as if the selection had not happened. In real regression work, candidate tests may be dependent and the selection rule may be more complicated, so this calculation illustrates the mechanism rather than supplying a universal correction.

Confidence Intervals After Selection

A conventional \(95\%\) coefficient interval is designed to cover a fixed coefficient with probability \(0.95\) over repeated samples under the specified-model assumptions. If a coefficient is chosen because it looked favorable among many candidates, that interval does not generally retain \(95\%\) coverage for the selected coefficient. The problem is the same selection step, expressed through interval coverage rather than a test’s false-rejection rate.

One simple, conservative safeguard for a family of \(m\) coefficient intervals is the Bonferroni method. To ensure simultaneous coverage of at least \(1-\alpha\) across the family, construct each two-sided interval at confidence level \(1-\alpha/m\). The Bonferroni inequality gives the guarantee without requiring independent tests. It may be conservative, and it does not model the detailed selection rule; it does provide family-wise coverage for the set of intervals under the ordinary assumptions for each individual interval.

P(\text{at least one interval misses}) \leq \sum_{j=1}^{m}P(\text{interval }j\text{ misses}) \leq m\left(\frac{\alpha}{m}\right)=\alpha.

Worked Example: Adjusting an Interval for Five Candidates

In a hypothetical pricing analysis, five candidate coefficients are screened using the same data. A selected coefficient estimate is \(1.8\) with standard error \(0.8\). Use a normal approximation to compare its ordinary \(95\%\) interval with a Bonferroni simultaneous interval for the five candidates at family-wise confidence level \(95\%\).

Method. The ordinary interval uses critical value \(1.96\). For the Bonferroni family, each two-sided interval has error probability \(0.05/5=0.01\), so its normal critical value is approximately \(2.576\).

The ordinary margin of error is \(1.96(0.8)=1.568\). Therefore:

1.8\pm1.568=[0.232,3.368].

The Bonferroni margin is \(2.576(0.8)=2.0608\). The simultaneous interval is:

1.8\pm2.0608=[-0.2608,3.8608]\approx[-0.261,3.861].

The ordinary interval excludes zero, while the Bonferroni interval includes zero. The family-wise interval is wider because it protects coverage across the five candidate coefficients, not just one coefficient treated in isolation. The check is consistent: using a larger critical value increases the margin of error. The normal approximation and standard-error calculation are assumed valid for each candidate coefficient.

Common mistake avoided: Taking the ordinary interval and interpreting its \(95\%\) label as a \(95\%\) guarantee after screening is not justified. The Bonferroni interval supplies a simultaneous guarantee under the stated setup, but it is not necessarily the shortest or most powerful adjustment.

Separating Selection From Inference

A practical way to reduce the direct effect of selection on ordinary inference is to use separate data for the two tasks. One sample is used to select a model; an independent sample is then used to estimate and test the selected coefficient. Conditional on the selection sample, the model choice is fixed for the independent inference sample. If the regression assumptions hold and the tested hypothesis is specified before examining the inference sample, ordinary inference can then be justified for that test.

The separation has a cost: each sample is smaller than the full data set, which can reduce precision and power. It also does not solve every multiplicity problem. If the analyst searches many coefficients or hypotheses in the inference sample, that search still needs to be accounted for. A prespecified analysis plan or independent replication can also separate confirmatory questions from exploratory model development.

Worked Example: Testing a Selected Predictor on Independent Data

A hypothetical agricultural insurer uses a training sample to choose one weather predictor for a regression of claim cost. On a separate, independent confirmation sample, the selected coefficient estimate is \(0.90\), its standard error is \(0.45\), and the regression assumptions for a normal approximation are taken to hold. The analyst tests the two-sided null hypothesis that the coefficient is zero at level \(0.05\), with no other hypotheses tested on the confirmation sample. Calculate the test statistic, p-value, and \(95\%\) interval.

Method. Because predictor selection used independent training data, the selected predictor is fixed relative to the confirmation outcomes. The confirmation-sample statistic is calculated as estimate divided by standard error:

z=\frac{0.90-0}{0.45}=2.00.

The two-sided p-value is \(2P(Z\geq2.00)=2(0.0228)=0.0456\), or \(0.0455\) using a more precise normal tail probability. Since \(0.0455<0.05\), reject the null for this prespecified confirmation-sample test.

The \(95\%\) interval uses critical value \(1.96\). Its margin is \(1.96(0.45)=0.882\), giving:

0.90\pm0.882=[0.018,1.782].

The interval excludes zero, agreeing with the two-sided test at the \(5\%\) level. This inference relies on the independence of the training and confirmation samples, the stated regression assumptions, and the fact that only the stated hypothesis was tested on the confirmation data. If the analyst had selected or altered the predictor after inspecting confirmation outcomes, this justification would no longer apply.

Common mistake avoided: Calling a random holdout sample “independent confirmation” is not enough if its outcomes are repeatedly inspected to revise the model. The separation must be preserved for the stated inference to be valid.

Interpreting Selected-Model Results

When reading a regression analysis, ask how the model was chosen before interpreting its coefficient table. A p-value can be reported as exploratory evidence, but it should not be presented as a prespecified confirmatory test if the same outcomes drove predictor selection. The relevant questions include:

How many predictors, models, transformations, or hypotheses were considered?
Did the selection use the response values later used for coefficient inference?
Does the reported adjustment match the selection process and the intended error guarantee?
Was inference performed on independent data, or were the results used to revise the model?

A Bonferroni adjustment is a clear option for controlling family-wise error across a specified set of tests, though it may be conservative. Specialized selective-inference methods can account more directly for a precisely defined selection event, but their validity depends on the method and assumptions. In all cases, adjustment cannot repair a poorly specified regression model or violations of its error assumptions.

The central interpretation is modest but important: after data-driven selection, ordinary coefficient output is not automatically confirmatory. Report what was searched, distinguish exploratory results from confirmatory inference, and make the adjustment or data separation match the question being answered.

Key takeaway: If the response data helped choose the predictor or model, ordinary p-values and confidence intervals generally ignore part of the analysis that produced the reported result. Account for the search, use independent inference data, or describe the results as exploratory.

Check Your Understanding

For each problem, identify the effect of selection and check the stated calculation or inference.

Under a global null, four independent two-sided tests are conducted at level \(0.05\), and the largest absolute statistic is reported. Calculate the probability of at least one false rejection. (Answer: \(1-0.95^4=0.1855\).)
A selected coefficient estimate is \(0.72\) with standard error \(0.30\), chosen from six candidates. Using a normal approximation, calculate its ordinary \(95\%\) interval and a Bonferroni simultaneous interval at family-wise level \(0.05\), using critical values \(1.96\) and \(2.638\), respectively. (Answer: ordinary interval \([0.132,1.308]\); Bonferroni interval \([-0.071,1.511]\), so only the ordinary interval excludes zero.)
An analyst tests 12 predictors, retains the one with the smallest p-value, and reports that unadjusted p-value as confirmatory. State the principal issue and give one simple family-wise procedure that could be used for the 12 tests. (Answer: the reported p-value ignores selection among 12 tests; Bonferroni uses per-test level \(0.05/12\approx0.00417\), or equivalently simultaneous intervals with error probability \(0.05/12\) each.)
A model is selected using a training sample; an independent confirmation sample gives a selected coefficient estimate of \(1.2\) and standard error \(0.5\). Assuming normal-approximation inference and one prespecified two-sided test, calculate the test statistic and approximate p-value. (Answer: \(z=1.2/0.5=2.4\); two-sided \(p\approx0.0164\).)
Three candidate coefficients have estimates \(0.6,1.1,-1.4\), each with standard error \(0.5\), and the analyst selects the largest absolute standardized statistic. What is the selected absolute statistic, and why is its ordinary single-test p-value not automatically a calibrated post-selection p-value? (Answer: the selected absolute statistic is \(1.4/0.5=2.8\); the result was chosen by searching the same data, so a single-test calculation ignores the selection rule and other candidates.)

What You'll Learn