Post-Selection Inference for Instrument Choice

Why Instrument Choice Can Change Inference

When several candidate instruments are available, it can be tempting to fit a first stage for each, keep the one that looks strongest, and then report the corresponding IV estimate and confidence interval. Another tempting rule is to choose the instrument with the most striking association with the outcome. These procedures use the observed data twice: first to choose an instrument, and again to estimate an effect or test a hypothesis.

The issue is not merely that the reported instrument was selected. Selection makes the selected data look unusually favorable according to the criterion used to choose it. A first-stage association selected for being large tends to overstate its underlying strength. An outcome association selected for being extreme tends to look more significant than an association chosen in advance. Standard errors and p-values calculated as if the instrument had been fixed beforehand need not describe the resulting procedure.

For a simple illustration, consider a constant-effect linear IV model with one endogenous treatment \(D\), an outcome \(Y\), and a candidate instrument \(Z_j\). After accounting for any included covariates, a just-identified IV estimate has the ratio form \( \hat{\beta}_j = \widehat{\mathrm{RF}}_j / \widehat{\pi}_j \), where \(\widehat{\mathrm{RF}}_j\) is the estimated association of \(Z_j\) with \(Y\), and \(\widehat{\pi}_j\) is its estimated first-stage association with \(D\). This ratio is meaningful as a causal estimate only under the relevant IV assumptions, including instrument relevance, independence from unmeasured causes of the outcome, and exclusion of a direct effect on the outcome. Selecting an instrument does not establish those assumptions.

Selection on First-Stage Strength

A common selection rule is to choose the candidate with the largest observed first-stage statistic: for example, the largest absolute first-stage \(t\)-statistic, partial \(R^2\), or first-stage \(F\)-statistic. These are useful diagnostics when interpreted in the analysis plan, but choosing the maximum makes its observed value a selected maximum rather than an ordinary estimate of the chosen instrument’s strength.

The basic mechanism can be seen with two candidates that have the same true first-stage coefficient \(\pi\). Suppose their estimates are \( \hat{\pi}_1 = \pi + \epsilon_1 \) and \( \hat{\pi}_2 = \pi + \epsilon_2 \), where the errors are independent and each follows \(N(0,\sigma^2)\). If the rule selects the larger estimated coefficient, its selected estimate is \( \max(\hat{\pi}_1,\hat{\pi}_2) \). Since a maximum favors positive estimation error, its expectation exceeds \(\pi\).

Here \(\pi_{\mathrm{math}}\) denotes the mathematical constant (approximately 3.142), not a first-stage coefficient. The second equality follows from \(\max(a,b)=(a+b+|a-b|)/2\), and the last uses \(\epsilon_1-\epsilon_2 \sim N(0,2\sigma^2)\). The calculation is deliberately simple: real candidate first stages may be correlated, have different true strengths, or be selected by a more complicated statistic. The general lesson remains that selecting an unusually favorable noisy diagnostic can make the selected diagnostic overstate what would be seen in new data.

Selection on Outcome Associations

Choosing an instrument because its observed association with the outcome is largest, most statistically significant, or most favorable to a preferred conclusion creates a more direct multiplicity problem. Even if every candidate instrument is valid and all corresponding null hypotheses are true, random outcome-association estimates will vary. Selecting the most extreme estimate preferentially retains a large one. A conventional p-value for that selected estimate treats it as though it had been chosen in advance, and therefore does not generally measure the error rate of the whole search-and-report procedure.

This problem is separate from the exclusion restriction. A strong observed association between an instrument and the outcome does not show that the instrument affects the outcome only through the treatment. Conversely, selecting an outcome association because it is large does not itself prove that the instrument is invalid. It means the reported estimate and test are affected by the outcome-based choice, while validity still requires substantive justification.

In practice, instruments may have different first-stage strengths and their reduced-form estimates may be correlated. The simple calculation is not a universal correction. It demonstrates why a selected p-value cannot automatically be interpreted using the single-test reference distribution. A procedure designed for the actual selection process, or data reserved for inference after selection, is needed for a justified claim.

Prespecification and Sample Splitting

The clearest safeguard is to specify the instrument set and selection rule before examining the analysis outcomes. A prespecified choice can be based on substantive knowledge, design features, or external evidence. If a rule uses a first-stage diagnostic, it should be defined in advance, including which candidates are eligible and what criterion governs the choice. A researcher should not quietly switch instruments after seeing which one produces the preferred first stage, estimate, or p-value.

Prespecification prevents the particular data-driven search described here from distorting inference as if the choice had been fixed in advance. It does not guarantee that the chosen instrument is valid, sufficiently strong, or correctly modeled. Nor does it eliminate the need to use inference appropriate to instrument strength. A preplanned choice that turns out to be weak remains a weak-instrument problem.

When selection must be data-driven, sample splitting offers a transparent safeguard. Randomly divide independent observational units into a selection sample and an inference sample. Use the selection sample to choose the instrument according to the stated rule. Then estimate the IV model and conduct inference using only the inference sample, treating the selected instrument as fixed. Conditional on the selection sample, the selected instrument is fixed, and the independent inference sample has not been used to favor it.

Limits and Common Mistakes

Sample splitting works only if the inference data are genuinely independent of the data used to select the instrument. With clustered or repeated observations, splitting individual records may leave dependent observations on both sides; splitting independent clusters or other independent units may be necessary. Reusing the inference sample to revise the choice, or combining both samples in a conventional analysis after selection, can reintroduce the selection problem. Repeated splits or combining estimates across splits also require an inferential procedure that accounts for that extra step.

The same logic applies if selection uses outcome associations in the discovery sample: a separate independent inference sample can protect the held-out test from that particular selection, provided the analysis is otherwise justified. But it does not make a selected instrument valid. Nor does a strong first stage in the discovery data guarantee a strong first stage in the inference data. The held-out first stage and the possibility of weak identification still matter.

• Mistake: “The strongest observed first stage must be the best instrument.” A large selected diagnostic can partly reflect noise. Strength is not validity, and selection can make observed strength optimistic.
• Mistake: “The selected instrument’s ordinary p-value is still a 5% test.” That is generally not true after searching the same data and reporting the most favorable result. The choice process must be accounted for.
• Mistake: “Sample splitting proves the selected instrument is valid.” It separates selection from inference under independence; it does not establish exclusion, independence, or relevance.
• Mistake: “Selection always pushes the IV estimate in one direction.” Selection changes the estimator’s distribution, but the direction of bias depends on the data-generating process and selection rule.