Why Switching After a Strength Test Is Risky
A common analysis plan is to test whether the first stage appears sufficiently strong, then use a conventional IV confidence interval if it passes and a weak-IV-robust interval if it fails. The appeal is clear: conventional intervals may be narrower, while robust procedures are designed to remain valid when the first stage is weak. But the decision to switch is based on a random statistic from the same data used for inference. Passing the test does not turn a possibly weak instrument into a strong one, and the selected interval does not automatically retain its nominal coverage.
The issue is pretest selection. The first-stage statistic and the estimated IV effect are calculated from related data. Their sampling behavior can therefore be related. Conditioning on the first-stage test result can change the distribution of the estimate and the probability that an interval covers the target. A cutoff is a decision rule, not a guarantee about the coverage of the interval chosen afterward.
The Coverage Calculation
Consider a linear IV model with outcome \(Y\), endogenous treatment \(D\), instruments \(Z\), and included covariates \(X\). For a scalar treatment effect, one representation is
Here \(\beta\) is the target effect under the model, \(\pi\) describes the first-stage relationship, and \(u\) and \(v\) are structural and first-stage disturbances. The IV interpretation requires the relevant instruments to be relevant for \(D\), independent of the structural error conditional on \(X\), and excluded from the outcome equation except through \(D\). The model and inference also require appropriate sampling and variance assumptions. Weak-IV-robust inference addresses uncertainty caused by weak relevance; it does not repair invalid instruments, an incorrect structural model, or an unsuitable variance estimator.
Let \(S\) denote the event that the strength statistic passes the screening rule. Write \(C_{\mathrm{conv}}\) for the event that the conventional interval covers \(\beta\), and \(C_{\mathrm{rob}}\) for the event that the robust interval covers it. The selected interval covers exactly when the conventional interval covers on \(S\), or the robust interval covers on the complement of \(S\). Therefore:
This follows by partitioning the coverage event into the disjoint cases \(S\) and \(S^c\), then applying the definition of conditional probability to each case. The expression is a weighted average of conditional coverage rates. A procedure aimed at 95% coverage does not ensure that either conditional rate is 95%, and two nominal procedures do not automatically combine into another nominal procedure through screening.
Conventional Wald inference is justified in a strong-instrument asymptotic regime, with suitable regularity conditions and a valid standard error. Its familiar form is an estimate plus or minus a critical value times its standard error. When instruments are weak, the estimator's distribution can be poorly approximated by the usual normal approximation. A first-stage test has a random pass/fail outcome; passing it is not a proof that the strong-instrument approximation applies.
Worked Example: Combining Branch-Specific Coverage
Suppose a hypothetical repeated-sampling exercise has 10,000 replications under one fixed IV data-generating process. In 6,000 replications, the first-stage statistic passes the chosen cutoff. The conventional interval covers the true effect in 5,400 of those replications. In the other 4,000 replications, the robust interval covers in 3,840.
The pass probability is \(6{,}000/10{,}000=0.60\), and conventional coverage among passing replications is \(5{,}400/6{,}000=0.90\). Robust coverage among failing replications is \(3{,}840/4{,}000=0.96\). Thus selected-interval coverage is
Equivalently, the selected interval covers in \(5{,}400+3{,}840=9{,}240\) of 10,000 replications, or 92.4%. In this hypothetical example the rule undercovers a 95% target by 2.6 percentage points. The calculation illustrates the mixture; it is not a claim that every pretest-based rule must under-cover. Different conditional rates and pass probabilities can produce a different result.
What a Pass Does—and Does Not—Tell You
A first-stage strength test evaluates a feature of the first-stage relationship under its own reference distribution and assumptions. It does not test every condition needed for a causal IV interpretation. In particular, it does not establish exclusion or independence. Nor does failing to reject a zero-first-stage null establish that the instrument is weak: a test result also reflects sample size and sampling variability.
Even if the instrument is strong enough in the population for conventional inference to work asymptotically, the sample can fall on either side of a screening cutoff. Consequently, the event of selecting conventional inference can be associated with features of the sample that also affect its interval coverage. The cutoff does not remove this selection issue. The concern is especially consequential near the boundary where weak- and strong-instrument behavior are difficult to distinguish.
A weak-IV-robust confidence set, such as one obtained by inverting an Anderson–Rubin test under its assumptions, is designed not to rely on the usual strong-first-stage approximation. It may be wide, unbounded, or disconnected, and it can have lower power against some alternatives. Those features are not evidence that the method failed: they can communicate that the data do not sharply identify the effect. Robustness to weak relevance is not robustness to every modeling failure.
Worked Example: A Passing Statistic and Two Intervals
Suppose a hypothetical study reports a first-stage statistic of \(F=12.4\), and the analysis plan uses \(c=10\) as its screening cutoff. It reports a conventional estimate of \(0.42\) outcome units per treatment unit with standard error \(0.15\). For illustration, use the normal critical value \(1.96\) for a 95% Wald interval. The rule selects the conventional interval because \(12.4 \geq 10\).
The conventional interval is
Suppose the same hypothetical analysis also gives a weak-IV-robust confidence set of \([-0.10,\ 0.91]\) outcome units per treatment unit. The narrower conventional interval is the one selected by the stated rule. But the observed pass does not, by itself, establish that its coverage is 95%; that claim depends on the strong-instrument approximation and its assumptions, not on the cutoff alone. The robust set is wider here and includes zero, as does the conventional interval. Neither interval proves that the true effect lies within it; the 95% interpretation concerns the long-run performance of the procedure under its assumptions.
How the Cutoff Changes the Selected Procedure
The cutoff affects which method is used and how often each branch contributes to coverage. Even if each method's performance in broad strength groups is known, the coverage of the switching rule must be calculated from the appropriate branch-specific rates. The following hypothetical table makes that dependence visible. Each row summarizes 1,000 replications in a strength-statistic range.
| First-stage statistic range | Replications | Conventional interval covers | Robust interval covers |
|---|---|---|---|
| Below 5 | 200 | 160 (80%) | 190 (95%) |
| 5 to below 10 | 200 | 180 (90%) | 194 (97%) |
| 10 to below 15 | 300 | 270 (90%) | 291 (97%) |
| 15 or above | 300 | 285 (95%) | 294 (98%) |
Worked Example: Comparing Two Screening Cutoffs
Use the hypothetical replication counts above. With cutoff \(c=10\), choose the robust interval below 10 and the conventional interval at or above 10. The number of selected intervals that cover is \(190+194+270+285=939\). Since there are 1,000 replications, selected coverage is \(939/1{,}000=93.9\%\).
With cutoff \(c=15\), choose the robust interval below 15 and the conventional interval at or above 15. The covering count is \(190+194+291+285=960\), giving \(960/1{,}000=96.0\%\). The two cutoffs yield different coverage in these hypothetical data because they select different branches for the 10-to-below-15 group. These calculations do not identify a universally best cutoff; they show why a cutoff is part of the inference procedure and why its consequences cannot be ignored.
Alternatives to Data-Dependent Switching
Use a procedure whose validity does not require a strong first stage, provided the IV assumptions and the procedure's other conditions hold. This avoids choosing the interval based on a first-stage pass/fail result, though the resulting set may be less precise.
When the design and asymptotic argument support conventional Wald inference, state that basis and the variance assumptions. Do not treat a sample screening pass as a substitute for that justification.
Where useful, report conventional and weak-IV-robust results with clear labels and assumptions. Presenting both can reveal how conclusions depend on the inference method, but selecting whichever interval is more favorable after seeing results does not create a guaranteed-coverage procedure.
The first option prioritizes protection against weak relevance; the second prioritizes conventional efficiency when its approximation is credible. Neither option removes the need to assess instrument validity and model specification. If a strength test is reported as a diagnostic, it is useful information about the observed first stage, but it should not be portrayed as an automatic license to switch to conventional inference.
Common Mistakes and Practical Checks
- “The test passed, so the instrument is strong.” A pass is a sample result relative to a chosen test and cutoff. It does not establish strong-instrument asymptotics or instrument validity.
- “Each interval is 95%, so the selected interval is 95%.” Selection changes the relevant calculation. The combined coverage depends on the pass probability and the coverage rates conditional on each branch.
- “The robust interval should be used only when the test fails.” That decision rule itself is a pretest. If the goal is weak-IV-robust coverage, using the robust procedure throughout avoids this particular switching step.
- “A wider robust set is worse.” Width reflects precision, while coverage is a repeated-sampling property. A set can be wide precisely because the data provide limited information under weak identification.
- “The first-stage test checks the IV assumptions.” It does not verify exclusion or independence. Strength diagnostics and causal validity are distinct questions.
When evaluating an analysis, identify the exact selection rule, state which interval is used in each branch, and ask whether coverage is justified for that full rule rather than for either interval considered in isolation. If evidence comes from simulation, examine branch-specific rates as well as the overall rate; an aggregate alone can hide where the selection rule is performing poorly.
Check Your Understanding
Answer each question, then compare with the solutions below.
- A rule selects the conventional interval when a strength test passes with probability \(0.70\). Its coverage conditional on passing is \(0.92\), and robust-interval coverage conditional on failing is \(0.97\). What is selected coverage?
- Why does rejecting a zero-first-stage null not establish that a conventional IV interval has nominal coverage?
- What does a weak-IV-robust interval protect against, and name one important problem it does not fix.
- In a hypothetical set of 500 replications, a rule selects conventional intervals for 300 cases, which cover 270 times. It selects robust intervals for 200 cases, which cover 194 times. Calculate total selected coverage.
Solutions. 1. The coverage is \(0.92(0.70)+0.97(0.30)=0.644+0.291=0.935\), or 93.5%. 2. The test result is random and does not prove that strong-instrument asymptotics apply; moreover, first-stage strength is not the only requirement for valid inference. 3. It protects against reliance on a strong-first-stage approximation, under its stated assumptions. It does not correct invalid instruments, such as a violation of exclusion. 4. The selected interval covers \(270+194=464\) times, so coverage is \(464/500=0.928\), or 92.8%.