Why the Instrument Set Matters
An instrumental-variables analysis depends on the instruments included in the model. Adding a defensible instrument can increase first-stage information, but it can also change the estimate, the uncertainty, and the weak-instrument-robust confidence set. A result that changes substantially across plausible instrument subsets is a warning that the conclusion depends on modeling choices—not a reason to report only the most convenient subset.
A useful comparison holds the sample, outcome, treatment, included exogenous covariates, and estimation conventions fixed while changing only the candidate instrument set. Each candidate set must still be defensible: instrument relevance, independence from unmeasured causes of the outcome, and exclusion of a direct effect on the outcome are substantive assumptions. First-stage strength does not establish those assumptions.
What to Compare
For a set of \(q\) instruments, a first-stage diagnostic tests whether their coefficients in the treatment equation are jointly zero, conditional on the included covariates. With heteroskedasticity-robust covariance estimation, the joint Wald statistic is asymptotically chi-square with \(q\) degrees of freedom under that null. Reporting the statistic, degrees of freedom, and p-value is more informative than treating a single cutoff as a universal pass/fail rule. The statistic divided by \(q\) can be shown as a descriptive scaled measure, but it is not automatically a classical \(F\) statistic with its usual finite-sample reference distribution.
First-stage diagnostics concern relevance. They do not test whether the instruments satisfy independence or exclusion. Also, when the number of instruments changes, the degrees of freedom and the way strength is summarized change too. Compare the underlying statistics and their degrees of freedom, and avoid interpreting a difference in a scaled statistic as though every set had the same testing problem.
Weak-IV-robust tests and confidence sets provide a complementary comparison. For an Anderson–Rubin (AR) analysis, a candidate treatment effect \(b\) is tested by asking whether the reduced-form outcome effects are consistent with \(b\) times the first-stage effects. Inverting these tests gives a confidence set that remains valid under weak identification, subject to the model and covariance assumptions. It may be wide, unbounded, or disconnected; those shapes are part of the diagnostic information, not calculation errors.
A Scalar Derivation for the AR Confidence Set
The scalar case makes clear why changing instruments can reshape a confidence set. For one instrument, let \(\hat{p}\) be the estimated first-stage coefficient and \(\hat{r}\) the estimated reduced-form coefficient, using the same sample and covariates. Let \(V_p\) and \(V_r\) be their estimated variances and \(C\) their estimated covariance. Under the null that the treatment effect is \(b\), the reduced-form discrepancy is \(\hat{r}-b\hat{p}\). Its estimated variance is
This follows from the variance rule \(\operatorname{Var}(A-bB)=\operatorname{Var}(A)+b^2\operatorname{Var}(B)-2b\operatorname{Cov}(A,B)\). The robust AR statistic is the squared discrepancy divided by this variance. Inverting a two-sided 95% test therefore retains the values of \(b\) satisfying
The covariance estimate must come from a valid joint covariance calculation for the reduced-form and first-stage estimates; it cannot be omitted just because the two estimates are reported separately. For multiple instruments, the analogous AR statistic uses the vector of reduced-form discrepancies and its covariance matrix, with a chi-square reference distribution whose degrees of freedom equal the number of instruments under the usual large-sample conditions. Confidence sets are obtained by inverting the corresponding tests.
Worked Example: A Weak Set and a Stronger Set
Suppose two prespecified, defensible single-instrument models use the same hypothetical sample and covariates. In each model, the reduced-form and first-stage estimates have \(V_r=0.0009\), \(V_p=0.0004\), and \(C=0.0002\). Set A gives \(\hat{p}=0.04\), \(\hat{r}=0.06\); Set B gives \(\hat{p}=0.12\), \(\hat{r}=0.18\). The corresponding first-stage Wald statistics are \((0.04/0.02)^2=4\) and \((0.12/0.02)^2=36\). The point estimate in each case is \(\hat{r}/\hat{p}=1.50\).
For Set A, substitution into the 95% AR inequality gives
Expanding and moving the right side to the left gives
The roots are approximately \(0.04375\) and \(51.429\). Since the quadratic coefficient is positive, the inequality holds between the roots. The approximate 95% AR confidence set is therefore \([0.04375,\,51.429]\), in the units of the treatment-effect parameter.
For Set B, the corresponding inequality is
which expands to
Its roots are approximately \(1.009\) and \(2.230\), giving the confidence set \([1.009,\,2.230]\), again in treatment-effect units. Both sets contain the point estimate \(1.50\), as the AR discrepancy is zero at that value. The first set is much wider despite having the same point estimate: the first-stage Wald statistic is 4 rather than 36. The comparison illustrates how weak first-stage information can leave a large range of effects compatible with the data.
These calculations do not establish that either instrument is valid. The covariance matrices are positive definite in this illustration: \(V_rV_p-C^2=0.0009(0.0004)-(0.0002)^2=0.00000032>0\). In an actual analysis, the variance estimates and covariance must be calculated from the fitted models, and the assumptions supporting the chosen instruments must be argued separately.
Interpreting Differences Across Subsets
A useful comparison distinguishes changes in strength from changes in the substantive result. If the first-stage diagnostic weakens sharply for one subset and its AR set becomes extremely wide, the data provide limited information about the effect under that specification. If estimates or confidence sets shift markedly even among subsets with reasonably informative first stages, investigate whether the instruments identify different effects, whether an exclusion restriction is questionable, or whether model and sample choices differ inadvertently.
In a multiple-instrument model with reduced-form estimate vector \(\hat{\mathbf r}\) and first-stage estimate vector \(\hat{\mathbf p}\), the AR discrepancy at \(b\) is \(\hat{\mathbf r}-b\hat{\mathbf p}\). Its covariance matrix must account for uncertainty and covariance in both vectors. The test statistic compares this discrepancy with that covariance; its inversion can yield a set unlike the scalar interval in the example. Changing the instrument subset changes both the discrepancy vector and the covariance calculation, so there is no general rule that confidence sets must become narrower as instruments are added.
Worked Example: Adding a Candidate Instrument
In a hypothetical analysis, instrument \(Z_1\) has a first-stage coefficient of \(0.10\) with robust standard error \(0.04\). A second candidate, \(Z_2\), has coefficient \(0.02\) with robust standard error \(0.04\). Suppose their first-stage estimates are uncorrelated for this illustration. With \(Z_1\) alone, the one-degree-of-freedom Wald statistic is
With both instruments, the joint Wald statistic is
with two degrees of freedom. Its statistic divided by the number of instruments is \(6.50/2=3.25\), which is a descriptive scaled value, not a classical \(F\) statistic by default. The added instrument increases the joint Wald statistic only slightly while adding a degree of freedom; the joint test has an asymptotic chi-square reference distribution with two degrees of freedom.
This arithmetic is evidence about first-stage relevance, not instrument validity. Before including \(Z_2\), the analyst must assess whether its independence and exclusion assumptions are credible. If it is defensible, compare weak-IV-robust confidence sets from the \(Z_1\)-only and \(Z_1,Z_2\) specifications. If the larger set is wider or changes materially, report that sensitivity rather than assuming that adding an instrument must improve inference.
Worked Example: Summarizing Confidence-Set Instability
Suppose a hypothetical analysis prespecifies two defensible instrument subsets and obtains AR 95% confidence sets of \([0.4,\,4.8]\) and \([1.1,\,1.9]\), respectively, in outcome units per treatment unit. Their widths are \(4.8-0.4=4.4\) and \(1.9-1.1=0.8\), so the first set is \(4.4/0.8=5.5\) times as wide. Both contain the estimate \(1.5\), but the first permits a much broader range of effects.
The correct conclusion is that the range of effects compatible with the data is sensitive to the instrument set, with substantially less precision under the first specification. It is not correct to say that the second subset proves the effect is near \(1.5\), or that the first interval gives a 95% probability that the fixed parameter lies in it. The confidence procedure has its stated repeated-sampling coverage under its assumptions; the comparison is a sensitivity analysis, not a probability distribution over the parameter.
Diagnostics, Alternatives, and Pitfalls
For a clear comparison, state which subsets were considered and why they are substantively defensible. Keep the sample, covariates, outcome and treatment definitions, and covariance approach fixed whenever possible. Then report the first-stage coefficient estimates, their uncertainty, the joint diagnostic with degrees of freedom, and the weak-IV-robust confidence set for every subset. If a set is unbounded or disconnected, report its form rather than replacing it with a convenient finite interval.
Compare weak-IV-robust results such as AR confidence sets across the subsets. Conventional 2SLS intervals may be reported as a secondary comparison, but they can be misleading under weak identification. The earlier material on weak-IV-robust tests and confidence sets covers their construction; here the essential point is to apply a consistent procedure to each defensible set rather than selecting a set after seeing which result is most appealing. Other robust procedures may have different power or set shapes, so identify the procedure used and do not treat results from different procedures as interchangeable.
A particularly important limitation is that stability cannot validate instruments. Several sets may produce similar estimates while sharing the same invalid exclusion assumption. Conversely, differences can arise from sampling variability, weak identification, or treatment-effect heterogeneity, as well as from a problematic instrument. If effects differ across the populations or margins identified by different instruments, a change in the estimand may be substantively meaningful rather than a simple model failure.
- Changing several things at once. If the sample or covariates also change, the effect of changing the instrument set cannot be isolated. Hold other choices fixed or clearly explain each change.
- Equating strength with validity. A large first-stage statistic only indicates relevance under its model; it does not test exclusion or independence.
- Selecting the “best” set after inspecting outcomes. Choosing instruments because they yield a preferred estimate or narrow interval makes the reported sensitivity comparison misleading. Prespecify defensible subsets where possible and show the full comparison.
- Calling an AR set centered on the ratio estimate. The ratio \(\hat{r}/\hat{p}\) is included when defined, but the AR variance depends on \(b\); the set need not be symmetric or centered at that estimate.
- Reading one cutoff as decisive. First-stage diagnostics depend on the number of instruments, covariance estimator, and sample information. Report the diagnostic and uncertainty rather than relying on a universal threshold.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Two instrument subsets have similar first-stage diagnostics but substantially different AR confidence sets. What can be concluded, and what cannot be concluded about validity?
- A one-instrument analysis has \(\hat{p}=0.05\) and robust standard error \(0.025\). Calculate its Wald statistic. What does that number address?
- Why can adding an instrument fail to narrow a weak-IV-robust confidence set?
- In a scalar AR analysis, why does the confidence set not have to be centered at \(\hat{r}/\hat{p}\)?
- What should be held constant when comparing instrument subsets, and why?
Solutions. (1) The inference is sensitive to the chosen subset, and the difference warrants investigation. Similar first-stage diagnostics do not establish independence or exclusion, so neither subset is thereby shown to be valid. (2) The statistic is \((0.05/0.025)^2=2^2=4\). It addresses first-stage relevance under the specified model, not instrument validity. (3) Adding an instrument changes the dimension of the test and the covariance of the reduced-form discrepancies; the added instrument may provide little first-stage information, so the inverted robust test need not yield a narrower set. (4) Although the ratio makes \(\hat{r}-b\hat{p}=0\), the AR statistic's denominator depends on \(b\), so the accepted set need not be symmetric around that ratio. (5) Hold the sample, outcome and treatment definitions, covariates, and estimation and covariance conventions fixed. Otherwise, changes in results cannot be attributed specifically to the instrument subset.