Common Errors in Weak-Instrument Diagnosis

Introduction: Strength Is Not Validity

Weak-instrument diagnosis involves several different questions that are easy to collapse into one. Does an instrument predict the endogenous regressor? Are the assumptions that make it a valid instrument credible? And does the chosen inferential procedure remain reliable when the first-stage relationship is weak? A first-stage statistic addresses the first question, not automatically the other two.

This distinction matters because a strong but invalid instrument can produce precisely estimated answers to the wrong model. Conversely, when identification is weak, a conventional Wald interval may look straightforward while its usual approximation is unreliable. The useful response is not to search for one magic diagnostic number, but to examine relevance, validity, and inference separately.

Definition: In the linear instrumental-variables model, an instrument is relevant if it predicts the endogenous regressor after conditioning on included exogenous regressors. It is valid only if the required exogeneity and exclusion restrictions also hold. Relevance is a property of the first-stage relationship; validity is a property of the assumptions connecting the instrument to the outcome equation.

Mistake 1: Treating a Large First-Stage F as Proof of Validity

A first-stage F statistic tests whether excluded instruments predict the endogenous regressor, conditional on included controls. A large value is evidence against a weak first-stage relationship in the setting covered by that statistic. It does not test whether an instrument is independent of the structural error or whether it affects the outcome only through the endogenous regressor.

The separation can be seen directly in a simple model. Let \(Z\) be an instrument, \(X\) an endogenous regressor, and \(Y\) an outcome:

X=\pi Z+v,\qquad Y=\beta X+\delta Z+u.

Here \(\pi\) determines the first-stage relationship. If \(\pi\) is large, \(Z\) can predict \(X\) strongly. But when \(\delta\ne 0\), the outcome equation includes a direct effect of \(Z\), violating the exclusion restriction for using \(Z\) as an instrument for \(X\). The first-stage regression contains no information about \(\delta\).

Result: A first-stage strength statistic alone cannot establish instrument validity. In the model above, changing \(\delta\) leaves the joint distribution of \((Z,X)\), and therefore every first-stage statistic based on \((Z,X)\), unchanged.

Proof. Hold the joint distribution of \((Z,v)\) and the value of \(\pi\) fixed. The first-stage variable is \(X=\pi Z+v\), so its joint distribution with \(Z\) does not depend on \(\delta\), \(u\), or the outcome equation. Any statistic calculated solely from the first-stage regression is therefore unchanged when \(\delta\) changes. But \(\delta=0\) satisfies the displayed exclusion restriction, while \(\delta\ne0\) violates it. Thus the same first-stage evidence can occur under both a model with and a model without exclusion. \(\square\)

Worked Example: A Strong First Stage with a Direct Instrument Effect

Suppose \(Z\), \(v\), and \(u\) are mutually independent, each with mean zero, and \(\operatorname{Var}(Z)=\operatorname{Var}(v)=\operatorname{Var}(u)=1\). Consider \(X=4Z+v\) and \(Y=2X+0.5Z+u\). Does a strong first stage make \(Z\) valid?

The first-stage regression is \(X=4Z+v\). Its population \(R^2\) is \[ \frac{\operatorname{Var}(4Z)}{\operatorname{Var}(4Z)+\operatorname{Var}(v)} =\frac{16}{16+1}=\frac{16}{17}\approx0.941. \] This is a strong population relationship: \(Z\) explains about 94.1% of the variation in \(X\). The corresponding first-stage signal grows with sample size; under the usual homoskedastic single-instrument calculation, its noncentrality is of order \(16n\).

But the outcome equation contains \(0.5Z\) directly. The exclusion restriction would require this direct effect to be zero, so it fails. Independence of \(Z\) and \(u\) does not repair the violation: the instrument affects \(Y\) through both \(X\) and the additional \(0.5Z\) term. The first stage is strong, but the instrument is not valid for the stated model.

Instrument validity generally requires substantive design knowledge, institutional argument, or additional assumptions. Some specification tests can assess implications of validity assumptions in particular settings, but they do not turn a first-stage F statistic into a validity test.

Mistake 2: Applying One Threshold in Every Setting

A threshold is meaningful only together with the statistic, model, and performance criterion for which it was developed. Stock–Yogo weak-instrument thresholds, for example, are indexed by features such as the number of instruments and endogenous regressors, and by a chosen criterion such as a bound on relative bias or size distortion. They are not universal cutoffs that certify adequate identification in every model.

The relation between a joint first-stage F and partial \(R^2\) also depends on the degrees of freedom. For \(q\) excluded instruments and residual degrees of freedom \(\nu\), the usual homoskedastic joint-test identity is

F=\frac{R^2_{\mathrm{partial}}/q}{(1-R^2_{\mathrm{partial}})/\nu}, \qquad R^2_{\mathrm{partial}}=\frac{qF}{\nu+qF}.

Thus even the same reported F does not imply the same partial \(R^2\) across designs. The relevant threshold comparison also has to match the assumptions behind the diagnostic. A heteroskedasticity-robust first-stage statistic is not automatically calibrated by a critical value derived for a homoskedastic statistic.

Worked Example: The Same F, Different Partial R-Squared

Compare two hypothetical first-stage tests, each reporting \(F=12\). In the first, one excluded instrument is tested and the residual degrees of freedom are \(\nu=198\). In the second, ten excluded instruments are tested and \(\nu=189\). Calculate the partial \(R^2\) in each case.

For \(q=1\), \[ R^2_{\mathrm{partial}}=\frac{1(12)}{198+1(12)} =\frac{12}{210}\approx0.0571. \] For \(q=10\), \[ R^2_{\mathrm{partial}}=\frac{10(12)}{189+10(12)} =\frac{120}{309}\approx0.3883. \]

The reported F is the same, but the partial \(R^2\) values differ substantially because the number of restrictions and residual degrees of freedom differ. Neither calculation, on its own, establishes that instruments are valid. Nor does this comparison determine which design has acceptable weak-instrument bias: that requires the relevant model, assumptions, and inferential criterion.

A disciplined report names the diagnostic and its assumptions, identifies the numbers of instruments and endogenous regressors, and states the criterion used to interpret the result. With heteroskedasticity, use an appropriate robust diagnostic rather than silently treating a conventional homoskedastic F as interchangeable. Even then, a robust first-stage statistic remains a strength diagnostic, not a validity test.

Mistake 3: Reporting a Conventional Wald Interval Without Checking Weak-IV Robustness

Under strong identification, the first-stage coefficient stays separated from zero as the sample grows. Standard large-sample approximations can then justify a roughly normal distribution for an instrumental-variables estimator. Under weak identification, the first-stage coefficient may be small enough that sampling variation is comparable to the signal. The estimator then behaves like a ratio with a random denominator; the usual approximation based on a stable denominator is not justified.

This point can be made in a just-identified model with one instrument and one endogenous regressor. Suppose observations satisfy \(X_i=\pi_n Z_i+v_i\) and \(Y_i=\beta X_i+u_i\), with no additional controls. Assume the observations are iid, \(E[Z_i]=0\), \(E[Z_i^2]=Q>0\), \(E[Z_i u_i]=E[Z_i v_i]=0\), and the vector \((Z_i u_i,Z_i v_i)\) has finite second moments and satisfies a multivariate central limit theorem. Let the first-stage coefficient follow the local-to-zero sequence \(\pi_n=c/\sqrt n\). The just-identified IV estimator satisfies the exact identity

\hat{\beta}-\beta =\frac{\sum_{i=1}^n Z_i u_i}{\sum_{i=1}^n Z_i X_i} =\frac{n^{-1/2}\sum_{i=1}^n Z_i u_i} {c\,n^{-1}\sum_{i=1}^n Z_i^2+n^{-1/2}\sum_{i=1}^n Z_i v_i}.

The numerator and the random part of the denominator have joint limiting distributions by the assumed central limit theorem, while \(n^{-1}\sum Z_i^2\xrightarrow{p}Q\). If \(\operatorname{Var}(Z_i v_i)>0\), the limiting denominator \(cQ+V\) has a continuous distribution and is nonzero with probability one. The continuous mapping theorem therefore gives the ratio limit \(U/(cQ+V)\), where \((U,V)\) is the joint normal limit of the two sums. In general this is not the normal approximation obtained by treating the denominator as fixed. The strong-identification justification for a conventional Wald interval is therefore missing in this weak-identification regime.

Worked Example: A Wald Interval and an Unbounded Robust Set

Consider a simplified Gaussian reduced-form model. Let \(A\) estimate the first-stage coefficient \(\pi\), and let \(B\) estimate the reduced-form coefficient \(\beta\pi\). Suppose \(A\) and \(B\) are independent normal variables with known standard errors \(0.05\) each. The observed values are \(A=0.06\) and \(B=0.12\). The ratio estimate is \[ \hat{\beta}=\frac{B}{A}=\frac{0.12}{0.06}=2. \]

A delta-method standard error for this ratio is \[ \sqrt{\frac{0.05^2}{0.06^2}+ \frac{0.12^2(0.05^2)}{0.06^4}} =\sqrt{0.6944+2.7778} \approx1.863. \] The corresponding approximate 95% Wald interval is \(2\pm1.96(1.863)\), or approximately \([-1.65,5.65]\). But \(A/0.05=1.2\): the observed first-stage estimate is only 1.2 standard errors from zero, so the denominator is poorly determined.

For a candidate value \(b\), an Anderson–Rubin-style test in this known-variance Gaussian model compares \(B-bA\) with its standard deviation \(\sqrt{0.05^2+b^2(0.05^2)}\). Inverting the two-sided 5% test gives \[ \frac{(0.12-0.06b)^2}{0.0025+0.0025b^2}\le1.96^2. \] Expanding and rearranging yields \[ 0.006004b^2+0.0144b-0.004796\ge0. \] The roots are approximately \(-2.695\) and \(0.296\). Since the quadratic opens upward, the confidence set is approximately \((-\infty,-2.695]\cup[0.296,\infty)\). Unlike the finite Wald interval, it is unbounded and disconnected. That is not a calculation error: it reflects how little the data determine the ratio when the first-stage coefficient is uncertain.

Result: In the independent Gaussian reduced-form model above, Anderson–Rubin inversion has exact coverage at its nominal level for every first-stage coefficient \(\pi\), including \(\pi=0\), when the variances are known.

Proof. Suppose \(A\sim N(\pi,s_A^2)\) and \(B\sim N(\beta\pi,s_B^2)\) independently, with known positive \(s_A^2\) and \(s_B^2\). For the true value \(\beta\), the difference \(B-\beta A\) is normal with mean \[ E[B-\beta A]=\beta\pi-\beta\pi=0 \] and variance \[ \operatorname{Var}(B-\beta A)=s_B^2+\beta^2s_A^2, \] by independence. Hence \((B-\beta A)/\sqrt{s_B^2+\beta^2s_A^2}\sim N(0,1)\) for every \(\pi\). A two-sided test using the standard normal 97.5th percentile therefore rejects the true \(\beta\) with probability \(0.05\), and its inverted confidence set contains \(\beta\) with probability \(0.95\). This exact argument is specific to the stated Gaussian model with known variances; it does not assert exact finite-sample coverage under arbitrary designs or error distributions. \(\square\)

A Practical Diagnostic Sequence

Weak-instrument diagnosis is most useful when it keeps separate questions separate. The following sequence prevents a strength result from being asked to answer questions it was not designed to answer.

Assess relevance.
Report the first-stage diagnostic appropriate to the design, along with the number of instruments, endogenous regressors, controls, and relevant degrees of freedom.

Interpret strength in context.
Use a threshold only when its assumptions and performance criterion match the analysis. Consider partial \(R^2\) and robust diagnostics where appropriate; do not treat one cutoff as universal.

Assess validity separately.
Explain the substantive basis for instrument exogeneity and exclusion. First-stage strength does not verify either assumption.

Check inference under weak identification.
When strength is doubtful, report a weak-instrument-robust test or confidence set, such as Anderson–Rubin or conditional likelihood-ratio inference when its conditions are met. Describe unbounded or disconnected sets rather than replacing them with a convenient finite Wald interval.

Common Pitfalls

“The F statistic is large, so the instrument is valid.” The F statistic concerns first-stage prediction. It does not test exclusion or instrument exogeneity.
“The F statistic exceeds the usual cutoff, so weak identification is settled.” A cutoff depends on the diagnostic, model dimensions, assumptions, and criterion. A robust statistic cannot automatically be interpreted using a threshold derived for a different setting.
“The Wald interval is finite, so the estimate is informative.” A finite interval can rely on an approximation that is poor when the denominator is weakly determined. Inspect weak-IV-robust inference and report its shape honestly.
“An unbounded robust confidence set means the method failed.” It can be a valid indication that the data do not pin down the parameter under weak identification.

Key takeaway: Diagnose instrument strength, instrument validity, and inferential reliability as separate issues. A strong first stage cannot establish validity, no single threshold covers every setting, and conventional Wald intervals should not be trusted automatically when identification is weak.

Check Your Understanding

Use these questions to check whether you can distinguish the three diagnostic issues.

In \(X=\pi Z+v\) and \(Y=\beta X+\delta Z+u\), which parameter controls first-stage strength, and which term violates exclusion when nonzero?
Why can the same first-stage F correspond to different partial \(R^2\) values in different designs?
What features of a Stock–Yogo threshold should be checked before applying it to a reported diagnostic?
Under weak identification, what makes the denominator in the just-identified IV estimator a concern for conventional Wald inference?
In what sense can an unbounded Anderson–Rubin confidence set be informative rather than erroneous?