Why Passing an Overidentification Test Does Not Validate Instruments

What a Passing J Test Actually Says

With more instruments than endogenous regressors, the overidentifying restrictions provide a way to assess whether the instruments are compatible with the fitted instrumental-variables model. The J test summarizes that compatibility. Its limits matter: not rejecting the restrictions does not establish that the instruments meet the causal assumptions required for valid instrumental-variables inference. A test can fail to detect violations because those violations fit the model's restrictions, because the test has limited power, or because different violations cancel in the contrasts the test examines.

Consider a linear model with one endogenous regressor \(D\), an outcome \(Y\), and \(q\) excluded instruments collected in the vector \(Z\), where \(q>1\). For simplicity, suppose any included controls have already been partialled out. The structural equation is \(Y=\beta D+u\). Instrument validity requires more than relevance: the instruments must satisfy the appropriate independence or moment restrictions, and the exclusion restriction must hold. In this model, a central implication is \(E[Zu]=0\). Relevance requires the instruments to predict \(D\).

Definition: An overidentification test tests whether the extra moment restrictions implied by having more instruments than endogenous regressors are compatible with the fitted IV model. With one endogenous regressor and \(q\) instruments, the usual J test has \(q-1\) degrees of freedom. Under the null restrictions, identification, and suitable regularity conditions, a properly constructed J statistic has an asymptotic chi-square distribution with \(q-1\) degrees of freedom.

The null is a statement about the tested moment restrictions, not a direct test of every part of instrument validity. For the usual large-sample reference distribution to apply, the model must be correctly specified for the test being used, the instruments must identify the endogenous regressor, and the covariance estimate used in the statistic must be appropriate. In particular, robust Hansen J and homoskedasticity-based Sargan tests rely on different covariance assumptions. A numerical p-value is meaningful only with the test's assumptions and construction in view.

Why Some Violations Are Invisible to the Test

Let \(\pi=E[ZD]\), a \(q\)-vector describing the instruments' associations with \(D\), and let \(g=E[Zu]\), the vector of moment violations at the true structural coefficient \(\beta\). For a proposed coefficient \(b\), the population moment vector is

E[Z(Y-bD)] = g-\pi(b-\beta).

Changing the fitted coefficient moves the moment vector in the direction of \(\pi\). The fitting procedure can therefore absorb a component of the violation vector that lies in that direction. The J test concerns the remaining overidentifying discrepancies; it does not separately require that every instrument satisfy every causal assumption.

A short calculation makes the limitation clear. If \(g=c\pi\) for some constant \(c\), then setting \(b=\beta+c\) gives

E[Z(Y-(\beta+c)D)] = c\pi-\pi c=0.

Thus the population moments can all be satisfied at a coefficient different from the causal coefficient. In this case the instruments may be invalid, yet the restrictions checked by the J test are satisfied. The test's failure to reject is not a guarantee of an unbiased causal estimate.

Worked Example: Two Instruments With Aligned Direct Effects

Suppose the hypothetical data-generating process is \(D=Z_1+Z_2+v\) and \(Y=\beta D+0.4Z_1+0.4Z_2+\varepsilon\). Assume \(Z_1\) and \(Z_2\) have mean zero, variance one, and are uncorrelated with each other and with \(v\) and \(\varepsilon\). These assumptions make the instruments relevant, but the direct effects in the outcome equation violate exclusion.

The first-stage association vector is \(\pi=(1,1)'\). At the true coefficient, the moment-violation vector is \(g=(0.4,0.4)'\), because \(E[Z_1u]=0.4\) and \(E[Z_2u]=0.4\). In particular, \(g=0.4\pi\). At \(b=\beta+0.4\), the residual is

Y-(\beta+0.4)D =0.4Z_1+0.4Z_2+\varepsilon-0.4(Z_1+Z_2+v) =\varepsilon-0.4v.

By the stated independence assumptions, both \(E[Z_1(\varepsilon-0.4v)]\) and \(E[Z_2(\varepsilon-0.4v)]\) equal zero. The population overidentifying restriction is therefore satisfied at \(\beta+0.4\), even though both instruments have direct effects and the resulting coefficient is shifted by 0.4 outcome units per unit of \(D\). A J test may pass; that result would not make either instrument valid.

This example illustrates a general distinction. The J test can detect some forms of disagreement among instruments, but it cannot establish the untestable causal story behind them. A set of instruments can agree with one another because they all share a problematic feature. Agreement among estimates is not the same as validity.

Failure to Reject Can Also Reflect Low Power

Even when a violation does create a discrepancy the test could detect, the sample may be too small or the discrepancy too modest for rejection to be likely. Under fixed alternatives detectable by the test, the J statistic can grow with sample size; in a finite sample, however, the observed statistic also varies. A nonsignificant result can therefore occur even when the restrictions are false.

The p-value is calculated under the null model and the test's reference distribution. It is not the probability that the null restrictions are true, nor the probability that the instruments are valid. The significance level \(\alpha\) sets the test's long-run Type I error rate under the null; it does not specify the probability of detecting a particular violation. That probability is the test's power under that alternative.

Worked Example: A Nonsignificant Result With Limited Power

Suppose a hypothetical analysis has two instruments for one endogenous regressor, so the J test has \(q-1=1\) degree of freedom. It reports \(J=1.21\). Under the chi-square reference distribution, the p-value is \(P(\chi^2_1\geq 1.21)\approx 0.271\). At \(\alpha=0.05\), the result is not statistically significant: the tested restrictions are not rejected.

That conclusion is not evidence that the probability the instruments are valid is 72.9%; \(1-0.271\) is not a validity probability. To see why nonrejection may be unsurprising under a violation, suppose, purely illustratively, that under a particular alternative the statistic follows a noncentral chi-square distribution with 1 degree of freedom and noncentrality parameter \(\lambda=1\). Its approximate power at the 5% critical value \(3.841\) is

P(\chi^2_1(1)>3.841) =P(|N(1,1)|>\sqrt{3.841}) \approx P(N(1,1)>1.960)+P(N(1,1)<-1.960) \approx 0.170.

Under this illustrative alternative, the test rejects only about 17% of the time, so it fails to reject about 83% of the time. The calculation does not assert that this particular dataset had that alternative or power; it shows why a nonsignificant sample result alone cannot rule out violations.

Opposing Violations Can Cancel in the Tested Contrasts

The phrase “overidentifying restrictions” can suggest that the test checks each instrument's validity separately. It does not. With one endogenous regressor, fitting the coefficient uses one direction in the \(q\)-dimensional moment vector. The test uses the remaining \(q-1\) directions to assess incompatibility. If violations in different instruments line up with the fitted direction, their signs may differ and still leave no discrepancy in those tested contrasts.

Worked Example: Opposite-Signed Violations That Leave No Overidentifying Discrepancy

Consider the hypothetical model \(D=Z_1-Z_2+v\) and \(Y=\beta D+0.3Z_1-0.3Z_2+\varepsilon\). Let \(Z_1\) and \(Z_2\) be independent, mean-zero, variance-one instruments, uncorrelated with \(v\) and \(\varepsilon\). Each instrument has a direct effect on the outcome, so both violate exclusion. The first-stage association vector is \(\pi=(1,-1)'\), and the moment-violation vector is \(g=(0.3,-0.3)'\). Thus \(g=0.3\pi\), even though the two violations have opposite signs.

There is one overidentifying contrast for two instruments and one endogenous regressor. A contrast in the direction \((1,1)'\) is orthogonal to the first-stage vector, since \((1,1)\pi=0\). The same contrast applied to the violation vector gives

$$ (1,1)g=0.3-0.3=0. $$

The opposite-signed violations cancel in this contrast. Equivalently, at \(b=\beta+0.3\),

Y-(\beta+0.3)D =0.3Z_1-0.3Z_2+\varepsilon-0.3(Z_1-Z_2+v) =\varepsilon-0.3v,

whose moments with both instruments are zero under the stated assumptions. The restriction can pass despite both instruments being invalid, and the fitted coefficient is shifted by 0.3 outcome units per unit of \(D\). The cancellation is a feature of the moments and fitted model, not proof that the direct effects are absent.

Interpreting the Test in Practice

A significant J test is a warning that the tested restrictions do not fit well with the maintained model and assumptions. It does not, by itself, identify which instrument is invalid or which assumption failed. Conversely, a nonsignificant test means only that the data did not provide sufficient evidence to reject those restrictions at the chosen significance level. It does not certify exclusion, independence, or the causal interpretation of the estimate.

The test is most informative as one diagnostic among several. Its interpretation should be considered alongside the substantive rationale for each instrument, the design that generated instrument variation, the first-stage evidence, and the assumptions needed for the causal claim. Those pieces of reasoning address questions the J test cannot settle, including whether instruments share a common source of bias.

Key takeaway: A passing J test is not an instrument-validity certificate. Instruments can be jointly invalid while satisfying the tested moments, a real discrepancy may be hard to detect with limited power, and violations may cancel in overidentifying contrasts. Report nonrejection as nonrejection, not validation.

Common Mistake

Wrong reasoning: “The J test was not significant, so the instruments are valid.” This treats failure to reject as proof of the null and treats a test of moment compatibility as a test of every causal assumption.
Correction: State that the restrictions were not rejected at the specified significance level. Then discuss power and the independent substantive assumptions supporting instrument validity; the J test alone cannot establish them.
Another wrong inference: “A significant J test proves exactly which instrument is invalid.” Rejection indicates incompatibility with the tested restrictions under the maintained setup; it does not uniquely diagnose the cause.

Check Your Understanding

Answer each question by distinguishing what the J test examines from what instrument validity requires.

With one endogenous regressor and three instruments, what is the usual degrees of freedom for the J test?
Why does a p-value above 0.05 not mean there is more than a 95% probability that the instruments are valid?
In the aligned-violation examples, why can the fitted coefficient shift while the population overidentifying moments equal zero?
What does low power imply about the interpretation of a nonsignificant J statistic?
If a J test is significant, does it identify which instrument or assumption is responsible? Explain.