Overidentification Tests Under Treatment-Effect Heterogeneity

Why Valid Instruments Can Disagree

With one endogenous treatment and more than one instrument, a common-coefficient instrumental-variables model imposes more moment restrictions than are needed to estimate its coefficient. An overidentification test examines whether those restrictions fit the data. It is tempting to interpret rejection as evidence that at least one instrument is invalid. But the restrictions can also fail when treatment effects vary across people: different instruments may shift treatment for different groups, whose average effects need not agree.

This distinction matters for causal interpretation. Independence and exclusion concern how an instrument is assigned and how it affects outcomes. A common-coefficient model makes an additional claim: all instrument-induced outcome changes are compatible with the same treatment-effect coefficient. If that claim fails, a test based on the common coefficient may reject even though the instruments satisfy independence and exclusion.

Moment Restrictions and the J Test

Definition: Let $D$ be a scalar treatment, $Y$ an outcome, and $Z=(Z_1,\ldots,Z_q)$ a vector of instruments. The constant-coefficient linear IV model with an intercept imposes the centered population moment restrictions $$ E[(Z_j-EZ_j)(Y-\beta D)]=0,\qquad j=1,\ldots,q, $$ for a single coefficient $\beta$. If there are $q$ linearly independent instrument moments and one coefficient, the model is overidentified when $q>1$. An overidentification test assesses the joint fit of the extra restrictions.

In sample, let $\bar g(\hat\beta)$ be the vector of estimated moment discrepancies evaluated at the IV estimate $\hat\beta$, and let $\hat\Omega$ estimate the covariance matrix of the scaled moment vector. A robust Hansen J statistic has the form

J=n\,\bar g(\hat\beta)^\mathsf{T}\hat\Omega^{-1}\bar g(\hat\beta).

Under the null that the moment restrictions hold, with appropriate regularity conditions, valid identification, and a suitable covariance estimate, $J$ is approximately distributed as $\chi^2_{q-k}$, where $k$ is the number of estimated endogenous-regressor coefficients. Here $q=2$ and $k=1$, so the reference distribution has one degree of freedom. The Sargan statistic is a related version under homoskedasticity; the covariance estimator and resulting test must match the assumptions being used.

The null is a joint statistical restriction on the moments for a common coefficient. It is not simply the statement that every instrument is independent of potential outcomes and satisfies exclusion. Under a constant treatment effect, those causal assumptions can imply the common moments. Under heterogeneous effects, independence and exclusion alone need not do so.

How Heterogeneity Changes the Moments

For a binary instrument $Z_j$, define its marginal outcome contrast as $E(Y\mid Z_j=1)-E(Y\mid Z_j=0)$, and define its marginal treatment contrast similarly. The identity

\operatorname{Cov}(Z_j,Y-\beta D) =P(Z_j=1)P(Z_j=0) \left\{ \Delta_jY-\beta\Delta_jD \right\}

holds when $\Delta_jY=E(Y\mid Z_j=1)-E(Y\mid Z_j=0)$ and $\Delta_jD=E(D\mid Z_j=1)-E(D\mid Z_j=0)$. It follows by writing the covariance of a binary variable with any outcome as its success probability times its failure probability times the difference in conditional means. If $\Delta_jD\ne0$, that instrument’s marginal moment is zero at its Wald ratio $\Delta_jY/\Delta_jD$.

The causal interpretation of this ratio requires additional assumptions. For a relevant binary instrument and a binary treatment $D$, independence of assignment from potential treatment and outcome, exclusion (the instrument affects $Y$ only through $D$), and monotonicity (no units move against the instrument’s direction) identify a local average treatment effect among compliers: units whose treatment changes with that instrument. With several instruments, the complier groups need not be the same. If their average treatment effects differ, the instruments’ Wald ratios can differ while each instrument satisfies independence and exclusion.

A short implication follows directly. If two relevant binary instruments have Wald ratios $\beta_1\ne\beta_2$, then the first moment requires $\beta=\beta_1$ and the second requires $\beta=\beta_2$. No single $\beta$ can satisfy both population moments. An overidentification test can therefore reject the common-coefficient restrictions. This is a model incompatibility, not by itself proof of a causal-validity violation.

Worked Example: Two Valid Instruments and Different Complier Effects

Consider a hypothetical population with independent binary instruments $Z_1$ and $Z_2$, each having probability $0.5$ of being one. Let $U$ be uniform on $[0,1]$, independent of both instruments, and define treatment by

D=1\{U\leq 0.25+0.25Z_1+0.125Z_2\}.

Set $Y=Y_0+\tau(U)D$, with $Y_0=0$, and let $\tau(U)=8$ on $[0.25,0.375)$, $\tau(U)=-4$ on $[0.375,0.5)$, and $\tau(U)=8$ on $[0.5,0.625)$. The values of $\tau$ elsewhere do not affect either instrument’s treatment contrast.

Check the causal assumptions. The instruments are independent of $U$, which determines treatment response and treatment effects. The outcome depends on the instruments only through $D$, so exclusion holds. Increasing either instrument never decreases $D$, so monotonicity holds for each instrument. Both are relevant, as the first-stage contrasts below are positive.

For $Z_1$. At either value of $Z_2$, changing $Z_1$ from zero to one raises the treatment threshold by $0.25$; hence $\Delta_1D=0.25$. When $Z_2=0$, the induced units have $U$ in $[0.25,0.5)$; their aggregate treatment-effect contribution is $0.125(8)+0.125(-4)=0.5$. When $Z_2=1$, the interval is $[0.375,0.625)$; its contribution is $0.125(-4)+0.125(8)=0.5$. Averaging over the two equally likely values of $Z_2$ gives $\Delta_1Y=0.5$. Thus the $Z_1$ Wald ratio is $0.5/0.25=2$.

For $Z_2$. At either value of $Z_1$, changing $Z_2$ raises the threshold by $0.125$, so $\Delta_2D=0.125$. When $Z_1=0$, the induced interval is $[0.25,0.375)$, with outcome contribution $0.125(8)=1$. When $Z_1=1$, it is $[0.5,0.625)$, also with contribution $0.125(8)=1$. Therefore $\Delta_2Y=1$, and this instrument’s Wald ratio is $1/0.125=8$.

Both instruments satisfy the stated causal assumptions, yet their complier-specific effects are 2 and 8 outcome units per treatment unit. The common-coefficient moment restrictions cannot both hold. The negative effect in the middle interval helps make the first instrument’s average differ; heterogeneity does not require every subgroup effect to have the same sign.

Checking the Moment Arithmetic

Worked Example: Covariance Moments Require Different Coefficients

Suppose a hypothetical population has $P(Z_1=1)=P(Z_2=1)=0.5$. Its marginal treatment and outcome contrasts are $\Delta_1D=0.25$, $\Delta_1Y=0.5$, $\Delta_2D=0.125$, and $\Delta_2Y=1$, as in the preceding construction. Compute the centered covariance moments.

For either instrument, $P(Z_j=1)P(Z_j=0)=0.5(0.5)=0.25$. Therefore

\begin{aligned} \operatorname{Cov}(Z_1,D)&=0.25(0.25)=0.0625,\\ \operatorname{Cov}(Z_1,Y)&=0.25(0.5)=0.125,\\ \operatorname{Cov}(Z_2,D)&=0.25(0.125)=0.03125,\\ \operatorname{Cov}(Z_2,Y)&=0.25(1)=0.25. \end{aligned}

The first moment is $0.125-0.0625\beta$, which is zero at $\beta=2$. The second is $0.25-0.03125\beta$, which is zero at $\beta=8$. These are exactly the two marginal Wald ratios: multiplying a contrast by the same positive factor $P(Z_j=1)P(Z_j=0)$ does not change the value of $\beta$ that sets its moment to zero. In particular, $\operatorname{Cov}(Z_2,Y)=0.25$, not $0.125$; using the wrong covariance would give an incorrect second moment and ratio.

This calculation concerns each centered binary-instrument moment individually. With multiple instruments, an estimator combines the moments according to its weighting scheme, but that combination cannot make two incompatible population zero restrictions simultaneously true.

What a Rejection Says—and Does Not Say

A J-test rejection is evidence against the joint moment restrictions used by the fitted IV model, at the chosen significance level and under the test’s approximation. Possible explanations include a violation of independence or exclusion, an incorrect functional form, treatment-effect heterogeneity that makes the common coefficient moments incompatible, or other misspecification. The test does not distinguish these explanations, nor does it identify which instrument is responsible.

Conversely, failure to reject does not establish instrument validity or rule out heterogeneity. The test may have limited power, and heterogeneous effects can still produce compatible moments—for example, if the relevant instrument-induced groups happen to have the same average effect. A non-rejection is not proof that the instruments are valid or that the treatment effect is constant.

Worked Example: Interpreting a Hypothetical J Statistic

Suppose a hypothetical analysis uses two instruments for one endogenous treatment, estimates a heteroskedasticity-robust covariance matrix for the moments, and obtains a J statistic of $14.44$. The overidentifying degrees of freedom are $q-k=2-1=1$. The upper-tail probability is

P(\chi^2_1\geq 14.44)\approx 0.0001447.

At $\alpha=0.05$, the statistic exceeds the one-degree-of-freedom critical value of approximately $3.841$, and the p-value is below $0.05$. The analysis rejects the joint common-coefficient moment restrictions. In a setting where independence and exclusion are credible, heterogeneous effects across instrument-specific complier groups are one possible explanation. The calculation alone does not establish that explanation or show that either instrument is invalid.

A Practical Reading of the Test

Before interpreting an overidentification test, identify the restriction being tested: is the fitted model asking several instruments to support one coefficient? Then consider whether the instruments may shift treatment for different populations. If so, different local effects can produce systematic moment disagreement without violating independence or exclusion. This causal possibility complements, rather than replaces, attention to relevance, finite-sample behavior, and the assumptions behind the test’s reference distribution.

Rejection: the joint moment restrictions are inconsistent with the data at the test’s resolution; the reason is not identified by J alone.
Non-rejection: the data do not provide sufficient evidence against those restrictions; this does not certify validity.
Heterogeneity: it can generate disagreement across instruments, but it does not guarantee rejection in every population.

A common mistake is to reason, “The J test rejects, so at least one instrument must violate exclusion.” That conclusion is too strong. The correction is to say that the test rejects the joint moments of the specified common-coefficient model. Instrument invalidity is one possible cause; heterogeneous instrument-specific effects can also make those moments incompatible even when independence and exclusion hold.

Key takeaway: Overidentification tests assess whether the instruments’ moments support a common coefficient. Under treatment-effect heterogeneity, valid instruments can identify different complier-specific effects, so rejection is not synonymous with invalid instruments—and non-rejection is not validation.

Check Your Understanding

Answer each question using the assumptions and distinctions developed above.

With two relevant binary instruments, why can their population Wald ratios differ even if both satisfy independence and exclusion?
For a binary instrument with probability $0.4$ of being one, what factor multiplies its outcome contrast in the covariance with $Y$?
If two instrument moments require coefficients 3 and 6 respectively, can a single constant-coefficient model satisfy both population restrictions?
What does a J-test rejection establish, and what does it leave unresolved?
Can heterogeneous treatment effects exist without causing a J-test rejection? Explain briefly.