Why Agreement Across Instruments Does Not Prove Validity

Why Agreement Is Not a Validity Test

When two instruments produce similar instrumental-variable estimates, the agreement can feel reassuring: perhaps both are isolating the same causal effect. But an instrument’s validity depends on assumptions about how it relates to potential treatment and outcome values. Similar estimates do not verify those assumptions. In particular, the instruments can move treatment for different people whose treatment effects happen to average to the same value; violations of exclusion can offset differences in those effects; and instruments related to unobserved outcome determinants can produce matching biased estimates.

The relevant comparison is usually between estimates based on separate binary-instrument contrasts. Each estimate is a ratio of an outcome contrast to a treatment contrast. To interpret that ratio as a causal effect for compliers, one needs assumptions, not merely agreement with another ratio.

What a Binary-Instrument Wald Ratio Identifies

For one binary instrument $Z$ and a binary treatment $D$, let $D(z)$ be the treatment that would be received if the instrument were set to $z$, and let $Y(d)$ be the outcome that would be observed if treatment were set to $d$. The observed values obey consistency: $D=D(Z)$ and $Y=Y(D)$. Assume there is no interference between units and that these potential outcomes are well-defined.

Definition: The Wald ratio is the difference in mean outcomes across instrument values divided by the difference in mean treatments: $$ \frac{E[Y\mid Z=1]-E[Y\mid Z=0]} {E[D\mid Z=1]-E[D\mid Z=0]}. $$ Its causal interpretation as a complier average treatment effect requires, among other conditions, a binary treatment, independence of $Z$ from the potential treatments and outcomes, exclusion, monotonicity, and a nonzero first stage.

Independence means $Z$ is independent of $\{D(0),D(1),Y(0),Y(1)\}$. Exclusion means the instrument affects the outcome only through treatment, so the outcome potential outcomes do not need an instrument argument. Monotonicity requires $D(1)\geq D(0)$ for every unit; it rules out defiers, who would take treatment when $Z=0$ but not when $Z=1$. Relevance requires $E[D\mid Z=1]-E[D\mid Z=0]\ne 0$. The binary-treatment condition matters: the standard Wald-to-complier-effect result is not generally the same for a treatment that can take more than two values.

Under monotonicity, units fall into three principal strata: compliers have $D(1)=1,D(0)=0$; always-takers have $D(1)=D(0)=1$; and never-takers have $D(1)=D(0)=0$. Under independence of $Z$ from the potential treatments, the first stage equals the complier share. Under independence and exclusion, the outcome contrast is the complier share multiplied by the average treatment effect among compliers. Dividing by the first stage gives the complier average treatment effect (CATE in the IV sense; here it means the average treatment effect for compliers, not a conditional average treatment effect): $$ \frac{E[Y\mid Z=1]-E[Y\mid Z=0]} {E[D\mid Z=1]-E[D\mid Z=0]} = E[Y(1)-Y(0)\mid D(1)>D(0)]. $$

With multiple instruments, each instrument can induce a different set of compliers. Even when every instrument is valid, its Wald ratio generally targets the treatment-effect average for its own complier group, not a universal effect for the whole population.

Worked Example: Valid Instruments, Different Compliers, Same Estimate

Consider a hypothetical population divided equally among four groups, A, B, C, and D, each comprising 25% of the population. Two randomized binary encouragements, $Z_1$ and $Z_2$, are considered in turn, with the other encouragement held at zero. When $Z_1$ changes from zero to one, groups A and B take treatment and groups C and D do not. When $Z_2$ changes from zero to one, groups C and D take treatment and groups A and B do not. Thus each instrument has a first stage of 0.50. Suppose both instruments are independent of potential outcomes and affect outcomes only through treatment, and that treatment effects are constant within each group: 1 outcome unit in A, 3 in B, 0 in C, and 4 in D.

For $Z_1$, the compliers are A and B. They have equal shares, so their average treatment effect is $$ \frac{1+3}{2}=2. $$ The outcome contrast is the complier share times their average effect, $0.50\times2=1.00$; dividing by the first stage gives the Wald ratio $1.00/0.50=2.00$.

For $Z_2$, the compliers are C and D, also in equal shares. Their average effect is $$ \frac{0+4}{2}=2. $$ Their outcome contrast is $0.50\times2=1.00$, and their Wald ratio is again $1.00/0.50=2.00$. The estimates agree exactly, although the treatment effects vary substantially across groups and the two instruments move treatment for different people. Agreement here does not show that the instruments identify an effect for the same population.

Offsetting Violations of Exclusion

Exclusion can fail if changing the instrument affects the outcome in a way not mediated by treatment. Examples include an encouragement that changes access to another service or an eligibility rule that directly changes resources. A useful way to see how such a failure affects the Wald ratio is to allow the potential outcome to depend on both treatment and instrument, writing $Y(d,z)$. For this calculation, suppose consistency holds, the instrument is independent of the potential treatments and outcomes, treatment and instrument are binary, monotonicity holds, and the first stage is positive. Do not assume exclusion.

Let $D(z)$ denote potential treatment, $p_C=P(D(1)>D(0))$ the complier share, and $D_1=D(1)$. By independence and consistency, the outcome contrast is $E[Y(D(1),1)-Y(D(0),0)]$. Add and subtract $Y(D(1),0)$. The remaining treatment contrast is zero for always-takers and never-takers, and equals $Y(1,0)-Y(0,0)$ for compliers. The first stage equals $p_C$. Therefore,

\begin{aligned} \frac{E[Y\mid Z=1]-E[Y\mid Z=0]} {E[D\mid Z=1]-E[D\mid Z=0]} &= E[Y(1,0)-Y(0,0)\mid D(1)>D(0)]\\ &\quad+\frac{E[Y(D(1),1)-Y(D(1),0)]}{p_C}. \end{aligned}

The first term is a complier treatment effect at instrument value zero. The second term is the average direct effect of changing the instrument while holding treatment at its value under $Z=1$, divided by the complier share. With exclusion, that direct-effect term is zero. Without exclusion, it can shift the ratio, and its impact is larger when the complier share is smaller. Direct effects can differ across instruments, and so can their contributions to the respective Wald ratios.

Worked Example: Direct Effects Offset Different Complier Effects

Suppose two hypothetical instruments each have a first stage of 0.50 and are independent of the potential treatments and outcomes. Instrument 1 has a complier treatment effect of 2.5 outcome units; instrument 2 has a complier treatment effect of 3.5 units. Suppose instrument 1 also has a direct effect of $+0.25$ units for every unit, while instrument 2 has a direct effect of $-0.25$ units for every unit. These direct effects violate exclusion. For each instrument, take the direct effect at the treatment value under $Z=1$ to be the stated constant.

For instrument 1, the reduced-form outcome contrast is $$ 0.50(2.5)+0.25=1.25+0.25=1.50. $$ Its Wald ratio is $1.50/0.50=3.00$.

For instrument 2, the reduced form is $$ 0.50(3.5)-0.25=1.75-0.25=1.50. $$ Its Wald ratio is also $1.50/0.50=3.00$. The matching estimates conceal different complier treatment effects and nonzero, oppositely directed direct effects. Thus, agreement is compatible with exclusion violations that cancel differences in the causal components.

Agreement Cannot Establish Independence Either

Instrument independence is a separate condition: the instrument must be unrelated to the potential treatments and outcomes. A similar pair of estimates does not show that condition holds. If an instrument is associated with potential outcomes, the outcome contrast can include a difference in baseline outcome risk as well as any effect transmitted through treatment.

Worked Example: Matching Ratios With Instrument–Outcome Dependence

Consider two hypothetical binary-instrument comparisons. For each comparison, everyone is a complier, so $D(z)=z$ and the first stage is $1.00$. Let the outcome obey $Y(d)=2d+U$, where $U$ is an unobserved outcome determinant. Exclusion holds because $Y(d)$ has no direct dependence on the instrument. However, suppose each instrument is associated with $U$: its group with $Z=1$ has mean $U=1$, and its group with $Z=0$ has mean $U=0$. This violates instrument independence.

For each instrument, the outcome contrast is $$ \{2(1)+1\}-\{2(0)+0\}=3, $$ and the treatment contrast is $1-0=1$. Each Wald ratio is $3/1=3.00$. Yet the treatment effect for every unit is $Y(1)-Y(0)=2$ units. Both ratios are biased upward by the one-unit difference in mean $U$, and their agreement does not expose the shared independence violation.

How to Use Cross-Instrument Comparisons

A material difference between estimates can be a warning: it may reflect different complier groups, heterogeneous effects, different violations of assumptions, or sampling uncertainty. Similar estimates are less diagnostic. They can result from equal average effects for different complier groups, from offsetting direct effects, from shared dependence on outcome determinants, or from combinations of these mechanisms.

A comparison is most informative when it is tied to a clearly stated target and explicit assumptions. Ask which units each instrument moves into treatment, whether those units plausibly have different treatment effects, and whether the instruments could affect outcomes through other channels or be associated with potential outcomes. These are substantive questions about the design; matching point estimates cannot answer them by itself.

An important practical distinction is between evidence that estimates are numerically similar and evidence that the assumptions hold. Similarity can be reported as a descriptive diagnostic, with uncertainty taken into account, but it is not a test that establishes independence or exclusion. Conversely, different estimates do not by themselves prove invalidity: valid instruments can identify different complier effects when effects are heterogeneous.

Common Mistake

Common mistake: “Two instruments give nearly the same estimate, so both must be valid and identify the same causal effect.”

Correction: A Wald ratio has a complier-effect interpretation only under stated assumptions, including binary treatment, independence, exclusion, monotonicity, and relevance. Even if both instruments satisfy those assumptions, their complier groups may differ. If assumptions fail, biases can also offset. Agreement is not proof of validity or of a shared target population.

Key takeaway: With a binary treatment and valid instrument assumptions, each Wald ratio identifies an average treatment effect for that instrument’s compliers. Similar ratios do not establish independence or exclusion, and they do not imply that the instruments move treatment for the same people.

Check Your Understanding

Answer using the assumptions and examples developed above.

Which assumptions, in addition to relevance, are needed for a binary-instrument Wald ratio to identify a complier average treatment effect?
Two valid instruments have different complier groups but equal average treatment effects for their respective compliers. What can you conclude from matching Wald ratios about the population effects?
In the direct-effect formula, why can a given average direct effect have a larger impact on the Wald ratio when the complier share is small?
In the example with first stage 0.50 and treatment effects 2.5 and 3.5, verify how direct effects of $+0.25$ and $-0.25$ produce equal Wald ratios.
Can matching estimates establish instrument independence if both instruments are associated with an unobserved outcome determinant? Explain.