Why Relevance Does Not Establish Instrument Validity

Relevance Is One Requirement, Not a Validity Test

An instrument that strongly predicts treatment may seem persuasive: people receiving one level of the instrument are much more likely to receive treatment than people receiving the other. That pattern is important, because without a treatment contrast induced by the instrument, an instrumental-variable comparison has no leverage. But a strong first stage answers only whether the instrument and treatment are associated. It does not establish why they are associated, whether the instrument also affects the outcome through another route, or whether treatment responses move in the required direction for every individual.

For a binary instrument \(Z\) and binary treatment \(D\), relevance is commonly summarized by the first-stage contrast \(E[D\mid Z=1]-E[D\mid Z=0]\). The contrast can be large even when other conditions needed for an instrumental-variable interpretation fail. The counterexamples below hold relevance strong while violating, in turn, independence, exclusion, and monotonicity.

The independence condition says, in effect, that instrument assignment does not sort individuals according to their potential treatments or outcomes. Exclusion says that changing the instrument cannot change the outcome except by changing treatment. Monotonicity rules out individuals whose treatment would move in the opposite direction from the instrument. These are claims about assignment mechanisms, causal pathways, or individual responses—not merely about observed differences in treatment rates.

A Strong First Stage Can Accompany Dependence

In an observational setting, an instrument may be strongly associated with treatment because both are related to a third variable. The resulting association says little by itself about whether the instrument is independent of potential outcomes. The first example makes this distinction explicit.

In this construction, exclusion and monotonicity can hold: the outcome has no direct dependence on \(Z\), and \(D(1)=1\ge 0=D(0)\) for everyone. Failure of independence alone is enough to undermine the causal interpretation. A strong observed treatment-rate contrast cannot reveal whether assignment was as-if random with respect to potential outcomes.

A Strong First Stage Can Coexist With an Outcome Pathway

Even if instrument assignment is randomized and treatment responds monotonically, the instrument may affect the outcome through a route other than the treatment under study. That possibility is what exclusion rules out. Relevance does not assess it.

Randomization addresses independence; it does not guarantee exclusion. An instrument can be assigned independently and still change the outcome through, for example, additional services, attention, or behavior that is not represented by the treatment definition. Whether such pathways are plausible requires substantive knowledge of how the instrument operates, not inspection of the first-stage contrast alone.

A Strong First Stage Can Hide Defiers

Monotonicity is an individual-level condition. A positive, even substantial, average treatment-rate difference does not show that every individual responds in the same direction. Some individuals may be compliers, with \(D(1)=1\) and \(D(0)=0\); others may be defiers, with \(D(1)=0\) and \(D(0)=1\). Their responses can coexist with a strong positive first stage if compliers are more numerous.

Why the Conditions Must Be Assessed Separately

The algebra clarifies the role of monotonicity in the last example. With a randomized binary instrument and exclusion, the first-stage contrast is the proportion of compliers minus the proportion of defiers. The outcome contrast is the complier share times their average treatment effect minus the defier share times their average treatment effect. If the first-stage contrast is nonzero, their ratio is

where \(C\) denotes the complier group and \(F\) the defier group. This expression follows because assignment changes treatment from 0 to 1 for compliers and from 1 to 0 for defiers; never-takers and always-takers contribute no assignment-induced treatment change. Under monotonicity, \(P(F)=0\), leaving the complier average treatment effect. Without it, a nonzero denominator alone does not give that interpretation. The example's contrast is \((0.70(1)-0.20(4))/(0.70-0.20)=-0.20\), as calculated.

Relevance is observable from treatment rates across instrument levels, but it is only one part of an instrumental-variable argument. Independence concerns how the instrument is assigned relative to potential outcomes and treatments. Exclusion concerns causal pathways from the instrument to the outcome. Monotonicity concerns the joint potential treatment responses of individuals, which are not both observed for any one person. A large first-stage contrast does not directly verify any of these claims.

Common Mistake

A common mistake is to reason, “The treatment rate differs sharply by instrument level, so the instrument is valid.” The first clause establishes relevance, not validity. The first example has a perfect first stage but a confounded instrument; the second has a perfect first stage but a direct outcome effect; the third has a substantial first stage but defiers. A more defensible conclusion is that the instrument is relevant, while independence, exclusion, and monotonicity require their own design-based or substantive justification. A weak first stage can be a problem, but a strong first stage cannot certify the other assumptions.