Auditing an Instrumental Variable Argument

What an Instrumental-Variable Audit Must Decide

A proposed instrumental-variable analysis is not justified merely because an instrument predicts treatment or because the resulting estimate can be computed. The argument depends on several distinct claims: the instrument changes treatment, its assignment is independent of relevant potential outcomes, it affects the outcome only through treatment, and it does not push some individuals toward treatment while pushing others away. An audit asks what supports each claim, what contradicts it, and what causal target would follow if the claims were credible.

This tutorial considers a binary instrument \(Z\), a binary treatment \(D\), and an outcome \(Y\). Code \(Z=1\) as the instrument condition proposed to encourage treatment and \(Z=0\) as the comparison condition. The direction of the coding matters: it determines how the first stage and compliance types are described. The audit is not a test that can certify all assumptions from observed data. Several assumptions concern unobserved potential outcomes and require design knowledge and substantive argument.

The target under the usual binary-instrument assumptions is the local average treatment effect (LATE): the average treatment effect among compliers. These are individuals whose treatment status changes from untreated to treated when the instrument changes from 0 to 1. They are not necessarily representative of all treated individuals, all eligible individuals, or the full study population.

The Assumptions to Audit

For the basic, unconditional binary-instrument argument, state the assumptions before interpreting a ratio of outcome and treatment differences:

• Relevance: the instrument changes treatment probability, so \(P(D=1\mid Z=1)-P(D=1\mid Z=0)\ne 0\).
• Independence: \(Z\) is independent of the potential treatment statuses and potential outcomes: \(Z\perp\{D(0),D(1),Y(0),Y(1)\}\). This is supported most directly by a credible random assignment mechanism, not by a non-significant balance test.
• Exclusion: the instrument has no effect on the outcome except through treatment; formally, \(Y_i(d,1)=Y_i(d,0)\) for each \(d\).
• Monotonicity: \(D_i(1)\ge D_i(0)\) for every individual. With the chosen coding, the instrument does not reduce treatment for anyone. This rules out defiers.
• Consistency and no interference: observed treatment and outcome agree with the potential outcomes corresponding to the instrument and treatment actually received, and one individual's instrument or treatment does not alter another individual's outcome. Treatment and instrument conditions must also be sufficiently well-defined for these potential outcomes to make sense.

The first stage is an observable contrast, but relevance alone does not establish the other assumptions. Independence is a claim about assignment and potential outcomes; exclusion and monotonicity are also generally not identified from the observed distribution without additional assumptions. The earlier material on IV assumptions distinguishes these claims from empirical checks; an audit puts them together for one proposed argument.

What the Wald Contrast Would Mean

A short derivation clarifies why the assumptions are audited jointly. Let the reduced-form outcome difference be \(E[Y\mid Z=1]-E[Y\mid Z=0]\), and let the first-stage difference be \(E[D\mid Z=1]-E[D\mid Z=0]\). Under independence and consistency, comparing instrument groups gives the corresponding differences in potential outcomes averaged over the same population. Under exclusion, the outcome under instrument value \(z\) is \(Y(D(z))\). Therefore, the reduced form is

For always-takers and never-takers, \(D(1)=D(0)\), so their contribution to this expression is zero. Compliers contribute \(Y(1)-Y(0)\), while defiers contribute \(-[Y(1)-Y(0)]\). Under monotonicity there are no defiers, and hence

where \(C\) denotes membership in the complier group. The same type-by-type argument applied to treatment gives

When the first stage is nonzero, dividing the first expression by the second gives the LATE among compliers. This justification shows why a direct effect of \(Z\) on \(Y\) breaks the interpretation: the reduced form would then include an outcome pathway not represented by the treatment contrast. It also shows why monotonicity matters: if defiers exist, the ratio is a signed mixture rather than the average effect for compliers alone.

Audit Each Claim Against the Design

A useful audit records the evidence for and against each assumption separately. Relevance is assessed from treatment rates by instrument group, with uncertainty and the practical size of the difference in view. Independence is assessed first from how the instrument was assigned: randomization, a transparent eligibility rule, or a plausible natural assignment mechanism may support it. Baseline balance checks can reveal problems, but a balanced set of measured covariates cannot establish independence from unmeasured causes or potential outcomes.

For exclusion, draw the plausible pathways from \(Z\) to \(Y\). Ask whether the instrument changes information, resources, access, monitoring, or behavior in a way that could affect the outcome without changing \(D\). A pathway that is plausible in the setting is a substantive threat even if it cannot be tested directly. Placebo outcomes, negative controls, and knowledge of how the instrument was implemented may reveal concerns, but passing such checks does not prove exclusion.

For monotonicity, ask whether the instrument's encouragement could plausibly discourage treatment for some people. A higher offer rate overall does not rule out individual-level defiers. Mechanisms such as displacement, stigma, added burdens, or substitution between services may create opposite responses for some individuals. Evidence about the process can make monotonicity more or less plausible, but observed group-level uptake rates do not identify individual compliance types.

Prioritize the Weakest Link

An audit should not turn into a checklist in which every assumption receives an unqualified “pass.” Summarize what is directly documented, what is only indirectly supported, and what remains uncertain. Then identify the assumption whose failure would most change the causal interpretation. In the clinic example, a plausible direct counseling effect is central. In the distance example, residential selection may be the main concern for independence. In another design, a weak first stage or a credible defier mechanism may be the dominant problem.

This prioritization is not a ranking by p-value. A large first stage cannot compensate for a doubtful exclusion restriction, and good measured covariate balance cannot guarantee independence. Likewise, declaring exclusion the weakest assumption does not prove it fails; it identifies the most important unsupported step in the proposed argument. Conclusions should state the target population and the assumptions on which the interpretation depends.

The main alternatives answer different questions or rely on different assumptions. A randomized treatment comparison directly estimates an effect of treatment assignment when treatment itself is assigned; it does not solve the problem of noncompliance by simply relabeling an encouragement as treatment. Covariate adjustment or propensity score matching can address confounding only to the extent that relevant confounders are measured and appropriately handled. IV methods are useful when treatment is confounded but a credible instrument exists; their price is dependence on independence, exclusion, and often monotonicity, with a complier-specific target rather than an automatic population-average effect.

A common mistake is to say, “The first stage is large, so the instrument is valid.” The first stage speaks to relevance, not to whether the instrument is independent of potential outcomes or has no direct outcome pathway. The correction is to report the relevance evidence and separately make a design-based argument for each remaining assumption, being explicit about the most serious unsupported claim.