Assessing A Natural Experiment Instrument In Practice

From a Policy Change to an Instrument

A policy change can create differences in treatment access that resemble an experiment. For example, a new eligibility rule may increase participation in a health program for some people but not others. The policy is a plausible instrument only if its assignment changes treatment uptake and the assumptions connecting policy assignment to potential outcomes are credible. Comparing outcomes across policy groups is informative, but the comparison does not validate those assumptions by itself.

This tutorial uses \(Z\) for a binary instrument, \(D\) for a binary treatment, and \(Y\) for an outcome. Let \(Z=1\) indicate exposure to the policy or eligibility rule, and \(Z=0\) indicate no exposure. Let \(D=1\) mean receipt of treatment. The potential treatment \(D(z)\) is the treatment a person would receive under instrument value \(z\); the potential outcome \(Y(d)\) is the outcome under treatment value \(d\).

Definition: A binary instrument is relevant when treatment uptake differs by instrument assignment, so \(E[D\mid Z=1]-E[D\mid Z=0]\ne 0\). The first-stage contrast is this difference in treatment uptake. The reduced-form contrast is \(E[Y\mid Z=1]-E[Y\mid Z=0]\), the difference in the outcome by instrument assignment.

A natural experiment is not automatically a randomized experiment. The instrument’s origin, the way people or places became exposed, and other simultaneous policy changes all matter. A sound assessment combines observed evidence with a substantive account of how assignment occurred.

Assumptions Behind the Unadjusted Wald Estimate

For a binary instrument and treatment, the unadjusted Wald estimate is the reduced-form contrast divided by the first-stage contrast. Its familiar causal interpretation requires more than a nonzero first stage. The following assumptions state one common sufficient set for interpreting it as the average treatment effect among compliers.

Consistency and no interference: each person’s observed treatment and outcome agree with the relevant potential values, and one person’s instrument or treatment does not change another person’s outcome.
Unconditional independence: \(Z\) is independent of the joint set of potential values \(\{D(0),D(1),Y(0),Y(1)\}\) in the target population. This is a population-level assumption, not just a claim that a few measured covariates look balanced.
Exclusion: the instrument affects the outcome only through treatment. In potential-outcome notation, the outcome under instrument value \(z\) and treatment \(d\) is \(Y(d)\), with no additional dependence on \(z\).
Relevance: instrument assignment changes treatment uptake in the population.
Monotonicity: \(D(1)\ge D(0)\) for every person. In this setting, policy exposure does not cause anyone who would otherwise take treatment to stop taking it.

Theorem: Under these assumptions, the population Wald ratio identifies the complier average causal effect, or CACE: the average treatment effect among people with \(D(1)=1\) and \(D(0)=0\), who take treatment when exposed to the instrument and do not take it otherwise. The result does not generally identify the average treatment effect for the whole population.

The derivation clarifies why each assumption matters. Independence allows instrument-group means to be expressed as means over the same distribution of potential values. Consistency and exclusion give the observed outcome as \(Y(D(Z))\). For the denominator, monotonicity means \(D(1)-D(0)\) equals 1 for compliers and 0 for everyone else. Thus:

\begin{aligned} E[D\mid Z=1]-E[D\mid Z=0] &=E[D(1)-D(0)]\\ &=P\{D(1)=1,D(0)=0\}. \end{aligned}

For the numerator, always-takers and never-takers receive the same treatment at either instrument value, so their potential outcomes cancel in the contrast. Only compliers contribute:

\begin{aligned} E[Y\mid Z=1]-E[Y\mid Z=0] &=E[Y(D(1))-Y(D(0))]\\ &=P(C)\,E[Y(1)-Y(0)\mid C], \end{aligned}

where \(C\) denotes membership in the complier group. Dividing the numerator by the positive first stage \(P(C)\) gives \(E[Y(1)-Y(0)\mid C]\), the CACE. This derivation also shows why an unadjusted marginal Wald ratio needs unconditional independence. If independence is established only conditional on baseline covariates \(X\), the marginal contrasts can mix causal effects with differences in the distribution of \(X\).

A Practical Assessment Sequence

A disciplined assessment first makes the policy comparison precise, then examines what is observable, and finally argues about assumptions that data cannot settle. The checks below diagnose concerns; none is a mechanical test of instrument validity.

Specify the design.
Define the population, policy exposure \(Z\), treatment \(D\), outcome \(Y\), time ordering, and unit of assignment. Clarify which people or places could plausibly have received either instrument value.

Check relevance.
Report treatment uptake by instrument group and their difference. Check that the direction and timing fit the proposed mechanism, and use appropriate uncertainty estimates rather than relying only on the observed point estimate.

Describe baseline patterns.
Compare variables measured before policy exposure. Large differences may point to selection or distinct settings; small differences are reassuring only about the measured variables and do not prove independence.

Inspect the outcome contrast.
Report the reduced-form difference by instrument group, with units and uncertainty. Check whether the timing and direction are compatible with the proposed treatment mechanism, while remembering that other policy pathways can produce the same contrast.

Make the causal case.
Explain why assignment is plausibly independent of potential outcomes, why there are no direct or alternative pathways to the outcome, and why monotonicity is reasonable. State remaining threats and the population to which the IV effect applies.

Worked Policy Example

Worked Example: A New Eligibility Rule and Program Uptake

Hypothetical data: A region introduces an eligibility rule that gives some adults easier access to a support program. The proposed instrument is eligibility under the rule (\(Z=1\)); treatment is program participation (\(D=1\)); the outcome is a symptom score at 12 months, with lower scores indicating fewer symptoms. In a hypothetical cohort of 1,000 adults, 500 are in each policy group. The mean symptom score is 38.0 for the exposed group and 42.0 for the unexposed group. Program uptake is 300 of 500 exposed adults and 200 of 500 unexposed adults.

The first-stage uptake rates are \(300/500=0.600\) and \(200/500=0.400\). Therefore the first-stage contrast is \(0.600-0.400=0.200\), or a 20-percentage-point increase in uptake under exposure. The reduced-form contrast is \(38.0-42.0=-4.0\) symptom-score points.

\widehat{\text{Wald}} =\frac{\text{reduced-form contrast}}{\text{first-stage contrast}} =\frac{-4.0\text{ points}}{0.200} =-20.0\text{ symptom-score points}.

If the assumptions for the unadjusted Wald result are credible, this estimate is the CACE: among adults whose participation is changed by the eligibility rule, program participation lowers the mean 12-month symptom score by an estimated 20.0 points. It is not an estimate for every adult. The uptake difference supports relevance, but the point estimates do not establish independence, exclusion, or monotonicity. The policy history must show, for example, that eligibility was not targeted to places already on different outcome trajectories and that the rule did not also change access to other services affecting symptoms.

Worked Example: Quantifying Baseline Balance

Suppose the same hypothetical comparison reports baseline age with mean 51 years and standard deviation 12 years in the exposed group, and mean 50 years and standard deviation 12 years in the unexposed group. A standardized mean difference (SMD) expresses the difference in means in pooled-standard-deviation units. With equal group sizes here, the pooled standard deviation is 12 years.

\text{SMD} =\frac{51-50}{12} =0.0833.

The exposed group is older by one year, a difference of 0.0833 standard deviations. This is a small descriptive imbalance in age, not proof that the instrument is independent. Suppose instead that a binary baseline high-risk indicator is present for 30% of the exposed group and 22% of the unexposed group. Using the pooled proportion \(p_{\text{pool}}=(0.30+0.22)/2=0.26\), its standardized difference is:

\text{SMD} =\frac{0.30-0.22}{\sqrt{0.26(1-0.26)}} =\frac{0.08}{0.4386} =0.182.

That difference is more concerning than the age difference and warrants investigation into how exposure was assigned. It still does not tell us whether the imbalance creates outcome bias: that depends on the variable’s relationship to potential outcomes and on the assignment process. Baseline balance is best reported across substantively important covariates, with attention to the policy mechanism, rather than reduced to a pass-or-fail significance test.

Why Conditional Independence Is Not Enough for the Marginal Ratio

Worked Example: Covariate Composition Can Reverse the Wald Contrast

Consider a hypothetical population with a binary baseline risk group \(X\). Without treatment, the outcome mean is 10 points for the low-risk group and 20 points for the high-risk group. Treatment lowers the outcome by 2 points in either group. Within each \(X\) group, suppose policy exposure is independent of all potential treatment and outcome values, and raises uptake from 0.20 to 0.60. Thus the within-group first stage is \(0.60-0.20=0.40\), and the within-group Wald ratio is \(-0.80/0.40=-2.00\) points.

Now suppose 80% of the exposed group is high-risk, while only 20% of the unexposed group is high-risk. The untreated baseline outcome mean is \(0.20(10)+0.80(20)=18\) in the exposed group and \(0.80(10)+0.20(20)=12\) in the unexposed group. Given uptake rates of 0.60 and 0.20, respectively, the observed outcome means are \(18-0.60(2)=16.8\) and \(12-0.20(2)=11.6\).

\text{Marginal first stage}=0.60-0.20=0.40,\qquad \text{marginal reduced form}=16.8-11.6=5.20, \qquad \text{marginal Wald}=\frac{5.20}{0.40}=13.0\text{ points}.

The marginal ratio is \(+13.0\) points even though treatment lowers the outcome by 2 points in both groups. The problem is that the instrument groups have different covariate composition. Independence holds within \(X\), not unconditionally. An appropriately specified covariate-adjusted IV analysis may be justified under conditional independence and additional conditions, but its target and weighting depend on that analysis and its assumptions. Conditional independence alone does not make this unadjusted marginal ratio a CACE.

Substantive Threats, Diagnostics, and Alternatives

Evidence should be interpreted in relation to the proposed assignment mechanism. Baseline covariates are useful diagnostics: imbalance in strong predictors of the outcome can undermine a claim that instrument groups are comparable, while apparent balance cannot rule out unmeasured differences. A pre-policy outcome history, when available, can reveal divergent trends or anticipation, but similar observed trends do not prove the exclusion restriction.

Exclusion is especially demanding for policy instruments. A policy may alter resources, information, provider behavior, or access to other services in addition to changing the treatment being studied. Any such route from \(Z\) to \(Y\) outside \(D\) violates exclusion. Independence can fail if policy adoption responds to local need, political pressure, or expected outcomes. Relevance can be weak if the policy barely changes uptake; an imprecise first stage makes ratio estimates unstable. Monotonicity is questionable if the same policy encourages some people but discourages others from taking treatment.

Compare IV with the main alternatives in terms of their assumptions, not just their numerical estimates. Propensity score matching and regression adjustment address measured confounding under their own identification assumptions; they do not automatically solve unmeasured confounding. IV can address certain forms of confounding only when the instrument assumptions hold, and generally identifies an effect for compliers rather than a population-wide average effect. A randomized treatment assignment avoids some assignment concerns but may be unavailable or ethically infeasible. Disagreement between methods is a prompt to examine different target populations and assumptions, not a reason to declare one estimate correct by default.

Common mistake: “The exposed and unexposed groups look similar on measured baseline covariates, so the instrument is valid and the marginal Wald ratio is causal.” Similarity on measured variables neither proves independence from all potential outcomes nor establishes exclusion. If independence is only conditional on covariates, marginal outcome and uptake differences can reflect covariate composition, as the worked example shows. State the relevant independence assumption, justify it from the policy context, and use an adjusted IV strategy when the design supports conditional rather than unconditional independence.

A credible natural-experiment analysis therefore reports the first stage and reduced form with uncertainty, documents baseline patterns, and makes the exclusion and independence arguments explicit. It also describes the policy’s timing and implementation, considers alternative pathways and interference, and states that a Wald effect pertains to compliers under the stated assumptions. No balance table or diagnostic statistic can replace this reasoning.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

A policy instrument changes uptake from 35% to 50%. What is the first-stage contrast, and does it establish instrument validity?
In a hypothetical study, the reduced-form contrast is \(-3\) outcome units and the first stage is \(0.15\). Calculate the unadjusted Wald ratio and state its interpretation only under the required assumptions.
Why can a small standardized difference in measured baseline covariates not prove unconditional independence?
Suppose instrument assignment is independent of potential outcomes only conditional on \(X\). Is the unadjusted marginal Wald ratio generally identified as the CACE? Explain.

Solutions. (1) The first stage is \(0.50-0.35=0.15\), a 15-percentage-point increase in uptake. It supports relevance, but does not establish independence, exclusion, or monotonicity. (2) The ratio is \(-3/0.15=-20\) outcome units per treatment. Under unconditional independence, exclusion, relevance, monotonicity, consistency, and no interference, it estimates the CACE; it is not automatically a population average effect. (3) Measured covariates are only part of the potential sources of dependence, and balance does not test exclusion. Unmeasured differences may remain. (4) No. Covariate composition can affect marginal outcome and treatment contrasts. A covariate-adjusted IV estimand may be appropriate under conditional independence and further conditions, but the unadjusted marginal ratio does not generally identify the CACE.