What Makes an Experiment “Natural”?
A treatment difference can arise because an institution assigns access according to a rule, even when researchers did not design or control that rule. The central question is not whether the circumstances seem unusual. It is whether the rule makes assignment independent of the causes of the outcome, either in the study population or after conditioning on appropriate pre-assignment characteristics.
For example, a new service may be introduced through an administrative schedule. That fact alone does not make the resulting comparison causal. If the schedule gives earlier access to people with greater need, the assigned groups may differ in outcome risk even without the service. A credible natural experiment therefore requires an account of the assignment mechanism: who or what determined assignment, when it was determined, what information the decision-maker used, and whether people could influence it.
Let \(Z\) denote assignment under the mechanism, \(D\) actual treatment received, and \(Y\) the outcome. The assignment \(Z\) may be an offer, eligibility, or access rule rather than treatment itself. Distinguishing \(Z\) from \(D\) matters: people may decline an offer, obtain treatment elsewhere, or receive treatment despite not being assigned access. Potential outcomes \(Y(z)\) describe outcomes under assignment \(z\); potential treatment statuses \(D(z)\) describe treatment receipt under that assignment.
For a direct comparison of assignment groups, a key condition is exchangeability: \(Z\) is independent of the potential outcomes \(Y(0)\) and \(Y(1)\). If the assignment is plausibly random only within strata defined by pre-assignment covariates \(X\), the condition is \(Z \perp (Y(0),Y(1)) \mid X\), together with a positive chance of either assignment at each relevant value of \(X\). The conditioning variables should describe the assignment process and be measured before assignment; adjusting for variables caused by assignment can introduce bias rather than remove it.
Assignment, Treatment, and the Causal Claim
When assignment itself is the intervention of interest, the assignment effect is the average contrast between \(Y(1)\) and \(Y(0)\). Under exchangeability, consistency (the observed outcome equals the potential outcome under the assignment actually received), and no interference between units, the difference in mean outcomes by assignment identifies this effect. This is often called an intention-to-treat effect. It answers what the assignment or offer changes, not necessarily what receiving treatment changes.
If assignment changes treatment receipt but not everyone complies, it may serve as an instrument for \(D\). In the binary-assignment, binary-treatment setting, a familiar IV interpretation requires more than a plausible assignment mechanism: assignment must affect treatment receipt (relevance); assignment must be independent of potential treatment and outcome values; assignment must affect the outcome only through treatment (exclusion); and there must be no defiers, meaning no units who would take treatment when unassigned but not take it when assigned. Under these conditions, the Wald ratio identifies the average treatment effect among compliers, the units whose treatment receipt is changed by assignment. These assumptions are substantive claims about the mechanism, not consequences of calling a setting a natural experiment.
Worked Example: An Offer as an Instrument
Hypothetical data. An institution allocates an offer to attend a support program using a documented administrative rule. Among 50 people offered access, 40 attend and 30 recover. Among 50 not offered access, 20 attend through other routes and 20 recover. Suppose the rule supports as-if random assignment, and the exclusion and monotonicity assumptions are also credible for this example.
The assignment effect on recovery is the difference in recovery rates:
The offer increases recovery by 0.20, or 20 percentage points, on average in this hypothetical population. The first-stage difference in attendance is:
The Wald ratio is \(0.20/0.40=0.50\). Under the stated IV assumptions, this estimates a 0.50, or 50-percentage-point, increase in recovery from treatment among compliers. It is not the effect for every person offered access, nor does the observed 20-percentage-point assignment effect itself equal the treatment effect. If the offer also changes recovery through attention, information, or another channel, exclusion fails and the Wald interpretation is not justified.
Evaluate the Mechanism, Not the Story
A credible argument begins with institutional detail. Identify the rule that generated assignment and specify its timing. Establish whether the rule was applied as described, whether decision-makers could override it, and whether people could sort into assignment categories. Ask whether the rule used information related to prognosis, treatment motivation, or likely outcomes. If it did, explain why that information is fully accounted for by a defensible conditioning set—or acknowledge that the comparison may be confounded.
Observed covariate balance can help assess whether the proposed mechanism behaves as expected. For example, compare pre-assignment characteristics across assignment groups, preferably within the strata used in the design. Large differences in important predictors can signal sorting, selective implementation, or a misspecified conditioning set. Similar observed characteristics are reassuring but cannot establish independence: unmeasured characteristics may still differ. Institutional records and process evidence are therefore essential complements to statistical checks.
Worked Example: Unusual Exposure Is Not Enough
Hypothetical data. Two facilities introduce a service at different times. The earlier facility serves more high-risk patients, and assignment to the earlier service was partly based on managers’ forecasts of patient need. During the evaluation period, 60 of 100 patients at the early facility recover, compared with 45 of 100 at the later facility.
The observed recovery-rate difference is:
The early facility’s recovery rate is 15 percentage points higher. That arithmetic describes the observed groups, but it does not identify the service’s causal effect. The timing rule used forecasts related to need, so assignment may be associated with potential outcomes. The direction and size of the resulting bias cannot be determined from these two rates alone. Calling the staggered introduction unusual or externally imposed does not remove this concern. A causal interpretation would need a credible account of why the forecasts do not create residual outcome differences, or a different design that addresses the assignment process.
Conditional Quasi-Random Assignment
Sometimes assignment is not comparable across the entire sample but is plausibly as-if random within known groups. In that case, the relevant comparison is within groups, followed by standardization to a clearly stated target distribution. This is not a license to adjust for every available variable; the groups should be justified by the assignment mechanism and defined using pre-assignment information.
Worked Example: Standardizing Within Assignment Strata
Hypothetical data. An administrative rule plausibly assigns an offer as-if randomly within each of two pre-assignment risk strata. In stratum A, 80 of 100 offered people recover, while 14 of 20 not offered recover. In stratum B, 8 of 20 offered people recover, while 24 of 80 not offered recover. Suppose both assignment options occur in each stratum, and the as-if random claim within strata is credible.
The crude recovery rate among those offered is \((80+8)/(100+20)=88/120=0.7333\). Among those not offered it is \((14+24)/(20+80)=38/100=0.3800\). The crude difference is \(0.3533\), or 35.33 percentage points. But the offered group is mostly stratum A, while the not-offered group is mostly stratum B.
Within stratum A, the difference is \(80/100-14/20=0.80-0.70=0.10\). Within stratum B, it is \(8/20-24/80=0.40-0.30=0.10\). For a target population with equal proportions in the two strata, standardize each assignment rate using weights of 0.5:
The standardized assignment contrast is \(0.60-0.50=0.10\), or 10 percentage points, for this target distribution. The calculation relies on the assumed within-stratum exchangeability and positivity; it does not establish those conditions. The crude 35.33-point difference mixes assignment contrasts with the different stratum composition of the groups.
The justification is the law of total expectation. If assignment is exchangeable within \(X\), then, for each assignment value \(z\), the observed conditional mean \(E(Y\mid Z=z,X=x)\) equals the corresponding potential-outcome mean \(E(Y(z)\mid X=x)\). Averaging those conditional means over the same target distribution of \(X\) gives the marginal potential-outcome means for that target:
where the sum is over the strata and the displayed equality applies when exchangeability and consistency hold and each stratum has positive probability of each assignment. Subtracting the standardized means therefore gives the assignment effect for that target population. Changing the weights changes the target population and can change the effect being summarized.
Diagnostics, Alternatives, and Limits
Several checks can reveal problems, although none proves quasi-random assignment. Examine whether assignment follows the documented rule, whether assignment probabilities differ sharply across relevant pre-assignment groups, and whether people or decision-makers could manipulate the assignment process. Check missing outcomes and attrition by assignment: selective loss can undo an initially credible comparison. For IV use, inspect whether assignment meaningfully changes treatment uptake; a weak first stage makes the instrument uninformative and can make estimates unstable. Consider plausible direct paths from assignment to outcome, since covariate balance does not test the exclusion restriction.
| Approach | Source of comparison | Key limitation |
|---|---|---|
| Randomized trial | Researcher assigns treatment by design | May be infeasible or differ from routine institutional conditions |
| Natural experiment | An institutional mechanism creates plausibly as-if random assignment | Credibility depends on the mechanism and its assumptions |
| Observational adjustment | Adjusts for measured differences between treatment groups | Unmeasured confounding can remain |
| Regression discontinuity | Uses assignment changes at a cutoff | Identifies an effect local to the cutoff under its design assumptions |
Natural experiments can offer credible causal evidence when random assignment is unavailable, but they do not automatically generalize beyond the institutions, people, or time periods that generated the assignment. The assignment effect may be policy-relevant even when it is not the treatment effect. An IV estimate, when justified, concerns compliers rather than necessarily the full population. Comparisons should therefore state the assignment, target population, estimand, and assumptions precisely.
Common Mistake
- Wrong reasoning: “The exposure was surprising or outside the researchers’ control, so it was random.”
- Correction: An external or unusual event can still affect groups selectively. Identify the actual assignment rule and justify why it is independent of potential outcomes, possibly conditional on pre-assignment characteristics. If that case cannot be made, the observed contrast remains an association rather than an identified causal effect.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- An institution assigns access using a rule, but staff sometimes override it based on expected patient need. What threat does this raise?
- An offer raises treatment uptake from 30% to 50% but does not change the outcome rate. What is the assignment effect, and what does the Wald numerator imply?
- Why might a covariate-balanced assignment comparison still fail to support an IV estimate?
- In a conditional design, why must both assignment options have positive probability in each target stratum?
Solutions. (1) Overrides based on expected need can associate assignment with potential outcomes, undermining exchangeability. (2) The assignment effect is zero percentage points; with a 20-percentage-point first stage, the Wald ratio is \(0/0.20=0\), subject to IV assumptions. (3) Balance on measured covariates does not prove independence from unmeasured factors, and it does not establish exclusion or monotonicity. (4) Without both assignment options in a stratum, its missing conditional outcome mean cannot be learned from the observed data, so the target standardized contrast is not identified from that comparison alone.