From an Unexpected Event to a Research Design
A storm, transit interruption, or infrastructure failure can change who receives a treatment. That change may create useful variation, but the event’s unexpectedness alone does not make treatment assignment random. The key questions are more specific: which units were exposed, when did exposure occur, how did it change treatment receipt, and could it also affect the outcome through another route?
For example, a sudden road closure might make a health program harder to attend for residents on one side of a town. The closure is the event; the residents’ exposure to it is the potential source of treatment variation. Researchers must establish the route from the event to attendance, determine who was affected, and examine whether the closure could also change health outcomes directly, such as by delaying access to emergency care.
A useful evaluation starts with a timeline. Record when the event occurred, when exposure was determined, when treatment could change, and when outcomes were measured. If units could anticipate the event or adjust their behavior beforehand, the timing may not generate the intended variation. Also specify the unit of analysis: a storm may affect neighborhoods, while treatment and outcomes are recorded for individuals. Mixing those levels can obscure who was actually exposed.
Assumptions for an Instrumental-Variable Interpretation
For a binary exposure \(Z_i\), binary treatment \(D_i\), and outcome \(Y_i\), define \(D_i(z)\) as the treatment status unit \(i\) would have under exposure \(z\). Define \(Y_i(d)\) as the outcome under treatment status \(d\), assuming exclusion: the outcome depends on \(Z_i\) only through treatment. A standard Wald interpretation also needs the following conditions.
- Relevance: exposure changes treatment receipt, so \(E[D_i\mid Z_i=1]\) differs from \(E[D_i\mid Z_i=0]\) in the target population.
- Independence: exposure is independent of the relevant potential treatment and outcome values. In the unconditional version, \(Z_i\) is independent of \(\{D_i(0),D_i(1),Y_i(0),Y_i(1)\}\).
- Exclusion: exposure affects the outcome only by changing treatment. In potential-outcome notation, the outcome under treatment \(d\) does not depend on exposure.
- Monotonicity: exposure does not cause some units to take treatment while causing other units to avoid it; for every unit, either \(D_i(1)\geq D_i(0)\) or the reverse direction is specified consistently. The usual positive-first-stage version uses \(D_i(1)\geq D_i(0)\).
- No interference and well-defined exposure: one unit’s exposure does not change another unit’s outcome in a way omitted from the design, and the exposure and treatment have sufficiently clear definitions.
Under these conditions, the Wald ratio identifies the average treatment effect among compliers, units with \(D_i(0)=0\) and \(D_i(1)=1\). It does not generally identify the average effect for everyone. The independence condition above is unconditional. If exposure is independent of potential outcomes only after conditioning on covariates, the unadjusted marginal Wald ratio need not identify the complier effect: exposed and unexposed groups can have different covariate distributions.
A short derivation clarifies what this ratio requires. Under unconditional independence, the difference in mean outcomes by exposure equals the difference in mean potential outcomes across exposure groups, averaged over the same population. Exclusion gives \(Y_i=Y_i(D_i)\). Under monotonicity, units fall into always-taker, never-taker, and complier groups; the defier group is absent. Always-takers and never-takers have the same treatment under either exposure, so their treatment-induced outcome contribution cancels in the reduced form. Only compliers contribute. Thus the reduced-form difference is the proportion of compliers multiplied by their average treatment effect. The first-stage difference is that same proportion. Dividing yields the complier average treatment effect:
If independence is justified only conditional on covariates \(X_i\), analysis must account for that conditioning. A stratum-specific IV estimate can target compliers within a covariate stratum when the assumptions hold there. A covariate-adjusted IV analysis can also be appropriate, but its specification and target estimand must be made explicit. Simply inserting a conditional-independence claim and then reporting the unadjusted marginal ratio is not justified.
Worked Example: Mapping a Transport Shock to Exposure
Suppose a hypothetical city experiences an unannounced bridge closure shortly before a free job-training course begins. Researchers define \(Z_i=1\) for applicants whose usual route to the training site crosses the bridge, using routes documented before the closure. In a sample of 160 applicants, 80 have \(Z_i=1\) and 80 have \(Z_i=0\). Course attendance is the treatment \(D_i\).
Timing and affected units. The route definition predates the closure, and the closure occurs before course attendance is measured. This reduces concern that applicants chose their exposure classification in response to the event. The exposed units are applicants with a pre-closure route over the bridge, not all people living near it. Researchers should still check whether bus routes, alternative bridges, or travel patterns make the binary exposure definition misleading.
First stage. Suppose 56 of the 80 exposed applicants attend, compared with 32 of the 80 unexposed applicants. The attendance rates are \(56/80=0.700\) and \(32/80=0.400\), so the first-stage difference is \(0.700-0.400=0.300\), or 30 percentage points. Exposure is relevant in this sample: attendance is more common among exposed applicants.
That calculation does not establish independence or exclusion. Applicants on the affected route may differ in employment access or income, and a closure might affect job searches directly. The event supplies a plausible treatment contrast to investigate, not an automatic causal design.
Calculating an Effect and Auditing Its Meaning
Worked Example: A Wald Estimate Under Stated Assumptions
Consider a separate hypothetical setting in which a rare, unexpected weather shift changes access to an optional certification course. Let \(Z_i=1\) indicate that an applicant was exposed to the access disruption, \(D_i=1\) indicate course completion, and \(Y_i=1\) indicate certification within six months. Suppose exposure is as-if random in the defined study population, there are no spillovers, and the substantive case for exclusion and monotonicity is credible. Among 250 exposed applicants, 150 complete the course and 180 earn certification. Among 250 unexposed applicants, 100 complete the course and 160 earn certification.
The treatment-rate difference is \((150/250)-(100/250)=0.600-0.400=0.200\). The certification-rate difference is \((180/250)-(160/250)=0.720-0.640=0.080\). Therefore,
Under the stated assumptions, the estimate is a 40-percentage-point average effect of course completion on certification among applicants whose completion status is changed by exposure. It is not the effect for every applicant, nor does the arithmetic establish the assumptions. In particular, if the weather shift also changes study time, exam attendance, or employment opportunities, the exclusion condition may fail even though the first stage is large.
Worked Example: A Strong First Stage Does Not Rescue Exclusion
In another hypothetical evaluation, a sudden transit outage reduces attendance at a clinic treatment. Among 100 people exposed to the outage, 50 receive treatment and 60 have a favorable outcome. Among 100 unexposed people, 70 receive treatment and 50 have a favorable outcome. Define \(Z_i=1\) as outage exposure and \(D_i=1\) as treatment receipt.
The first-stage difference is \(0.50-0.70=-0.20\), and the outcome difference is \(0.60-0.50=0.10\). The arithmetic Wald ratio is \(0.10/(-0.20)=-0.50\). But suppose the outage also delays emergency care, which directly changes the favorable outcome. Then exclusion is implausible: \(Z_i\) affects \(Y_i\) through a pathway other than \(D_i\). The ratio of \(-0.50\) is not a defensible estimate of the treatment effect, regardless of the strength or statistical precision of the first stage.
The negative first stage also requires careful coding and interpretation. Here outage exposure reduces treatment receipt, so the denominator’s sign is expected to be negative. Reversing the coding of \(Z_i\) would reverse both differences and leave the ratio unchanged, but it would not repair the direct pathway.
Diagnostics, Alternatives, and Limits
A credible assessment traces the shock through four linked questions: exposure, timing, affected units, and competing pathways. Useful diagnostics include a map or rule defining exposure before outcomes are examined; a timeline showing that the shock preceded treatment and outcome measurement; treatment rates by exposure; and checks for spillovers, anticipation, differential attrition, and changes in outcome measurement. Pre-event characteristics and outcomes can reveal imbalances or divergent patterns, but balance checks cannot prove independence, and a strong first stage cannot prove exclusion.
List every plausible route from the shock to the outcome. Classify each route as one that operates through treatment or as a competing direct pathway. For instance, if exposure changes both treatment attendance and access to other services, exclusion may fail. If the shock affects a whole community, unexposed comparison units may still experience price changes, congestion, or changed service availability. These spillovers challenge the assumed contrast even when individual exposure is measured accurately.
| Approach | What creates the comparison | Key limitation for a shock study |
|---|---|---|
| Randomized assignment | Design-based assignment to treatment | May be unavailable or infeasible; a shock is not randomized merely because it was unexpected. |
| Matching or propensity-score adjustment | Observed covariate similarity between treated and comparison units | Does not by itself address unmeasured confounding or provide an instrument for treatment receipt. |
| Before-after comparison | Outcomes before versus after the event | Other time changes can be mistaken for the shock’s effect. |
| Shock-based IV | Exposure-induced changes in treatment | Requires relevance, independence, exclusion, and (for the usual complier interpretation) monotonicity. |
The choice depends on the causal question and the design evidence. A before-after pattern may describe what followed an event, but does not by itself isolate a treatment effect. Matching can balance measured covariates, but does not automatically exploit the shock as an instrument. IV targets the effect for compliers only under its assumptions. If exposure varies systematically across measured covariates, conditional independence requires an appropriately adjusted analysis; an unadjusted Wald ratio is not a substitute.
Common mistake: “The storm was unexpected, so exposed and unexposed people were randomly assigned.” Unexpectedness concerns the event’s timing, not necessarily which units experience it. Exposure may depend on location, infrastructure, or behavior, all of which can relate to outcomes. The correction is to define exposure using a defensible rule, investigate how that rule relates to potential outcomes, and separately assess exclusion and treatment relevance.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A snowstorm occurs unexpectedly. Why does that fact alone not establish independence of exposure from potential outcomes?
- In a hypothetical study, treatment rates are 0.55 for \(Z=1\) and 0.35 for \(Z=0\), while outcome rates are 0.70 and 0.64. Calculate the Wald ratio and state its conditional interpretation.
- Exposure is independent of potential outcomes only within rural and urban strata. Is the unadjusted marginal Wald ratio automatically a complier average treatment effect?
- A power outage changes treatment attendance and also prevents some participants from taking the outcome test. Which assumption is threatened, and why?
Solutions. (1) The event’s unexpected timing does not imply that exposure is unrelated to location, infrastructure, behavior, or potential outcomes; independence needs separate justification. (2) The first stage is \(0.55-0.35=0.20\), the reduced form is \(0.70-0.64=0.06\), and the ratio is \(0.06/0.20=0.30\). If the IV assumptions hold, this is a 30-percentage-point effect among compliers, not necessarily among all units. (3) No. Covariate-stratum differences can make the exposed and unexposed groups compositionally different, so a marginal ratio need not identify the complier effect. Use a justified stratified or covariate-adjusted IV analysis and specify its estimand. (4) Exclusion is threatened because exposure can affect whether the outcome is observed, apart from its effect through treatment; differential missingness may also distort the measured outcome contrast.