Natural Experiments and Quasi-Experiments

When Treatment Assignment Is Not Controlled

Suppose a city changes a transport policy, a storm affects some neighborhoods more than others, or a school program is introduced in one district but not another. Researchers may be able to compare outcomes for people exposed to the change with outcomes for people who were not. These comparisons can provide evidence about possible effects, but they do not automatically have the protection of random assignment.

The key question is how did people or groups end up receiving the treatment? In a randomized experiment, the investigator uses chance to assign treatments. In a natural or quasi-experiment, treatment assignment is not fully controlled by the investigator. The source of the assignment, and whether it makes the groups comparable, determines how cautiously a comparison must be interpreted.

Definition: A natural experiment occurs when an external event, rule, or circumstance creates different treatment exposures, without the researcher assigning them. A quasi-experiment is a study that compares outcomes across treatment conditions without random assignment, often after a program or policy is introduced. Natural experiments are commonly treated as a kind of quasi-experiment; usage varies, so the most important description is how treatment was assigned.

Neither label means that the comparison is automatically fair. A naturally occurring event can affect groups differently for reasons that also affect the outcome. A researcher may deliberately evaluate an intervention, but if participants or sites were not randomly assigned, the treated and comparison groups may differ in important ways before the intervention. The material on Confounding in Study Design and Association, Causation, and Study Conclusions is directly relevant: a difference in outcomes is an association, and alternative explanations must be considered before making a causal claim.

What a Comparison Can Estimate

Let \(D=1\) indicate that a unit receives the treatment and \(D=0\) indicate that it does not. Let \(Y(1)\) be the outcome that unit would have under treatment and \(Y(0)\) the outcome it would have without treatment. These are called potential outcomes. For each unit, only one is observed: the outcome is \(Y(1)\) if \(D=1\), and \(Y(0)\) if \(D=0\). The other is counterfactual: what would have happened to that same unit under the condition it did not receive.

A natural or quasi-experimental study usually compares the average observed outcome among treated units with the average observed outcome among untreated units. For a numerical outcome, this might be a difference in sample means. That difference describes the observed groups. It estimates a causal effect only if the groups provide a credible comparison for the missing counterfactual outcomes.

Key distinction: A comparison group shows what happened to units that did not receive the treatment. It represents what would have happened to the treated units only if the groups are sufficiently comparable for the causal question. Similar labels, locations, or sample sizes alone do not establish that comparability.

The following identity makes the difficulty precise. Assume that the observed outcome equals the relevant potential outcome for each unit: treated units reveal \(Y(1)\), and untreated units reveal \(Y(0)\). Then the observed difference in population means can be decomposed as follows:

\begin{aligned} E[Y\mid D=1]-E[Y\mid D=0] &= E[Y(1)\mid D=1]-E[Y(0)\mid D=0]\\ &= \bigl(E[Y(1)\mid D=1]-E[Y(0)\mid D=1]\bigr)\\ &\quad+\bigl(E[Y(0)\mid D=1]-E[Y(0)\mid D=0]\bigr). \end{aligned}

The first bracket is the average treatment effect for the treated units: their average outcome under treatment minus what their average outcome would have been without treatment. The second bracket is a selection difference: how the treated and untreated groups would have differed even if neither group had received treatment. We cannot observe both potential outcomes for the same units, so the selection difference is not directly known. Random assignment is valuable because, on average, it makes treatment assignment independent of potential outcomes. Without random assignment, researchers need a credible design and assumptions to address this selection difference.

Worked Examples

Worked Example: A Natural Experiment After a Bridge Closure

Imagine that a bridge closes unexpectedly for repairs. Researchers compare average commuting times for residents whose usual route crossed the bridge with times for residents in a nearby area whose usual routes did not. Suppose the first group’s mean commute time rises from 28 to 36 minutes, while the comparison group’s mean rises from 27 to 29 minutes. These are hypothetical data. What do the comparisons show?

For the group affected by the closure, the change is \(36-28=8\) minutes. For the comparison group, the change is \(29-27=2\) minutes. A difference-in-differences comparison subtracts the comparison group’s change from the affected group’s change:

(36-28)-(29-27)=8-2=6\text{ minutes}.

The observed change was 6 minutes greater for residents whose usual routes crossed the bridge. The unexpected closure is an external event, so this is a natural experiment rather than an intervention assigned by the researchers. However, 6 minutes is not automatically the closure’s causal effect. For that interpretation, the comparison area must represent how commuting times in the affected area would have changed without the closure. For example, the groups should not have had different underlying trends or another event that affected only one area. The increase in the comparison group helps account for a change shared by both areas, but it cannot rule out every other explanation.

Worked Example: A Tutoring Program in One District

Suppose a school system introduces a tutoring program in District A but not District B. Assignment is by district policy, not by random allocation of students or schools. The hypothetical mean test scores are 70 before and 75 after the program in District A, and 66 before and 68 after in District B. Scores are measured in points. Calculate and interpret the difference-in-differences comparison.

District A’s mean score changes by \(75-70=5\) points. District B’s changes by \(68-66=2\) points. The comparison of changes is therefore:

(75-70)-(68-66)=5-2=3\text{ points}.

The increase in District A’s mean score was 3 points greater than the increase in District B’s mean score. This is a quasi-experimental comparison because the program was introduced, but treatment was not randomly assigned. A causal interpretation would require, among other things, that without the program the districts would have followed parallel outcome trends over this period. In other words, the difference between their mean scores would have changed similarly in the absence of tutoring. One pre-program measurement for each district cannot establish that assumption. Differences in student composition, other district policies, or distinct trends could also contribute to the 3-point result.

Worked Example: Eligibility Set by a Date Rule

Imagine that a hypothetical training program is offered to everyone born on or after a stated calendar date, while those born just before that date are ineligible. Researchers compare missed workdays over the next year for people born shortly before and shortly after the cutoff. The hypothetical means are 3.8 days for the ineligible group and 3.1 days for the eligible group. Calculate the observed difference, taking eligible minus ineligible.

The difference is \(3.1-3.8=-0.7\) days. People in the eligible group averaged 0.7 fewer missed workdays over the year than those in the ineligible group. Eligibility is determined by a rule, not by the researchers’ random assignment, so this is a quasi-experimental comparison.

The difference could support a causal interpretation for people near the cutoff if, apart from program eligibility, people born just before and after the date are comparable in ways that matter to missed workdays. It would also matter whether another policy changes at the same cutoff and whether people can manipulate which side of the rule they fall on. Even if these conditions are plausible, the comparison is local to people near the date; it does not by itself establish the program’s effect for people born much earlier or later.

Assessing the Comparison Group

Before interpreting a natural or quasi-experimental comparison as evidence of an effect, describe the assignment mechanism and ask what makes the groups comparable. A naturally occurring change may provide a useful source of variation, but “outside the researcher’s control” is not the same as “unrelated to the outcome.” Similarly, an intervention may be introduced by an institution for reasons that predict outcomes. These features can create confounding even when the comparison is carefully planned.

Several questions help clarify what the design can support:

What determined exposure? Name the event, rule, policy, or decision and explain which units it affected.
Who is the comparison group? State why these units are relevant to the counterfactual question, rather than simply noting that they did not receive treatment.
What else changed at the same time? Another change that affects the treatment group differently may explain part or all of the observed outcome difference.
Were the groups already different? Differences in baseline outcomes or other characteristics can signal that a simple comparison may be misleading.
How broad is the conclusion? A result for a particular place, time, or group need not apply to other populations or settings.

These questions do not turn a nonrandomized study into a randomized experiment. They help distinguish a descriptive group difference from a causal conclusion and make the assumptions behind that conclusion explicit.

Why the Distinction Matters

Natural and quasi-experiments are useful because random assignment may be impractical, unavailable, or inappropriate, while an external event or policy can still create an informative comparison. They can help researchers study real-world changes that could not be deliberately assigned. Their strength depends on how plausibly the assignment mechanism isolates treatment exposure from other causes of the outcome—not on the label “natural experiment” alone.

A common mistake is to say, “The treatment was not chosen by the researchers, so the groups were effectively randomized.” That reasoning is incorrect: an external event can still affect groups in systematically different ways. The correction is to identify exactly how exposure was determined, examine whether that mechanism makes the groups comparable, and qualify the conclusion if important alternative explanations remain. When those conditions are not well supported, report the observed association and avoid claiming that treatment caused the difference.

Key takeaway: Natural and quasi-experiments compare groups without researcher-controlled random assignment. The observed difference is real as a description of those groups; interpreting it as a treatment effect requires a defensible argument that the comparison group represents the missing counterfactual.

Check Your Understanding

Use the assignment mechanism and the comparison group—not just the study label—to answer these questions.

A new service is introduced in one town, and researchers compare outcomes with a nearby town where it is not introduced. Why is this not automatically a randomized experiment?
In a natural experiment, what does “external assignment” mean, and why does it not guarantee comparable groups?
A program group’s outcome rises by 4 units while a comparison group’s rises by 1 unit. Calculate the difference in changes and state what it describes.
For a causal interpretation of a difference-in-differences comparison, what does the parallel-trends assumption say?
Why might an eligibility-rule comparison support a conclusion about people near a cutoff but not about a much broader population?