Lottery-Based Natural Experiments

What a Lottery Randomizes

When a scarce opportunity is allocated by lottery, the random draw can create a credible comparison between people offered the opportunity and people not offered it. The first task is not to calculate an effect. It is to specify who was eligible for the draw, what the draw assigned, and what outcome is being compared. Those choices determine the population and treatment contrast that an estimate can support.

A lottery generally randomizes an offer or assignment, not whether a person ultimately receives or uses the opportunity. Someone offered a program may decline; someone not offered a place may obtain similar services elsewhere. This distinction is central: randomization directly identifies the effect of the offer, while the effect of actual receipt requires additional assumptions.

For example, if a city draws from qualified applicants for a limited number of training places, the lottery population is those qualified applicants who entered that draw—not all city residents, all job seekers, or necessarily everyone who could have qualified. If the assignment is an offer of a place, that is the randomized contrast even if some offer recipients do not attend.

Effects of the Offer and of Receipt

Write \(Y(1)\) and \(Y(0)\) for a person's potential outcomes under, respectively, being offered or not being offered the opportunity. With random assignment, the average difference in observed outcomes by lottery assignment estimates the intention-to-treat effect (ITT) in the lottery population:

This is the effect of being offered the opportunity, including whatever happens after the offer: acceptance, refusal, alternative services, and any consequences of receiving or not receiving an offer. It does not automatically equal the effect of treatment receipt.

To describe receipt, define \(D(1)\) and \(D(0)\) as the treatment a person would receive under an offer and under no offer. The lottery’s first stage is the effect of assignment on receipt:

If the offer affects outcomes only by changing treatment receipt, if nobody is made less likely to receive treatment by being offered it, and if the offer changes receipt for some people, the Wald ratio identifies the average treatment effect for compliers—people whose receipt is changed by the offer. This is a local effect for that group, not necessarily an average effect for every lottery entrant.

Why the Wald Ratio Is a Complier Effect

The derivation makes the scope of the receipt effect explicit. Under random assignment, the difference in mean outcomes by \(Z\) equals the average difference in potential outcomes under assignment. Under exclusion, assignment has no direct effect on \(Y\); it can affect \(Y\) only by changing \(D\). Under monotonicity, people fall into three relevant groups: always-takers, who receive treatment either way; never-takers, who receive it neither way; and compliers, who receive it if offered but not otherwise. There are no defiers.

For always-takers and never-takers, changing \(Z\) does not change \(D\), so exclusion implies that their assignment effect on \(Y\) is zero. Only compliers contribute to the outcome difference. Let \(P(C)\) be the proportion of compliers, and let \(E[Y(1)-Y(0)\mid C]\) be their average effect of receipt. The ITT and first stage are therefore:

The first-stage equality follows because only compliers change receipt when \(Z\) changes, and each complier changes receipt from 0 to 1. Dividing the two differences gives:

This derivation also explains why the assumptions matter. If an offer changes outcomes directly—for example, by changing expectations or access to information even for someone who does not attend—the numerator includes more than the effect operating through receipt. If some people are defiers, the denominator is no longer simply the proportion of compliers, and the usual complier interpretation need not hold.

Lottery Rules Shape the Comparison

The word “lottery” does not guarantee that every person in the analysis had the same probability of an offer. A draw may be conducted separately within priority groups, locations, or application periods, or it may use different assignment probabilities. The analysis should reflect the actual randomization. If offers were randomized within strata, compare offer and no-offer outcomes within each stratum and combine the stratum-specific effects using weights that define the target population. A simple pooled difference can be misleading when the offer group and control group contain different proportions of strata.

The population definition also concerns timing. Define eligibility and entry using information established before the draw. Removing people after assignment because they declined, failed to attend, moved, or had an unfavorable outcome can undo the benefit of randomization. In general, retain people in their assigned groups for the offer-effect analysis and report how outcomes were observed or became missing.

Diagnostics, Alternatives, and Limits

A careful analysis checks the assignment mechanism before relying on the word “random.” Confirm the eligible list, draw procedure, allocation probabilities, strata, and any changes to assignment. Baseline characteristics can be compared across assignment groups as a diagnostic, but an imbalance does not by itself prove the lottery failed, and balance does not prove that it succeeded. More direct evidence comes from records of the actual draw and from checking whether assignment was altered, manipulated, or selectively recorded.

Also report the first stage. If the offer barely changes receipt, the offer may still have a meaningful ITT, but the Wald ratio divides by a small number and can be highly imprecise. Do not interpret a large ratio as strong evidence of a large treatment effect without uncertainty estimates and a careful assessment of the first stage. Missing outcomes, treatment misclassification, treatment obtained outside the lottery, and interference between applicants can also compromise interpretation.

The main alternatives answer different questions. A comparison of people who actually received treatment with those who did not is vulnerable to differences in willingness, resources, or need, even when the offer was randomized. An adjusted observational comparison may address measured differences but relies on adequate measurement and adjustment for confounding. The randomized offer comparison avoids that particular confounding for the offer effect, while the complier effect requires the extra assumptions above. These approaches should not be treated as interchangeable estimates of one universal effect.

A lottery’s strongest claim is usually local and precise: it can identify the effect of an offer for the people who entered that particular eligible draw. Generalizing to people who did not apply, different eligibility groups, other places, or other versions of the opportunity requires further evidence. Keep the population, offer contrast, receipt definition, and estimand explicit whenever reporting results.

• Confusing the offer with receipt. The offer is randomized; participation may not be. Name the offer effect as the ITT.
• Calling a Wald estimate an effect for everyone. Under its assumptions, it is an effect for compliers in the lottery population.
• Ignoring the draw structure. If assignment was stratified or used varying probabilities, use an analysis that reflects those rules and the target population.
• Conditioning on post-lottery behavior. Excluding people because they did not take up an offer can break the randomized comparison.
• Assuming that a lottery proves every assumption. Random assignment supports the offer comparison, but exclusion, monotonicity, treatment definition, and outcome measurement still require scrutiny.