What a Design Audit Must Establish
A policy change, lottery, administrative rule, or institutional routine can create a useful causal comparison, but the label “natural experiment” does not establish that it did. The design must explain how units came to receive different assignments, why that process makes the proposed groups comparable for the question at hand, and exactly which population and effect the comparison represents. An audit makes those claims explicit before treating an observed difference as causal.
Let \(Z\) denote the assignment or offer generated by the proposed mechanism, \(D\) the treatment actually received, \(Y\) the outcome, and \(X\) measured characteristics determined before assignment. Write \(Y(z)\) for the outcome a unit would have under assignment \(z\), and \(D(z)\) for the treatment it would receive under that assignment. The target population must be named: it could be all eligible applicants, people assigned by a particular office during a period, or units near a cutoff. A design can be credible for one such population and not for a broader one.
For an assignment-based effect, the basic comparison is the difference in average outcomes by assignment. If assignment is as-if random in the target population, this contrast identifies the average effect of assignment, whether or not everyone takes up the treatment. In an instrumental-variables setting, assignment may instead serve as an instrument for treatment received. That requires additional assumptions; an assignment effect is not automatically the treatment effect.
State the Mechanism and Identifying Claim
An audit should describe the mechanism as an operational account, not as a label such as “exogenous” or “quasi-random.” Record who or what assigned units, the assignment rule or routine, when assignment occurred relative to eligibility and outcomes, and whether staff or units could influence the process. Specify the relevant assignment probabilities or thresholds where possible. If assignment depended on office, date, queue, judge, or eligibility stratum, say so and define the comparison within those conditions.
For a binary assignment, a useful identifying assumption for a target population \(S\) is conditional exchangeability: after conditioning on pre-assignment \(X\), assignment is independent of the potential outcomes. Also require consistency (the observed outcome under the received assignment equals the corresponding potential outcome) and positivity (both assignments are possible within the relevant \(X\) strata). For simplicity, the following derivation describes an assignment effect with no interference between units.
Under these conditions, the observed outcome mean in each assignment group represents the corresponding potential-outcome mean within each covariate stratum. Averaging over the target population’s distribution of \(X\) gives the assignment effect:
The first equality uses exchangeability to replace observed assignment-group means with potential-outcome means; consistency links those potential outcomes to observed outcomes. The second averages the stratum-specific contrasts over \(X\) in \(S\). Positivity ensures the needed assignment groups exist in those strata. This argument does not justify comparing groups outside \(S\), nor does it convert the assignment effect into an effect of treatment received.
When some assigned units do not receive treatment, or some unassigned units obtain it, the assignment contrast remains an intention-to-treat effect. Interpreting an instrumental-variable estimate as an effect of treatment received requires further conditions, including a relevant effect of assignment on treatment, an exclusion restriction, and—when interpreting the usual ratio as a complier effect—monotonicity. The audit should name these separately rather than treating them as consequences of a plausible assignment story.
Worked Audits
Worked Example: Auditing a Program Lottery
Hypothetical proposal. A training program has 1,000 eligible applicants for 500 places. A computer lottery assigns 500 applicants an offer. Of those offered, 72% attend; among those not offered, 18% obtain comparable training elsewhere. The employment rate one year later is 54% in the offer group and 46% in the no-offer group. Before assignment, mean age is 34.1 years (standard deviation 10.0) for those offered and 34.4 years (standard deviation 10.0) for those not offered.
Mechanism and target. The claimed mechanism is a computer lottery among the 1,000 eligible applicants, conducted after eligibility is fixed and before enrollment. The target is these lottery participants, not all job seekers. The audit should confirm the lottery list, randomization code or draw record, and that staff did not replace assignments.
Comparison and arithmetic. The directly identified candidate contrast is the offer-group employment rate minus the no-offer rate: \(0.54-0.46=0.08\). It is an 8-percentage-point assignment effect for these applicants if the lottery was implemented as described, outcomes were observed comparably, and interference is not consequential. It is not the effect of attending training: attendance differs from offer status, and 18% of the no-offer group received comparable training.
Diagnostic and judgment. The age standardized difference is \((34.1-34.4)/10.0=-0.030\), a small observed imbalance. The audit should also compare other pre-lottery characteristics, assess missing employment outcomes by assignment, and verify the allocation records. Similar baseline measurements are consistent with the lottery account but cannot prove it; the strongest evidence is the documented random draw. This design supports an offer effect. A treatment-received interpretation would need additional instrumental-variable assumptions and analysis.
Worked Example: Auditing a Capacity Threshold
Hypothetical proposal. An agency offers a housing subsidy to applicants with a priority score of at least 70 when funds are limited. The proposed comparison uses applicants just above and just below 70. Among applicants with scores from 68 to 69.9, 59% are housed within a year; among those scoring from 70 to 71.9, 68% are housed.
Mechanism and target. The proposed assignment is \(Z=1\) when the recorded score is at least 70. The target is applicants close to that threshold in this agency and period. The relevant identifying claim is local: absent the subsidy offer, expected potential outcomes would change smoothly through score 70, and applicants could not precisely manipulate their scores to sort across the cutoff. This is a cutoff-based comparison, not a claim that high- and low-score applicants are generally exchangeable.
Comparison and arithmetic. The unadjusted local outcome contrast is \(0.68-0.59=0.09\), or 9 percentage points. It is a candidate effect of eligibility or offer at the threshold, not necessarily a treatment effect of receiving the subsidy. A causal interpretation requires the cutoff rule to be implemented as stated and the local continuity and no-precise-sorting claims to be credible.
Diagnostic and judgment. Suppose the audit finds a pronounced excess of recorded scores just above 70, and a required pre-assignment income measure averages \$31,000 below the cutoff and \$36,000 above it. These findings raise concern about score manipulation, data processing, or a discontinuity in applicant composition. The 9-point contrast should not be presented as causal until the scoring and assignment records explain these patterns. Density and covariate checks are diagnostics: no visible discontinuity would not prove that potential outcomes are continuous.
Worked Example: Auditing Judge Assignment
Hypothetical proposal. A court assigns cases in rotation to judges. A researcher proposes using assignment to a judge with a higher tendency to impose incarceration as an instrument for incarceration when studying two-year re-arrest. In the audited sample, 200 cases go to each of two judges. The higher-incarceration judge’s cases have a 40% incarceration rate versus 25% for the other judge, while the pre-assignment high-severity share is 35% versus 25%.
Mechanism and target. The mechanism claim is rotation within courthouse, calendar, and case type; the target is cases actually covered by that rotation in the specified period. The audit must verify assignment logs and determine whether clerks or attorneys could direct cases, whether urgent cases bypassed rotation, and whether the two judges heard cases from comparable calendars.
Comparison and diagnostic arithmetic. The observed first-stage difference in incarceration is \(0.40-0.25=0.15\), or 15 percentage points. This supports relevance in the sample, but does not establish as-if-random assignment. The high-severity difference is \(0.35-0.25=0.10\), or 10 percentage points, a warning that case routing or calendar composition may differ. The audit should examine other pre-assignment case characteristics and assignment patterns within the claimed rotation strata.
Principal threat and judgment. Even with plausibly rotating assignment, the exclusion restriction may fail if judges affect re-arrest through other features of sentencing or court supervision, not only through the binary incarceration indicator. The assignment mechanism and first stage alone cannot settle that question. The design needs a defensible account of the treatment definition and of other pathways from judge assignment to the outcome; otherwise the proposed instrument does not identify the claimed treatment effect.
Diagnostics, Threats, and Alternatives
Choose diagnostics to match the mechanism and identifying assumption. Balance in pre-assignment characteristics can expose sorting or different composition; assignment logs can reveal deviations from a lottery or rotation; score distributions can reveal bunching around a threshold; and dates, locations, or policy records can expose anticipation or simultaneous changes. Examine missingness and outcome measurement by assignment as well: selective follow-up can create a comparison that was not present at assignment. Report the population, variables, and time windows examined rather than treating a single balance test as a design certificate.
A useful audit separates evidence into three categories: the mechanism’s documentation, observable implications that can be checked, and assumptions that remain substantively arguable. For example, a lottery record may directly document the draw; pre-assignment balance is an observable diagnostic; and the absence of interference between applicants is usually an assumption requiring contextual judgment. A non-significant balance test is not evidence that groups are identical, while a statistically detectable imbalance does not by itself reveal its cause. Magnitudes, uncertainty, and the assignment process all matter.
| Approach | What identifies the comparison | Key limitation |
|---|---|---|
| Natural experiment | A documented institutional or external assignment process plus design-specific assumptions | Mechanisms may be imperfect, and assumptions can be difficult to verify |
| Randomized trial | Random assignment designed and implemented by investigators | Noncompliance, attrition, interference, and limited target populations can still matter |
| Adjusted observational comparison | Measured covariates and an assumption that adjustment addresses relevant confounding | Unmeasured confounding can remain; adjustment does not repair a poorly understood assignment process |
The alternatives are not interchangeable labels. A credible natural experiment can be highly informative even when investigators did not allocate the policy or institution. Conversely, covariate balance cannot rescue a design whose assignment mechanism permits outcome-related sorting, and an elaborate adjustment model cannot compensate for an undefined target population. The audit’s purpose is to make the identifying argument inspectable and to limit the conclusion to what that argument supports.
Common Audit Mistakes
- Calling the mechanism random without explaining it. State who assigned what, by which rule, at what time, and what exceptions occurred.
- Using an unclear target population. A comparison among lottery applicants or cases near a threshold does not automatically generalize to nonapplicants or distant cases.
- Treating balance as proof. Similar measured characteristics do not establish balance in unmeasured potential outcomes; imbalance is a diagnostic requiring investigation, not an automatic verdict.
- Confusing assignment with treatment received. With noncompliance, the assignment comparison estimates an assignment effect unless additional instrumental-variable assumptions support a treatment-effect interpretation.
- Overlooking other assignment pathways. An instrument can affect an outcome through more than the named treatment, violating exclusion even when it strongly predicts treatment.
Check Your Understanding
Answer each question, then compare your reasoning with the brief solutions.
- A documented lottery assigns an offer, but 20% of those not offered obtain the service elsewhere. What effect does the direct offer-group comparison target?
- A policy cutoff produces an outcome contrast of \(0.62-0.55\). Express the contrast in percentage points and name one key local assumption needed for a causal interpretation.
- Why does a small standardized difference in measured baseline covariates not prove that assignment is as-if random?
- A judge assignment strongly predicts incarceration, but judges also differ in supervision practices that may affect the outcome. Which instrumental-variable assumption is threatened?
- Name two pieces of evidence an audit should seek when a proposal relies on rotating case assignment.
Solutions. (1) It targets the effect of being offered the service for the lottery population, not the effect of receiving it; control-group uptake does not change the assignment estimand. (2) \(0.62-0.55=0.07\), or 7 percentage points. For a cutoff design, a key assumption is that potential outcomes would vary continuously at the cutoff absent the assignment change, with no precise sorting that invalidates the local comparison. (3) The difference concerns only measured variables; unmeasured outcome-related differences or other failures of the assignment process can remain. (4) The exclusion restriction is threatened because judge assignment may affect the outcome through supervision apart from incarceration. (5) Inspect assignment logs or rotation rules and check for exceptions or diversion; also compare case characteristics and calendar composition within the claimed rotation conditions.