A Policy Evaluation With Placebo Outcomes

Why Use a Placebo Outcome in a Policy Evaluation?

Suppose some school districts introduce a tutoring policy and an analyst estimates its effect on student mathematics scores by comparing changes in adopting districts with changes in other districts. A useful diagnostic is to repeat the comparison using an outcome that the tutoring policy cannot cause, such as local rainfall during the school term. The policy should not change rainfall. If the same evaluation design nevertheless finds a substantial policy-associated change in rainfall, the result calls attention to features of the design, data, or comparison groups that may also matter for the mathematics-score estimate.

A placebo outcome is not a second estimate of the policy’s effect on the target outcome. It is a falsification check: the analyst asks whether the design reports an association where the causal effect should be absent. A nonzero result is a warning, not a measurement of the amount of bias in the target estimate. A null result is also limited: it may reflect low power, or the placebo outcome may fail to share the threats affecting the target outcome.

Specify the Falsification Design Before Calculating

For a policy adopted in some districts but not others, let \(G_i=1\) identify districts that adopt the policy and \(G_i=0\) identify comparison districts. Let \(t=0\) denote the pre-policy period and \(t=1\) the post-policy period. Let \(P_{it}\) be a district-level placebo outcome, such as mean daily rainfall in millimetres. A simple difference-in-differences placebo estimate is

Here, \(\bar{P}_{g,t}\) is the mean placebo outcome for group \(g\) in period \(t\). The first difference is the change in adopting districts; the second is the change in comparison districts. The estimate is expressed in the units of the placebo outcome. The design, outcome definition, time window, comparison group, covariate adjustment, and handling of missing observations should be chosen before looking at the result. Otherwise, selecting a specification because it gives a reassuring placebo estimate can undermine the test.

The interpretation requires more than the claim that the policy does not cause the placebo outcome. The comparison must also be credible for changes over time. In particular, the expected change in the untreated placebo outcome for adopting districts must equal the expected change in comparison districts. The outcome must be measured comparably across groups and periods, and the set of units contributing to each mean must not change in a way that creates a spurious contrast. If the policy can affect who is observed, how the outcome is recorded, or whether a unit remains in the sample, then even a seemingly unaffected outcome can show a policy-associated change.

The derivation makes clear what a nonzero estimate can and cannot diagnose. Write \(P_{it}(0)\) for the outcome unit \(i\) would have at time \(t\) without the policy. Suppose the policy cannot affect the placebo outcome, so the observed placebo outcome equals \(P_{it}(0)\) in both groups and periods. Under parallel trends for this placebo outcome,

Subtracting the right-hand change from the left-hand change gives zero. Thus, the expected difference-in-differences placebo contrast is zero. If an estimate differs from zero, at least one part of the joint story may be wrong: there may be differential trends, selection or measurement changes, chance variation, or the outcome may not actually be unaffected. The estimate alone does not identify which explanation is responsible.

Temporal Diagnostics: A Zero Post-Policy Contrast Is Not Enough

The post-policy placebo contrast is more informative when the analyst also examines the placebo outcome before adoption. A difference in pre-policy trends challenges the idea that adopting and comparison groups provide a useful counterfactual for each other. Similar pre-policy trends are reassuring only in a limited sense: they cannot prove that trends would have remained parallel after adoption, and noisy pre-period estimates may conceal real differences.

Temporal threats can arise even when the policy plainly cannot cause the placebo outcome. For example, adoption may coincide with an unusual weather period in the adopting districts, while comparison districts experience a different seasonal pattern. A policy announcement could also cause anticipation in outcomes that are not truly pre-exposure, or the analyst could choose pre- and post-windows that cover different calendar months. Plotting group-specific placebo means over several periods, marking policy announcements and implementation dates, can expose abrupt changes or divergent trajectories that a single contrast hides.

Selection and Measurement Can Create Placebo Associations

A policy that cannot change the underlying outcome can still change the observed data. This distinction is important when policies alter eligibility, reporting, service use, follow-up, or the composition of the units in the analysis. A weather station cannot have its rainfall caused by a tutoring policy, but a change in which stations are included can change the group mean. Similar problems occur when a policy changes which people are tested, which records are linked, or which sites report data.

Useful diagnostics include tabulating sample sizes and missingness by group and period, verifying that outcome definitions and data sources remain stable, and recalculating the contrast on a consistent panel when that is substantively appropriate. These checks should not be used to select whichever analysis produces a null placebo result. Instead, report how defensible design choices affect the estimate and explain why each choice addresses a specific threat.

What the Placebo Result Can and Cannot Establish

A nonzero placebo association can reveal a problem with the policy comparison, a failure of the placebo’s no-effect rationale, or ordinary statistical variation. It does not tell the analyst how large the bias in the target-outcome estimate is. The relationship between a bias in the target outcome and a placebo association depends on how the underlying threats affect each outcome; a rainfall contrast cannot generally be converted into a bias correction for mathematics scores.

A null result is not proof that the target estimate is unbiased. The placebo may be measured with substantial noise, may be too weakly related to the sources of residual confounding, or may be affected by a different set of selection mechanisms. Earlier material on power and equivalence for negative control tests is relevant here: a nonsignificant result is not the same as evidence that any meaningful association is absent. Interpret the estimate and its interval, not just whether a p-value crosses a threshold.

A placebo outcome should be compared with other approaches according to the threat each addresses. A placebo treatment asks whether an apparent effect appears at a time or for a group where treatment should not have an effect. A future exposure or future outcome can probe temporal structure in a different way. Multiple controls can provide converging or conflicting evidence, but each must have a defensible causal rationale. These strategies are not interchangeable: the value of any falsification test depends on whether its negative-control condition holds and whether it shares relevant sources of bias with the main analysis.

A useful evaluation treats the placebo analysis as part of a broader design argument: state why the policy cannot affect the outcome, apply the same comparison logic used for the target outcome, inspect the timing and composition of the data, and report uncertainty. The diagnostic is most persuasive when its substantive rationale is clear and its limitations are explicit.