Placebo Inference for Synthetic Control Studies

Why Placebo Comparisons Matter

A synthetic control estimates what would have happened to a treated unit without treatment by combining untreated donor units. After treatment begins, the difference between the treated outcome and its synthetic counterpart is an estimated gap. The central inferential question is whether that gap is unusually large relative to gaps that could arise even when no treatment effect is present.

Placebo comparisons address this question by repeating the synthetic-control procedure where no treatment effect should occur. In an in-space placebo, an untreated unit is temporarily treated as if it had received the intervention. In an in-time placebo, the intervention date is moved to an earlier period when the treated unit had not yet received treatment. These checks can reveal whether large gaps are common under the design, but they do not automatically create a randomized experiment or guarantee a valid p-value.

A placebo analysis requires choices made before inspecting the resulting ranks: which units or dates are eligible, how the synthetic controls are fit, which post-period summary is the test statistic, and how ties are counted. The statistic should represent the substantive departure of interest. It might be the average signed gap over a specified horizon, the average absolute gap, a post-treatment root mean squared prediction error (RMSPE), or a post-to-pre RMSPE ratio.

In-Space Placebos and Rank-Based Comparisons

For an in-space comparison, the actual treated unit is evaluated alongside eligible untreated units. For each eligible placebo unit \(k\), pretend it was treated at the actual intervention date. Fit a synthetic control for \(k\) using its pre-treatment outcomes and an appropriate set of untreated donors, then compute the same post-period statistic used for the actual treated unit. The treated unit itself must not be used as a donor for placebo units, since its post-treatment outcomes may be affected by treatment.

Let \(S_1\) be the observed statistic for the treated unit and \(S_2,\ldots,S_{M+1}\) the statistics for \(M\) eligible placebo units. For a statistic where larger values are more extreme, the common add-one rank-based placebo p-value is:

The numerator counts the treated unit itself plus placebo statistics at least as large as its statistic. The denominator is the treated unit plus the \(M\) placebo units. The weak inequality includes ties. For a statistic where smaller values are more extreme, the comparison must instead count placebo statistics no greater than the treated statistic. For a signed statistic, the direction of the alternative must be chosen in advance; for a magnitude statistic, such as an absolute gap or RMSPE ratio, larger values indicate more extreme departures in either direction.

This is an empirical rank, not automatically a conventional sampling-theory p-value. It has a randomization-inference interpretation when the assignment mechanism justifies comparing the treated unit with the placebo assignments under the null. In observational synthetic-control applications, treatment assignment is generally not randomized among the units. The rank is therefore best understood as evidence relative to the selected placebo distribution unless a defensible assignment or exchangeability argument supports stronger interpretation.

Why Pre-Treatment Fit and Comparability Matter

The ratio statistic is often used because a given post-treatment gap can mean different things for units that had very different levels of pre-treatment prediction error. A short justification makes the point. If an outcome is measured in a different unit, such as dollars rather than thousands of dollars, both pre- and post-treatment RMSPE are multiplied by the same positive conversion factor \(c\). Their ratio is unchanged:

The ratio is dimensionless and measures how much prediction error grows after the placebo or actual intervention date relative to its pre-period level. But normalization cannot repair a poor synthetic fit. If the denominator is very large, a ratio may look small despite a substantively large post-treatment gap; if the denominator is close to zero, a modest post-period error can produce an enormous ratio. Always inspect pre-treatment fit and post-treatment gaps themselves, rather than relying on the ratio alone.

A credible in-space comparison requires that placebo units be plausible comparisons for the treated unit. Their pre-treatment outcome paths should be fit well enough for post-period deviations to be informative, and the donor pools, predictor choices, time windows, and fitting procedure should be defined consistently. Differences in outcome scale, available predictors, or donor support may make nominally identical statistics incomparable. Treatment must also be absent from placebo units, and spillovers from the actual intervention must not contaminate their outcomes.

In-Time Placebos

An in-time placebo asks whether the synthetic control would have shown a similarly large gap before the actual intervention. Choose a fake intervention date that falls within the genuine pre-treatment period. Refit the synthetic control using only observations before that fake date, then compute the chosen statistic over a specified placebo post-period that remains untreated. The fake post-period should have a horizon comparable to the real post-period when feasible; using a much shorter or structurally different window can change the distribution of the statistic.

An in-time gap is not a placebo treatment effect. It is a diagnostic for pre-existing mismatch, unstable fit, or an outcome change that the synthetic control fails to track. A large gap at a fake date weakens the case that the actual post-treatment gap is uniquely associated with treatment timing. A small fake-date gap is reassuring about that particular period, but cannot establish that all alternative explanations are absent.

Conditions for Interpreting the Evidence

Placebo ranks are persuasive only to the extent that the treated unit and placebo comparisons are alike in the ways relevant to the statistic. Several conditions deserve explicit attention:

• Good pre-treatment fit: the synthetic control should reproduce the treated unit’s relevant pre-treatment outcomes. Placebo units with visibly poor fits are not informative benchmarks for a well-fit treated unit, and a poor treated-unit fit undermines interpretation of its own post-period gap.
• Comparable construction: use a consistent fitting method, outcome definition, predictor selection, and post-period statistic. Differences in donor availability or fit quality should be disclosed rather than hidden by a rank.
• Untreated placebos and donors: placebo units and donor units should not receive the intervention during the relevant comparison period. Spillovers can invalidate this condition even when a unit is formally untreated.
• Appropriate timing: in-time placebo windows should be genuinely untreated and should not overlap the actual treatment period. The statistic should be calculated over a meaningful, preferably comparable, horizon.
• A defensible inferential comparison: a rank-based p-value has a formal randomization interpretation only if the assignment process makes the treated and placebo assignments exchangeable under the null. In observational settings, that condition is not supplied by the calculation itself.

When the purpose is diagnostic rather than design-based inference, say so plainly. Report the placebo unit or date set, the statistic, the fit screening rule, and the observed ranks. Include the underlying gaps or ratios so readers can see whether the result depends on a single comparison or a questionable denominator. If eligibility or fit thresholds are chosen after viewing the results, the resulting rank can be selectively favorable and should not be presented as if it arose from a pre-specified reference distribution.

Common Mistakes

• Calling a placebo rank a randomized-experiment p-value: the formula alone does not make treatment assignment exchangeable. Explain the assignment basis or describe the result as a diagnostic rank.
• Including badly fit placebos without qualification: an extreme statistic may reflect poor pre-treatment prediction rather than a meaningful comparison. Show pre-treatment fit and state how eligibility was determined.
• Using the treated unit as a donor for placebo units: post-treatment outcomes of that unit may be affected by treatment. Exclude it from placebo donor pools.
• Comparing different statistics or windows: a rank is meaningful only when the treated and placebo statistics are constructed comparably.
• Misreading the rank: a value such as \(0.25\) is not the probability that the null is true. It is the observed tail rank in the specified placebo comparison, subject to the design’s assumptions.