In-Space Placebos and RMSPE Ratios

Why Compare RMSPE Ratios?

A synthetic control analysis compares an observed outcome with the outcome predicted by a weighted combination of donor units. The difference, or gap, is central to the analysis: it shows how far the treated unit is from its synthetic comparison over time. But a post-intervention gap is easier to interpret when we also know how large the gap was before intervention. A unit with poor pre-intervention fit may continue to show large discrepancies even if the intervention had no effect.

In-space placebos provide a comparative check. Each eligible donor unit is treated in turn as if it had received the intervention, and a synthetic comparison is constructed for it. We then compare the treated unit’s post- to pre-intervention root mean squared prediction error (RMSPE) ratio with the corresponding ratios for the placebo units. The comparison asks whether the treated unit’s discrepancy grew unusually much relative to its own pre-intervention fit.

Definition: For unit \(i\) and time \(t\), let \(Y_{it}\) be its observed outcome and \(\hat{Y}_{it}^{\,\mathrm{syn}}\) its synthetic-control outcome. The gap is \(g_{it}=Y_{it}-\hat{Y}_{it}^{\,\mathrm{syn}}\). Let \(T_{\mathrm{pre}}\) and \(T_{\mathrm{post}}\) be the sets of pre- and post-intervention time periods. The pre- and post-intervention RMSPEs are

\mathrm{RMSPE}_{i,\mathrm{pre}} = \sqrt{\frac{1}{|T_{\mathrm{pre}}|}\sum_{t\in T_{\mathrm{pre}}}g_{it}^{\,2}}, \qquad \mathrm{RMSPE}_{i,\mathrm{post}} = \sqrt{\frac{1}{|T_{\mathrm{post}}|}\sum_{t\in T_{\mathrm{post}}}g_{it}^{\,2}}.

The post-to-pre RMSPE ratio is \(R_i=\mathrm{RMSPE}_{i,\mathrm{post}}/\mathrm{RMSPE}_{i,\mathrm{pre}}\), provided the pre-intervention RMSPE is positive. RMSPEs have the units of the outcome; the ratio is dimensionless.

A ratio greater than 1 means the average squared discrepancy, expressed on the RMSPE scale, is larger after intervention than before. A larger ratio for the treated unit than for most placebo units indicates that its post-period divergence is relatively unusual in this comparison. The ratio does not show whether the gap is positive or negative: squaring removes direction. Therefore, interpret ratios alongside the gap trajectory and the substantive meaning of the outcome.

Worked Example: Computing the Treated Unit’s Ratio

Suppose a hypothetical treated region has four pre-intervention gaps, in outcome units, of \(2,-1,1,-2\), and four post-intervention gaps of \(5,6,4,5\). The positive and negative signs record whether the observed outcome is above or below its synthetic comparison.

The pre-intervention RMSPE is

\mathrm{RMSPE}_{\mathrm{pre}} = \sqrt{\frac{2^2+(-1)^2+1^2+(-2)^2}{4}} = \sqrt{\frac{10}{4}} = \sqrt{2.5} \approx 1.581.

The post-intervention RMSPE is

\mathrm{RMSPE}_{\mathrm{post}} = \sqrt{\frac{5^2+6^2+4^2+5^2}{4}} = \sqrt{\frac{102}{4}} = \sqrt{25.5} \approx 5.050.

Thus \(R_{\mathrm{treated}}=5.050/1.581\approx 3.194\). The post-period RMSPE is about 3.19 times the pre-period RMSPE. This is a statement about relative discrepancy, not by itself a conclusion that the intervention caused a particular effect. Here the post gaps are mostly positive, which supplies directional information that the ratio alone cannot provide.

Constructing the In-Space Placebo Comparison

For each eligible donor unit, assign a placebo intervention at the actual intervention date, construct a synthetic comparison using pre-intervention information, and calculate pre- and post-period RMSPEs using that unit’s gaps. The treated unit’s own outcome must not be used as a donor for a placebo if it was exposed to the intervention. Apply the same outcome definition, time windows, and donor-eligibility logic consistently; each placebo unit must also be left out of its own donor pool.

This procedure makes the treated and placebo ratios comparable in design, but it does not make the units identical. In particular, poor pre-intervention fit can make a placebo’s ratio difficult to interpret. A very small denominator can make a ratio enormous even when its post-intervention discrepancy is modest. Conversely, a large pre-intervention RMSPE can suppress the ratio. Fit-based exclusions are intended to avoid treating very poorly matched placebos as informative comparisons; they must be stated clearly rather than chosen after seeing which units support a preferred conclusion.

Worked Example: Applying a Pre-Fit Exclusion and Comparing Ratios

Suppose the treated region has pre-intervention RMSPE \(1.581\) and ratio \(3.194\), as above. Five hypothetical placebo units have the following RMSPEs, all in the outcome’s units. Before examining their post-period results, the analyst specifies a fit rule: exclude a placebo if its pre-intervention RMSPE is greater than 3.0 outcome units.

Unit	Pre RMSPE	Post RMSPE	Ratio	Decision
Placebo A	0.50	1.20	2.40	Retain
Placebo B	0.90	3.60	4.00	Retain
Placebo C	2.40	4.80	2.00	Retain
Placebo D	0.20	1.40	7.00	Retain
Placebo E	4.00	12.00	3.00	Exclude: pre RMSPE exceeds 3.0

For example, Placebo B has ratio \(3.60/0.90=4.00\), and Placebo E has ratio \(12.00/4.00=3.00\). The exclusion is based on the stated pre-fit rule, not on whether a placebo ratio is large or small. Four placebo units remain. Their ratios are \(2.40,4.00,2.00,\) and \(7.00\); two are at least as large as the treated ratio \(3.194\).

A simple rank summary including the treated unit is

\frac{1+\#\{\text{retained placebo ratios}\geq 3.194\}} {1+\#\{\text{retained placebo units}\}} = \frac{1+2}{1+4} = 0.600.

This says that the treated ratio is exceeded or matched by two of the four retained placebo ratios, so it is not especially high in this small comparison set. The rank summary is coarse: with four retained placebos, only a few ranks are possible. It is not automatically an exact randomization p-value. Such an interpretation requires an assignment mechanism that justifies treating the treated unit and placebos as exchangeable under the null; observational assignment to treatment does not supply that justification on its own.

Why the Ratio Normalizes—and What It Does Not Fix

The ratio can be understood by squaring it. Since both RMSPEs are nonnegative,

R_i^2 = \frac{\mathrm{RMSPE}_{i,\mathrm{post}}^2} {\mathrm{RMSPE}_{i,\mathrm{pre}}^2} = \frac{\displaystyle \frac{1}{|T_{\mathrm{post}}|} \sum_{t\in T_{\mathrm{post}}}g_{it}^{\,2}} {\displaystyle \frac{1}{|T_{\mathrm{pre}}|} \sum_{t\in T_{\mathrm{pre}}}g_{it}^{\,2}}.

Thus, the squared ratio compares mean squared gaps across the two periods; taking the square root expresses the comparison on the original outcome scale. A unit whose gaps are generally large in both periods is not automatically ranked as having a large increase. The comparison is relative to that unit’s pre-period discrepancy.

However, normalization is not a cure for unstable or incomparable data. If pre-intervention RMSPE is zero, the ratio is undefined; if it is close to zero, small changes in the post RMSPE can produce a very large ratio. Periods with different numbers of observations are handled by the separate averages in the RMSPE definitions, but differences in outcome variability or time windows can still affect comparisons. Report the pre- and post-RMSPEs as well as the ratio so readers can see the scale behind it.

Worked Example: A Large Ratio From a Small Denominator

Consider two hypothetical placebo units. Unit X has pre RMSPE \(0.05\) outcome units and post RMSPE \(0.50\); Unit Y has pre RMSPE \(1.00\) and post RMSPE \(4.00\). Their ratios are

R_X=\frac{0.50}{0.05}=10, \qquad R_Y=\frac{4.00}{1.00}=4.

Unit X has the larger ratio, even though its post RMSPE is only \(0.50\) outcome units, compared with \(4.00\) for Unit Y. The tenfold ratio for X reflects its exceptionally small pre RMSPE as well as its post fit. It should not be read as evidence that X has a larger absolute post-period discrepancy. Reporting both RMSPEs makes this distinction visible and helps identify when a ratio is dominated by a small denominator.

Interpretation and Reporting

The ratio comparison is most useful as a relative diagnostic for the treated unit’s post-intervention divergence. A treated ratio larger than most retained placebo ratios is evidence that the treated unit’s gap grew more than the gaps of those placebo units, after accounting for their pre-period fit. The strength of this evidence depends on the number and relevance of retained placebos, the quality of their pre-fit, and whether the comparison design is credible. It does not alone establish that the observed divergence is caused by treatment.

A clear report should give the treated unit’s pre RMSPE, post RMSPE, and ratio; the corresponding placebo results; the pre-specified fit exclusion rule; the number of eligible, excluded, and retained placebos; and the rank calculation if one is used. State whether ratios are being compared descriptively or under an assignment-based inference argument. Also show the signed gaps over time or summarize them in context, because RMSPE ratios contain no information about direction.

Common Mistakes

Reading the ratio as an effect size. A ratio of 3 means the post RMSPE is three times the pre RMSPE; it does not mean the treatment increased the outcome by three units or by 200 percent.
Ignoring the denominator. A striking ratio can result from near-perfect pre-fit. Inspect and report both RMSPEs.
Excluding placebos after seeing their post-period ratios. That makes the comparison vulnerable to selective reporting. Specify the fit rule and disclose all exclusions and their pre-fit values.
Treating every rank as a causal p-value. A placebo rank is a relative comparison. Its exact inferential meaning depends on a defensible assignment or exchangeability argument.
Using the ratio to infer direction. Because gaps are squared, the ratio is unchanged if every gap changes sign. Inspect signed gaps to describe whether the observed outcome lies above or below its synthetic comparison.

Key takeaway: In-space placebo RMSPE ratios compare post-intervention discrepancy with each unit’s own pre-intervention fit. They are informative only when the placebo construction and fit exclusions are transparent, denominators are inspected, and the resulting rank is interpreted as relative evidence rather than automatic proof of a causal effect.

Check Your Understanding

Work through these questions using the definitions and examples above.

A unit has pre RMSPE \(2.0\) outcome units and post RMSPE \(6.0\) outcome units. What is its ratio, and what does that ratio mean?
Why can a unit with a larger post RMSPE have a smaller post-to-pre ratio than another unit?
An analyst excludes placebo units only after learning that their ratios are inconvenient. What is the problem, and how should exclusions instead be handled?
Can an RMSPE ratio reveal whether the post-intervention gap is positive or negative? Explain.
What additional condition is needed before interpreting a placebo rank as an exact randomization p-value?