Diagnosing Pre-Intervention Fit and RMSPE

Why Diagnose Fit Before Interpreting a Synthetic Control?

A synthetic control is a weighted combination of untreated donor units intended to approximate the treated unit in the absence of intervention. Before interpreting the post-intervention gap as an effect, examine whether that combination actually reproduced the treated unit during the pre-intervention period. A synthetic control that misses the treated unit’s earlier outcome level or trajectory offers a weaker basis for the counterfactual comparison.

Pre-intervention fit is a diagnostic, not a guarantee of causal validity. Good fit shows that the selected donors and weights can reproduce certain observed features before treatment. It does not, by itself, establish that the treated unit would have continued to track the synthetic unit after intervention, or rule out other threats to identification. Conversely, poor fit is a reason to be cautious: the estimated post-intervention difference may partly reflect a mismatch that was already present.

Definition: Let unit 1 be the treated unit, and let $j=2,\ldots,J+1$ index untreated donor units. Let $Y_{1t}$ be the treated unit’s outcome in period $t$, and let $Y_{jt}$ be donor $j$’s outcome. A synthetic control uses weights $w_j$ satisfying $w_j \geq 0$ and $\sum_{j=2}^{J+1}w_j=1$. Its outcome in period $t$ is $\widehat{Y}_{1t}^{\,SC}=\sum_{j=2}^{J+1}w_jY_{jt}$. For each of the $T_0$ pre-intervention periods, the gap is $g_t=Y_{1t}-\widehat{Y}_{1t}^{\,SC}$. The pre-intervention root mean squared prediction error is $\mathrm{RMSPE}_{pre}=\sqrt{\frac{1}{T_0}\sum_{t=1}^{T_0}g_t^2}$.

RMSPE summarizes the typical size of the pre-intervention outcome gaps. Because the gaps are squared, positive and negative errors do not cancel, and larger errors contribute more than smaller ones. RMSPE has the same units as the outcome: an RMSPE of 2.4 means a typical discrepancy of about 2.4 outcome units, not 2.4 percent unless the outcome itself is expressed as a percentage. Whether that is small or large depends on the outcome’s scale and on the substantive question.

Calculating and Interpreting Pre-Intervention RMSPE

Worked Example: Computing RMSPE From Outcome Gaps

Suppose a hypothetical region is treated after four pre-intervention years. Its observed outcome is measured in cases per 10,000 residents. The treated and synthetic outcomes are as follows.

Pre-period	Treated outcome	Synthetic outcome	Gap $g_t$	$g_t^2$
1	100	98	2	4
2	104	103	1	1
3	102	103	-1	1
4	108	106	2	4

There are $T_0=4$ pre-intervention periods. The squared gaps sum to $4+1+1+4=10$, so

\mathrm{RMSPE}_{pre} =\sqrt{\frac{4+1+1+4}{4}} =\sqrt{2.5} \approx 1.581.

The synthetic outcome differs from the treated outcome by about 1.581 cases per 10,000 residents in root-mean-square terms across these four pre-intervention years. The gaps are small relative to outcome levels around 100, but that alone does not decide whether fit is adequate. A difference of this size might matter greatly for a rare outcome or a small policy effect. The individual gaps and outcome context are needed for interpretation.

RMSPE is most informative when reported alongside the time series of outcomes and gaps. A single summary can obscure whether errors are isolated or persistent, whether they change direction, and whether the synthetic trajectory tracks important changes in the treated trajectory. The comparison should cover the full pre-intervention window used to build and assess the synthetic control.

Inspecting Trajectories, Predictors, and Residuals

A practical diagnosis uses several complementary views. First, plot the treated and synthetic outcomes over the pre-intervention period on the same scale. Look for agreement in level and movement: do peaks and declines occur at similar times, and do the series move together? Then plot the gaps $g_t$ over time. This makes persistent offsets, drift, and unusually large period-specific discrepancies easier to see than a single RMSPE can.

Second, compare pre-intervention predictors for the treated unit with the same predictors aggregated using the synthetic-control weights. For predictor $k$, the synthetic value is $\widehat{X}_{1k}^{\,SC}=\sum_{j=2}^{J+1}w_jX_{jk}$, and the balance difference is $X_{1k}-\widehat{X}_{1k}^{\,SC}$. Interpret each difference in its own units and context. A difference of 0.2 may be negligible for one predictor and consequential for another. If scales differ substantially, a standardized difference can aid comparison, provided the scale used for standardization is stated and appropriate. Standardization does not make substantively important imbalance disappear.

Third, inspect residual patterns rather than treating all gaps as interchangeable. A long run of positive gaps indicates that the synthetic control persistently underpredicts the treated outcome; a long run of negative gaps indicates persistent overprediction. A gradual increase or decrease in gaps suggests the two trajectories diverge over time. Alternating gaps around zero with no visible trend are generally more reassuring, although they do not prove that post-intervention behavior will match.

Worked Example: Checking Predictor Balance in Context

Suppose a hypothetical treated region is matched using a weighted combination of donor regions. Three pre-intervention predictors are unemployment rate (percentage points), annual income (thousands of dollars), and hospital beds per 1,000 residents. The following values are observed for the treated unit and its synthetic control.

Predictor	Treated value	Synthetic value	Treated minus synthetic
Unemployment rate	6.0 percentage points	6.1 percentage points	-0.1 percentage points
Annual income	52.0 thousand dollars	51.5 thousand dollars	0.5 thousand dollars
Hospital beds	4.0 per 1,000 residents	4.2 per 1,000 residents	-0.2 per 1,000 residents

For unemployment, the difference is $6.0-6.1=-0.1$ percentage points. For income, it is $52.0-51.5=0.5$ thousand dollars, or $500. For hospital beds, it is $4.0-4.2=-0.2$ beds per 1,000 residents. The signs show whether the treated value is below or above its synthetic counterpart; the units show the size of each difference.

These results suggest close balance on the three listed predictors, but they do not establish that every relevant predictor is balanced or that the outcome trajectory is well matched. A full diagnosis would also inspect the pre-intervention outcomes and other predictors used in constructing the synthetic control. The interpretation must not treat numerically similar values as proof that the units are equivalent in all respects.

Why the RMSPE Can Hide Important Patterns

The RMSPE squares each gap and averages the results. This has a useful consequence: it avoids cancellation between positive and negative gaps. But it also means that two very different sequences can have the same RMSPE. For example, gaps that alternate around zero may produce the same summary as gaps that are positive for several years and negative for several years. The latter pattern could indicate a temporal mismatch even when the overall RMSPE looks modest.

A short decomposition clarifies how systematic offset contributes to RMSPE. Let $\bar{g}=\frac{1}{T_0}\sum_{t=1}^{T_0}g_t$ be the mean pre-intervention gap. Add and subtract $\bar{g}$ inside each squared gap, then expand:

\begin{aligned} \frac{1}{T_0}\sum_{t=1}^{T_0}g_t^2 &=\frac{1}{T_0}\sum_{t=1}^{T_0}\bigl[(g_t-\bar{g})+\bar{g}\bigr]^2\\ &=\frac{1}{T_0}\sum_{t=1}^{T_0}(g_t-\bar{g})^2 +\frac{2\bar{g}}{T_0}\sum_{t=1}^{T_0}(g_t-\bar{g}) +\bar{g}^{\,2}\\ &=\frac{1}{T_0}\sum_{t=1}^{T_0}(g_t-\bar{g})^2+\bar{g}^{\,2}. \end{aligned}

The middle term is zero because $\sum_{t=1}^{T_0}(g_t-\bar{g})=0$. Thus, mean squared gap is the average squared deviation around the mean gap plus the squared mean gap. A persistent level difference increases RMSPE even if the gaps fluctuate little from year to year. Conversely, a near-zero mean gap does not guarantee a small RMSPE if the gaps vary widely.

Worked Example: Same RMSPE, Different Residual Patterns

Consider two hypothetical eight-year pre-intervention gap sequences, measured in the same outcome units:

A=(1,-1,1,-1,1,-1,1,-1),\qquad B=(1,1,1,1,-1,-1,-1,-1).

For sequence A, every squared gap is 1, so $\mathrm{RMSPE}_{pre}=\sqrt{(1+1+1+1+1+1+1+1)/8}=1$. For sequence B, every squared gap is also 1, giving $\mathrm{RMSPE}_{pre}=\sqrt{(1+1+1+1+1+1+1+1)/8}=1$. The summaries are identical.

But the sequences tell different diagnostic stories. Sequence A alternates around zero. Sequence B has four consecutive positive gaps followed by four consecutive negative gaps, a pronounced change in direction. That pattern could reflect a mismatch in trajectory or a change in fit during the pre-period. RMSPE alone cannot reveal that structure. Plotting the gaps and checking the timing of divergence are therefore essential parts of the assessment.

RMSPE also depends on the outcome’s measurement scale. If the outcome is converted from cases to cases per 100,000 residents, its numerical RMSPE changes by the same conversion factor, even though the fit has not changed. When comparing fit across outcomes or settings, state the units and consider the size of the errors relative to meaningful variation in the outcome. Do not use a universal threshold for “good” RMSPE without a defensible context-specific rationale.

Putting the Diagnostics Together

A credible assessment combines numerical summaries with substantive inspection. The following sequence keeps the interpretation focused on what the pre-intervention data can show.

Calculate the pre-period gaps and RMSPE.
Use all designated pre-intervention periods, report the outcome units, and retain the individual gaps rather than reporting only their summary.

Compare the outcome trajectories.
Check whether treated and synthetic outcomes track in level and movement, and identify periods where they visibly diverge.

Review predictor balance.
Report treated and synthetic predictor values, their differences, and units; interpret size relative to the predictor’s context.

Look for structure in the gaps.
Check for sustained signs, drift, or clusters of large discrepancies that a single average can conceal.

State the limitation.
Describe pre-intervention fit as evidence about the quality of observed matching, not as proof of the unobserved post-intervention counterfactual.

Common Mistakes

Treating a small RMSPE as proof of identification. A close pre-period match does not establish that the synthetic unit supplies the correct post-intervention counterfactual. It supports the design’s credibility on observed pre-period features but cannot settle all identification concerns.
Reporting RMSPE without units or scale. “RMSPE = 2” is incomplete. State whether the outcome is, for example, cases per 10,000 residents, dollars, or percentage points, and assess whether two units are substantively small.
Looking only at the overall summary. Equal RMSPE values can accompany different residual patterns. Inspect the time-indexed gaps for persistent offsets, runs, and drift.
Calling close predictor values complete balance. Balance is about the predictors examined and their meaningful differences. It does not imply equality on unmeasured features or guarantee outcome fit.
Interpreting fit as a causal effect. Pre-intervention gaps diagnose the match before treatment; they are not themselves estimates of the intervention’s causal effect.

Key takeaway: Pre-intervention RMSPE quantifies the root-mean-square size of the outcome gaps, but it is only one diagnostic. Judge synthetic-control fit by combining RMSPE with trajectory plots, predictor differences, and the timing and pattern of residual gaps. Good pre-intervention fit is reassuring evidence, not a guarantee of causal validity.

Check Your Understanding

Answer each question using the definitions and diagnostic principles above.

A synthetic control has pre-intervention gaps of $0$, $2$, and $-2$ outcome units. Calculate its pre-intervention RMSPE and state its units.
Why can a sequence with a persistent positive gap be concerning even if its RMSPE is not large relative to the outcome level?
Two synthetic controls have the same RMSPE, but one has alternating gaps and the other has a long run of positive gaps followed by negative gaps. What additional diagnostic information would you inspect?
A predictor difference is reported as 0.3. What additional information is needed before judging whether that difference is small?
Does close pre-intervention outcome fit prove that the synthetic control gives the correct post-intervention counterfactual? Explain briefly.