Pre-Treatment Fit Diagnostics for Synthetic Controls

Why Diagnose Fit Before Interpreting Gaps?

A synthetic control estimates the untreated outcome path of a treated unit by combining outcomes from untreated donor units. The post-intervention difference between the treated outcome and this weighted comparison is meaningful only to the extent that the comparison provides a credible counterfactual. A close match before intervention is therefore an important diagnostic: it shows whether the weighted donors can reproduce the treated unit’s observed history over the period when both are untreated.

That check is necessary, but not sufficient. A synthetic control can match the past and still fail to represent what would have happened after intervention without treatment. Diagnostics should therefore be read as evidence about the construction and limitations of the comparison, not as proof of the causal assumptions or of a post-intervention effect.

The weights are treated as fixed while diagnostics are calculated. Their selection and tuning are separate decisions; here the question is how to inspect the resulting fit. The comparison also presumes that the donor units are suitable untreated comparisons, that outcomes are measured consistently, and that treatment does not begin affecting the treated unit before \(T_0+1\). Diagnostics cannot repair a donor pool that violates those design requirements.

Outcome Fit: Magnitude and Scale

The basic summary is the pre-treatment root mean squared prediction error (RMSPE). It combines the squared residuals across the pre-intervention periods and expresses the result in the outcome’s original units.

Squaring prevents positive and negative residuals from cancelling; taking the mean gives equal weight to each pre-intervention period; and the square root returns the measure to the outcome scale. RMSPE is sensitive to large errors because those errors are squared. It is not a universal pass/fail criterion: its practical importance depends on the outcome’s scale, the research question, and how large a mismatch would matter in context. Residuals should also be inspected individually or over time, because a single summary can conceal their pattern.

RMSPE should be compared cautiously across outcomes measured in different units or on very different scales. A discrepancy of 0.5 may be substantial for a rate measured between zero and one, but small for annual revenue measured in millions. A scale-normalized summary can sometimes help, but only if its denominator is clearly specified and meaningful. For example, dividing RMSPE by the treated unit’s pre-intervention standard deviation expresses error relative to its historical variation; this ratio is undefined if that standard deviation is zero and may be unstable when it is very small.

Predictor Balance

Matching outcomes over time is not the only diagnostic. Synthetic-control construction also uses pre-intervention predictors, such as covariates or summaries of earlier outcomes. For predictor \(k\), let \(X_{1k}\) denote its value for the treated unit and \(X_{jk}\) its value for donor \(j\). The synthetic predictor value is \(\widehat{X}_{1k}=\sum_j w_jX_{jk}\), and the raw balance gap is \(d_k=X_{1k}-\widehat{X}_{1k}\). A gap close to zero indicates balance on that predictor.

Raw gaps retain the predictor’s units, which aids interpretation but complicates comparisons across predictors. One common descriptive measure is the absolute standardized gap, \(|d_k|/s_k\), where \(s_k\) is a stated reference standard deviation, often the standard deviation among donor units. It describes the gap relative to that reference spread; it is not a hypothesis test or a universal balance threshold. If \(s_k=0\), this standardization is undefined, so the raw gap must be reported instead.

Balance is best examined across all substantively important predictors rather than summarized by one favorable variable. A close average can also conceal offsetting discrepancies in different predictors or a poor match on an especially important predictor. Report raw gaps when their units are informative, and explain the standardization when comparing magnitudes.

Residual Patterns Over Time

The time sequence of residuals provides information that RMSPE cannot. A residual plot places \(e_t\) against pre-intervention time, with a horizontal reference at zero. Residuals fluctuating around zero without an evident trend are different from residuals that remain mostly positive, mostly negative, or move steadily in one direction. A persistent positive residual means the treated outcome exceeds the synthetic outcome in those periods; a persistent negative residual means it falls below it.

Serial dependence matters for interpretation. Nearby residuals may be related because outcomes evolve over time. A sequence of small, same-signed residuals can reveal sustained underprediction or overprediction even when the overall RMSPE seems modest. A distinct mismatch near the end of the pre-period can also be especially concerning: the model may reproduce earlier levels but fail to track the treated unit as intervention approaches. Neither a visual pattern nor a small RMSPE alone establishes why the mismatch occurred.

Donor-Weight Concentration

A synthetic control may be built from many donors but rely heavily on only one or two. The Herfindahl concentration measure \(H=\sum_j w_j^2\) summarizes how concentrated the weights are. A useful descriptive transformation is the effective number of donors, \(N_{\mathrm{eff}}=1/H\). If \(m\) donors receive equal weights \(1/m\), then \(H=m(1/m)^2=1/m\), so \(N_{\mathrm{eff}}=m\). With unequal weights, this value is generally smaller than the number of donors with positive weight.

The effective number is an index of weight concentration, not a literal count of independent comparison units. A low value signals that the fit depends heavily on a small part of the donor pool. That concentration may be appropriate if one donor is unusually similar to the treated unit, but it can make the counterfactual sensitive to that donor’s particular trajectory. The concentration measure should be read alongside outcome fit, predictor balance, and substantive knowledge of the donors.

Putting the Diagnostics Together

A useful assessment combines evidence rather than applying a single cutoff. The following sequence keeps the distinct questions visible:

Poor pre-treatment fit means the synthetic control does not closely reproduce the treated unit’s observed pre-intervention history. A post-intervention gap from such a fit may partly reflect a mismatch that already existed, rather than a change attributable to treatment. Unstable-looking fit—such as a late residual trend or heavy reliance on one donor—also makes the counterfactual more vulnerable to departures from the observed pre-period relationship. In these cases, the estimated gap should be presented with stronger qualification, and the source of the mismatch should be investigated rather than hidden by one summary statistic.

Common Mistake: Treating Good Fit as Proof

Wrong reasoning: “The pre-treatment RMSPE is small, so the post-treatment gap must be the causal effect.” This conclusion goes beyond what the diagnostic establishes. Good pre-period fit shows agreement in observed outcomes before treatment; it cannot guarantee that the weighted donors would have continued to trace the treated unit’s untreated path afterward. Conversely, poor fit does not prove there is no treatment effect. It indicates that this synthetic comparison is a weakly supported counterfactual and that its post-treatment gaps require caution.