Sensitivity to Pre-Period and Predictor Choices

Why Examine Pre-Period and Predictor Choices?

A synthetic control is constructed by choosing weights on donor units so that their weighted outcomes and predictors resemble those of the treated unit before intervention. The construction depends on design choices: which pre-intervention observations enter the fit, which predictors are included, and how much importance each predictor receives. Different defensible choices can produce different weights, different pre-intervention fit, and different estimated post-intervention effects.

Sensitivity analysis makes those dependencies visible. It does not identify which specification is true, nor does it turn a set of estimates into a confidence interval. Its purpose is to show whether the substantive conclusion is stable across choices that could reasonably have been made using pre-intervention information and subject-matter knowledge.

What Changes Across Specifications?

Let unit 1 be treated, let \(j=2,\ldots,J+1\) index donor units, and let \(t=1,\ldots,T_0\) denote pre-intervention periods. The treatment begins after period \(T_0\). Under a specification \(s\), the synthetic-control weights \(w_j^{(s)}\) are nonnegative and sum to one. The synthetic outcome for period \(t\) is \(\sum_{j=2}^{J+1}w_j^{(s)}Y_{jt}\), where \(Y_{jt}\) is the observed outcome for donor \(j\).

The weight-fitting problem uses selected predictors and a selected pre-intervention window. Write \(X_1^{(s)}\) for the treated unit's predictor vector and \(X_0^{(s)}\) for the corresponding donor predictor matrix. A common constrained objective is:

Here \(V^{(s)}\) assigns relative importance to the predictors. The selected window may affect which outcome summaries enter the predictor vector, as well as how the outcome fit is assessed. Changing any of these ingredients can change the minimizing weights. The synthetic-control construction and its underlying identification assumptions are not replaced by sensitivity analysis: a causal interpretation still requires a defensible donor pool, no interference, no anticipation, and a synthetic control that adequately represents the treated unit's untreated outcome path.

For a chosen set of pre-intervention periods \(W\), define the root mean squared prediction error (RMSPE) as the root mean squared difference between the treated outcome and its synthetic counterpart:

The period-specific estimated effect after intervention is the treated outcome minus the synthetic outcome:

For a post-intervention set \(P\) of \(T_1\) periods, the average estimated effect is \(\frac{1}{T_1}\sum_{t\in P}\hat{\tau}_t^{(s)}\). It is expressed in the outcome's units. RMSPE is also in the outcome's units, but it describes pre-intervention discrepancies, not post-intervention effects.

Compare Fit on a Common Basis

When specifications use different fitting windows, their in-window RMSPE values do not summarize performance over identical periods. A shorter window may have low RMSPE partly because it contains fewer or easier-to-match observations. For a useful comparison, report RMSPE within each specification's fitting window and, when possible, calculate RMSPE for every specification over the same pre-intervention assessment periods. The latter comparison asks how well each fitted synthetic control matches the treated unit over a fixed period, even if some of those observations were not used to fit it.

The distinction is especially important when the selected window emphasizes recent periods. Fit over that window answers whether the synthetic control matches the data it was asked to match. Fit over a common assessment period helps show whether that match extends beyond the fitting target. Neither summary alone establishes that the synthetic control correctly represents the unobserved counterfactual after intervention.

Vary Predictors and Their Importance

A predictor set should be chosen because its components help describe the treated unit's pre-intervention outcome process or substantively relevant characteristics—not because a particular choice produces a preferred effect estimate. A sensitivity analysis can compare a primary set with reasonable alternatives, such as omitting a questionable measure, adding a pre-specified predictor, or altering the relative importance assigned to predictors. The alternatives should be defined using pre-intervention information.

Predictor importance and predictor inclusion are distinct choices. Including a predictor says it contributes to the fitting objective; its importance determines how strongly mismatch on that predictor counts relative to mismatch on others. Since predictor scales can differ, a meaningful comparison requires attention to scaling and to the interpretation of the importance scheme. Otherwise, numerical scale alone can make a predictor dominate the objective.

When a predictor set changes, do not compare only its fitting objective value with the objective from another set: the objectives may contain different components. Instead, compare interpretable diagnostics such as outcome RMSPE over the same pre-intervention assessment periods, and compare post-intervention gaps or their average over the same post-intervention periods.

A Practical Sensitivity Analysis

A useful analysis varies one design feature at a time before considering combinations. This makes it easier to see whether an estimate changes with the window, predictor set, or importance scheme. Afterward, a small set of justified combinations can show whether the conclusions are robust to joint changes. A large grid of arbitrary alternatives is usually less informative than a focused set with a clear rationale.

The following hypothetical summary illustrates how to report differences without hiding them. Each average is calculated over the same three post-intervention periods; the common-period RMSPE is also assessed over the same four pre-intervention periods.

For the primary specification, the average is \((-1.0-1.2-1.4)/3=-1.20\) outcome units. For the reduced predictor set, it is \((-0.9-1.1-1.15)/3=-1.05\) outcome units. These estimates have the same direction and similar magnitude, while the higher-covariate-importance specification produces a smaller average gap and worse common-period fit. A careful conclusion would report that the estimated effect is sensitive to predictor importance, rather than choosing the specification with the most favorable estimate or fit alone.

Why This Matters—and a Common Pitfall

Pre-intervention fit is a diagnostic, not a guarantee of post-intervention counterfactual accuracy. A specification can fit observed pre-intervention outcomes closely and still extrapolate poorly if the relationship between treated and donor units changes after intervention or if relevant features are not represented. Conversely, a modest increase in RMSPE does not automatically make a design unusable. Evaluate fit alongside the rationale for the design, the pattern of pre-intervention gaps, the donor pool, and the post-intervention trajectory.

A frequent mistake is to try many windows and predictor schemes, then present only the one that gives the smallest RMSPE or the most appealing post-intervention estimate. This uses design flexibility to select a result and conceals how much the conclusion depends on those choices. The correction is to justify the primary design independently of its post-intervention estimate, define a limited set of reasonable alternatives, and report the full set of results. If the estimates vary materially, that variation is part of the substantive finding and should temper the causal conclusion.