A Synthetic Control Case-Study Workshop

From a Policy Question to a Comparative Case Study

A synthetic control case study is useful when one unit receives an intervention and no single comparison unit is a credible stand-in for its untreated trajectory. The analyst constructs a weighted combination of untreated units, then compares the treated unit with that synthetic comparison over time. The calculation is only one part of the analysis: the intervention, comparison units, pre-treatment fit, and uncertainty assessment all affect what the estimated gaps can support.

This workshop uses a hypothetical example. A city, Harbor, introduces a fare-free public bus policy in 2016. The outcome is annual bus trips per resident. The study asks how ridership in Harbor changed after the policy relative to the ridership trajectory that might have occurred without it. The numbers below are invented for instruction and do not describe a real city or policy.

Specify the Intervention and the Donor Pool

The treated unit is Harbor, the intervention is the introduction of fare-free bus service, and the treatment date is the start of 2016. The outcome is annual bus trips per resident, measured consistently for every city and year. The pre-treatment period is 2012–2015; the post-treatment period is 2016–2019. Defining these elements before inspecting post-treatment gaps prevents the analyst from choosing a date or outcome window to make the result look favorable.

The donor pool should contain units that did not receive the intervention during the study period and that could plausibly help represent Harbor’s untreated trajectory. It is not simply a list of cities with available data. For example, the analyst would check whether a donor changed its fare policy, had a major transit disruption, or was exposed to an intervention that directly affected its ridership. Donor contamination and spillovers require particular care: units influenced by Harbor’s policy may no longer represent untreated comparisons.

A causal interpretation also requires substantive assumptions. In particular, without the intervention, a weighted donor combination that fits Harbor before 2016 must remain informative about Harbor’s post-2015 outcome path. The intervention should not have affected Harbor before its recorded start, outcome measurement should be comparable across units and time, and the selected donors should not be affected by Harbor’s treatment. Good pre-treatment fit is useful evidence about the comparison, but it does not prove these assumptions.

Fit Weights and Inspect the Pre-Treatment Match

The weights are nonnegative and sum to one, so the synthetic control is a convex combination of the donor outcomes. In practice, they are chosen to make the synthetic unit resemble Harbor on pre-treatment outcomes and, where appropriate, pre-specified predictors of the outcome. Predictor scaling and selection matter: variables measured on very different scales should not implicitly dominate the fit just because their numeric values are large. The weights and tuning choices must be based on pre-treatment information, not adjusted to improve the post-treatment result.

For this example, suppose the chosen pre-treatment fitting procedure gives weights \(0.50\) for Aster, \(0.30\) for Birch, and \(0.20\) for Cedar. These weights are nonnegative and sum to one. The donor outcomes and resulting synthetic outcomes are:

For example, the 2012 synthetic outcome is \(0.50(38)+0.30(43)+0.20(47)=41.3\) trips per resident. The pre-treatment gap is Harbor’s outcome minus the synthetic outcome, so a positive gap means Harbor’s ridership was higher in that year.

Pre-treatment diagnostics should include the plotted outcome paths, the gaps, and important predictors used to construct the fit. A single summary statistic can hide a poor match in one year or a difference in trend. Whether a discrepancy is acceptable depends on the outcome scale, the length and variability of the pre-period, and how large the post-treatment gaps are. A post-treatment difference that is only modestly larger than the pre-treatment discrepancies deserves more caution than one that is large relative to them.

Estimate and Plot Post-Treatment Gaps

Once the design and weights are fixed, calculate the synthetic outcome in each post-treatment year with the same weights. The estimated gap is the observed Harbor outcome minus its synthetic counterpart. In this case, it is an estimated treatment effect for Harbor in that year only under the counterfactual assumptions stated above. It is not automatically an average effect for other cities or policies.

A useful graph displays the observed and synthetic outcome paths together, with the intervention date marked, and a separate graph displays the yearly gaps with a zero reference line. The first plot shows whether the trajectories diverge; the second makes the direction, size, and persistence of the estimated differences easier to read. State the outcome units and treatment date on the plots, and do not hide pre-treatment gaps by showing only the post-treatment period.

Placebo-Based Uncertainty and Its Limits

With one treated unit, conventional uncertainty summaries based on many independently treated units may not directly describe the study’s design. A common diagnostic is an in-space placebo exercise: in turn, treat each eligible donor as if it had received the intervention in 2016, construct its synthetic comparison from the remaining donors, and calculate a comparable post-treatment statistic. The exercise asks whether Harbor’s post-treatment divergence is unusual relative to divergences observed for units assigned a false treatment date.

One possible statistic is the ratio of post-treatment RMSPE to pre-treatment RMSPE. It scales post-treatment divergence by the quality of the pre-treatment match. A large ratio can indicate an unusual post-period gap relative to the unit’s own pre-period fit, although it can also be large when pre-treatment RMSPE is exceptionally small. The statistic and any rules for excluding poorly fitted placebo units should be specified and reported transparently.

Placebo comparisons are informative only to the extent that the treated and placebo units are reasonably comparable and the same procedure is applied to each. They do not recreate random assignment. A small rank-based value is not the probability that the null hypothesis is true, and it does not remove concerns about a unique shock to Harbor, donor selection, or poor pre-treatment fit. Conversely, an unremarkable placebo rank does not prove the intervention had no effect; it may reflect limited comparison units or limited ability to distinguish the treated path from placebo paths.

Reporting the Case Study Carefully

A concise report should make the analysis reproducible and the causal claim appropriately bounded. It should identify the treated unit, intervention and date, outcome and units, pre- and post-periods, donor eligibility rules, predictor choices, and fitted weights. It should show pre-treatment fit diagnostics alongside the post-treatment outcome paths and gaps. If placebo-based uncertainty is reported, describe the placebo construction, statistic, exclusions, and the number of comparison units.

For the hypothetical Harbor analysis, a cautious summary would be: “Using weights of 0.50, 0.30, and 0.20 on Aster, Birch, and Cedar, the synthetic control had a pre-treatment RMSPE of 0.770 trips per resident. Harbor’s estimated post-treatment gaps were 2.6 to 3.6 trips per resident per year, with a mean of 3.35. Harbor’s post-to-pre RMSPE ratio exceeded the three donor placebo ratios, but the rank calculation was 0.25 and the placebo set was small. These estimates are consistent with increased ridership after the policy, conditional on the synthetic control providing a credible untreated counterfactual; the design cannot rule out other Harbor-specific changes.”

Common Mistake

Wrong reasoning: “The synthetic control closely matched Harbor before 2016, and the post-treatment gaps are positive, so the fare-free policy caused an increase of 3.35 trips per resident.” This treats a pre-treatment match as proof that the post-treatment counterfactual is correct and ignores other changes that may have affected Harbor.

Correction: The mean gap of 3.35 trips per resident per year is the estimated difference between Harbor and its synthetic comparison over 2016–2019. It has a causal interpretation only if the comparison remains credible after treatment and relevant assumptions hold. Report the estimate, diagnostics, placebo evidence, and limitations together; do not turn a favorable fit or a post-treatment divergence into certainty.