Uncertainty Assessment for Augmented Synthetic Control

Why Uncertainty Needs More Than a Point Estimate

An augmented synthetic control (ASC) analysis produces an estimated treatment effect, but a single number does not show how much that estimate depends on the donor pool, the fitted augmentation model, or the observed outcome trajectories. In a setting with one treated unit and a limited number of potential donors, familiar standard errors from a large-sample regression may not describe the uncertainty that matters. Uncertainty assessment therefore combines placebo-based comparisons with explicit checks of how estimates change under defensible alternative specifications.

These checks do not repair a weak counterfactual design. The causal interpretation still depends on the assumptions that make the donor-based estimate a credible representation of the treated unit’s untreated outcome, including an appropriate donor pool, no relevant spillovers, and adequate pre-intervention fit. Placebo comparisons and sensitivity analyses reveal how conclusions depend on the design and fitting procedure; they do not prove those assumptions.

Target, Estimate, and Fitting Procedure

Definition: For treated unit 1 and post-intervention periods $t=1,\ldots,T_1$, the average causal effect is $$ \tau_{1,\mathrm{post}}=\frac{1}{T_1}\sum_{t=1}^{T_1}\left[Y_{1t}(1)-Y_{1t}(0)\right], $$ where $Y_{1t}(1)$ is the treated unit’s outcome under intervention and $Y_{1t}(0)$ is its true outcome in that period had it remained untreated. The target involves the true untreated potential outcome, which is not observed after treatment. An ASC estimator substitutes an estimated untreated outcome for that unobserved quantity.

A common ASC form starts with synthetic-control weights $\hat{\gamma}_j$ on donor units $j=2,\ldots,J+1$, then adds a fitted bias correction. Let $X_i$ denote pre-intervention predictors for unit $i$, and let $\hat{\beta}$ be an estimated outcome-model coefficient vector. A representative estimated untreated outcome is

\widehat{Y}_{1t}(0)=\sum_{j=2}^{J+1}\hat{\gamma}_jY_{jt} +\left(X_1-\sum_{j=2}^{J+1}\hat{\gamma}_jX_j\right)^\top\hat{\beta}.

The first term is the weighted donor outcome; the second adjusts for the difference between the treated unit’s predictors and the weighted donors’ predictors. Details of how weights and coefficients are fitted vary by implementation, so an analysis must specify its fitting procedure, predictors, tuning choices, and data window. The estimated average effect is

\hat{\tau}_{1,\mathrm{post}}= \frac{1}{T_1}\sum_{t=1}^{T_1} \left[Y_{1t}-\widehat{Y}_{1t}(0)\right].

This is an estimate of the causal target, not its definition: it uses observed treated outcomes and a data-dependent estimate of the missing untreated outcomes. In particular, the fact that ASC corrects for predictor imbalance does not by itself guarantee that the correction is reliable. Extrapolation, unstable tuning, or poor pre-intervention fit can make the estimate sensitive to modeling choices.

Worked Example: Computing an ASC Estimate

Hypothetical data. A treated region has pre-intervention predictor $X_1=10$. Two donors have predictors $X_A=8$ and $X_B=12$, with weights $\hat{\gamma}_A=0.6$ and $\hat{\gamma}_B=0.4$. The fitted augmentation coefficient is $\hat{\beta}=0.5$ outcome units per predictor unit. In two post-intervention periods, the donor outcomes are $Y_{A1}=20,\;Y_{B1}=16$ and $Y_{A2}=24,\;Y_{B2}=18$. The treated outcomes are 23 and 26.

The weighted predictor value is $0.6(8)+0.4(12)=9.6$, so the predictor discrepancy is $10-9.6=0.4$. The bias correction is $0.4(0.5)=0.2$ outcome units in each period.

For period 1, the weighted donor outcome is $0.6(20)+0.4(16)=18.4$, giving $\widehat{Y}_{11}(0)=18.4+0.2=18.6$. The estimated effect is $23-18.6=4.4$ outcome units.

For period 2, the weighted donor outcome is $0.6(24)+0.4(18)=21.6$, giving $\widehat{Y}_{12}(0)=21.6+0.2=21.8$. The estimated effect is $26-21.8=4.2$ outcome units. Thus $\hat{\tau}_{1,\mathrm{post}}=(4.4+4.2)/2=4.3$ outcome units. This is the estimated average effect; the target itself averages the treated outcomes minus the true, unobserved untreated potential outcomes.

Placebo Comparisons for the Fitted ASC Procedure

A practical uncertainty assessment compares the treated unit’s post-intervention discrepancy with discrepancies obtained by assigning placebo treatment to eligible donor units. For each placebo, the ASC procedure should be refitted as if that unit were treated: it must not remain in that placebo’s donor pool. The same pre-period, predictor construction, tuning rule, and outcome statistic should be applied consistently. Placebo units with very poor pre-intervention fit may be uninformative comparisons; any fit-quality rule for excluding them should be specified and applied transparently.

One possible statistic is the absolute post-intervention gap divided by the pre-intervention root mean squared prediction error (RMSPE). For unit $i$, write this as

R_i=\frac{\left|\overline{\widehat{\tau}}_{i,\mathrm{post}}\right|} {\mathrm{RMSPE}_{i,\mathrm{pre}}},

where the numerator is the absolute average post-period gap for the treated or placebo unit. Scaling by pre-period RMSPE helps compare gaps for units with different scales of pre-period fit, but it does not make unlike units exchangeable. When the denominator is very small, the ratio can also become large and unstable. The unscaled gaps, pre-period fits, and ratio should therefore be considered together rather than treating one statistic as decisive.

Interpretation: A placebo comparison asks whether the treated unit’s statistic is unusual relative to the selected placebo units under the same fitting procedure. It is not automatically a confidence interval for the causal effect, and a placebo rank is not automatically a design-based test. Its inferential force depends on whether the treated unit and placebo units can reasonably be treated as comparable under a specified assignment or exchangeability argument.

Worked Example: A Placebo Rank and Its Limits

Hypothetical comparison. Suppose the treated unit’s post/pre-RMSPE statistic is $R_1=2.15$. Eight eligible placebo units, each refitted using the same procedure, have statistics $1.1,\;0.8,\;2.0,\;1.4,\;0.7,\;1.8,\;0.9,\;1.3$. None is at least as large as 2.15.

A commonly reported finite-placebo rank calculation, with the treated unit included in the reference set, is

p_{\mathrm{rank}}=\frac{1+\#\{i:R_i\ge R_1,\ i\text{ is a placebo}\}}{1+M} =\frac{1+0}{1+8}=\frac{1}{9}\approx 0.111.

The treated unit is more extreme than all eight placebos, but the smallest attainable value with eight placebos is $1/9$. Under an exchangeability-based rank interpretation, this is not below a conventional $0.05$ threshold. More importantly, if treatment was not assigned in a way that justifies exchangeability, 0.111 is a descriptive rank comparison, not a valid randomization p-value. The comparison also does not say there is an 11.1% probability that the null hypothesis is true.

What Placebo-Based Uncertainty Can Establish

The logic of a rank comparison is straightforward when its assumptions hold. If the treated unit and $M$ placebo units are exchangeable under a sharp null hypothesis, then their $M+1$ statistics have no systematically favored rank. With no ties, each rank is equally likely, so the probability of a rank among the $k$ most extreme is $k/(M+1)$. Counting ties as at least as extreme yields a conservative calculation. This argument justifies a rank-based test only under the stipulated exchangeability and null; it does not make observational treatment assignment random.

In synthetic control applications, placebo units may differ from the treated unit in outcome volatility, predictor support, pre-period fit, or vulnerability to the intervention. The donor pool is selected rather than automatically representative, and the treated unit may be unusual by construction. A placebo distribution can therefore diagnose whether an estimated gap is large relative to the fitted gaps in this particular comparison set, while leaving open whether that set supplies a valid sampling or assignment distribution.

Likewise, a quantile range of placebo estimates should not be labeled a confidence interval without an inferential procedure that establishes its coverage for the causal target. A set of alternative ASC estimates is not a confidence interval either. It measures dependence on specifications, not repeated-sampling uncertainty with a stated coverage probability.

Sensitivity Analysis as a Complement

Sensitivity analysis asks whether the substantive conclusion changes when reasonable analysis choices change. For ASC, relevant choices include excluding a donor that has substantial influence, altering the pre-intervention window, changing the predictor set, or varying the augmentation tuning parameter within defensible bounds. Each alternative should be refitted from the beginning; holding fitted weights or augmentation coefficients fixed would not show how the complete procedure responds to the change.

Worked Example: Comparing Defensible Specifications

Hypothetical results. The baseline analysis in the first example estimated an average effect of 4.3 outcome units. Suppose a pre-specified sensitivity analysis gives these estimates, each from a refitted ASC model:

Specification	Estimated average effect
Baseline donor pool, predictors, and window	4.3 outcome units
Exclude donor A	2.1 outcome units
Exclude donor B	5.6 outcome units
Use an alternative defensible pre-period window	3.7 outcome units
Use an alternative pre-specified predictor set	4.0 outcome units

The estimate ranges from 2.1 to 5.6 outcome units across these specifications. The arithmetic range is $5.6-2.1=3.5$ outcome units. The largest changes occur when either donor is excluded, so the conclusion is particularly dependent on donor composition. The estimates remain positive, but that stability of sign does not establish the identification assumptions or quantify sampling uncertainty. The range is a sensitivity summary, not a 95% confidence interval: no repeated-sampling coverage claim follows from its endpoints.

Interpret the sensitivity results alongside pre-intervention fit and donor contributions. A dramatic change after removing one donor can indicate that the estimated counterfactual relies heavily on that unit. If removing it also substantially worsens pre-period fit, the alternative is informative about the design trade-off but may not be an equally credible model. Conversely, a stable estimate across several choices is reassuring about specification dependence only across the choices examined; it cannot rule out an unmeasured source of bias or a poor donor pool shared by all specifications.

Define the target and primary procedure.
State the post-period causal target, outcome scale, donor pool, predictors, pre-period, and ASC fitting and tuning choices.

Inspect the counterfactual fit.
Report pre-intervention fit and whether the estimate relies on influential donors or extrapolation. A placebo result cannot compensate for a poorly supported counterfactual.

Refit placebo analyses consistently.
Apply the stated procedure to eligible placebo units, disclose eligibility and fit rules, and compare a clearly defined post-period statistic with its placebo distribution.

Vary defensible design choices.
Refit after donor exclusions or changes to windows, predictors, and tuning. Report the estimates and fit diagnostics rather than only a favorable specification.

Match the claim to the evidence.
Describe placebo ranks as inferential only when their assignment or exchangeability basis is justified. Label sensitivity ranges as sensitivity results, not confidence intervals.

Common Mistake

A common mistake is to say, “The treated unit exceeds every placebo, so the effect is statistically significant and the placebo range is a confidence interval.” The first part overlooks whether the treated and placebo units are exchangeable and whether the few available placebos provide adequate resolution. The second confuses variation across fitted placebo units or specifications with a sampling distribution for the target. A better conclusion states what was actually computed—for example, that the treated statistic exceeded all eight eligible placebo statistics—then explains the limited rank resolution, the fit and donor-selection constraints, and the sensitivity of the estimate to alternative specifications.

Key takeaway: ASC uncertainty assessment is strongest when placebo comparisons are transparent and consistently refitted, sensitivity analyses target consequential modeling choices, and interpretations separate evidence about relative extremeness from causal identification and interval coverage.

Check Your Understanding

Use the definitions and distinctions in this tutorial to answer the following.

Why does the causal target use the true untreated potential outcome, while the ASC estimate uses an estimated untreated outcome?
If the treated statistic exceeds all seven eligible placebo statistics, what is the smallest value of the rank calculation with the treated unit included? What does that value not establish by itself?
Why should the ASC fitting procedure be refitted for each placebo unit rather than reusing the treated unit’s fitted weights and augmentation model?
An estimate is positive under every specification examined, but changes substantially when one donor is removed. What does this say about specification sensitivity, and what does it not prove?
What additional justification is needed before interpreting a placebo rank as a randomization-based p-value?