Leave-One-Donor-Out Sensitivity Analysis for Synthetic Controls

Why Remove Donors One at a Time?

A synthetic control estimates the untreated path of a treated unit by combining units that were not treated. Its post-treatment estimate can therefore depend on which donor units are available and how the method weights them. A donor may receive a large weight because it helps reproduce the treated unit’s pre-treatment outcomes, or because the donor pool is small. Either way, the estimated effect may change substantially if that donor is removed.

Leave-one-donor-out sensitivity analysis examines this dependence directly. For each donor in turn, remove it from the donor pool, refit the synthetic control using the stated fitting procedure, and compare the resulting post-treatment gaps with the baseline gaps. The exercise asks whether the substantive conclusion is robust to each individual donor’s availability. It does not, by itself, establish that the baseline estimate is causal or provide a confidence interval.

What Must Be Recomputed?

For each specification, the synthetic outcome in period \(t\) is the weighted donor outcome, and the estimated gap is the treated outcome minus that synthetic outcome:

Here \(Y_{1t}\) is the observed outcome for the treated unit, \(Y_{jt}\) is the observed outcome for donor \(j\), and \(\widehat{Y}_{1t}(0)\) estimates the treated unit’s outcome in the absence of treatment. A positive gap means the observed outcome exceeds the estimated untreated outcome. This sign interpretation depends on how the outcome is defined; for example, a positive gap in a cost outcome is an increase in cost, not necessarily a benefit.

After removing donor \(k\), the weights are generally different. Write them as \(w_j^{(-k)}\), with \(j\) restricted to the remaining donor pool. The new gap is

The superscript \((-k)\) labels the specification that omits donor \(k\). It is not an arithmetic operation on the baseline estimate. In particular, setting \(w_k\) to zero and leaving all other weights unchanged usually violates the requirement that weights sum to one. Even if the remaining weights are renormalized, that calculation is not generally equivalent to refitting: the optimizer may choose a different combination to improve pre-treatment fit.

A consistent sensitivity analysis specifies what is held fixed. For example, keep the pre-treatment period, predictor definitions, outcome scale, and fitting rule the same, while rerunning the weight optimization on each reduced donor pool. If tuning choices are data-dependent, state whether they are held at the baseline choices or retuned according to a prespecified rule. Changing the procedure as well as the donor pool makes it harder to attribute differences to donor deletion.

Measure Weight Concentration Alongside the Gaps

A leave-one-out estimate can change because the omitted unit had a large role, but it can also change because the remaining pool cannot reproduce the treated unit’s pre-treatment trajectory as well. Report fit diagnostics alongside post-treatment results. One concise measure of concentration is the Herfindahl index of the donor weights:

For nonnegative weights summing to one, \(H\) is larger when weight is concentrated on fewer donors. \(N_{\mathrm{eff}}\), sometimes called the effective number of donors, is a summary of that concentration: equal weights on \(m\) donors give \(H=1/m\) and \(N_{\mathrm{eff}}=m\). It is not a literal count of independent units, and it does not quantify uncertainty in the treatment effect.

For an average post-treatment summary over periods \(T_1,\ldots,T_2\), calculate the mean gap separately for each specification:

Do not rely only on this average. A similar mean can conceal differences in sign, timing, or persistence. A useful comparison presents the time-specific gaps, the post-treatment average, the weight concentration, and a pre-treatment fit measure for the baseline and each deletion.

A Short Derivation of the Concentration Summary

The effective donor count has a simple interpretation when weights are equal. If \(m\) donors each receive weight \(1/m\), then

More generally, the weights sum to one, so the Cauchy–Schwarz inequality gives \(1=(\sum_j w_j)^2 \leq m\sum_j w_j^2=mH\), where \(m\) is the number of available donors. Therefore \(H\geq 1/m\), with equality for equal weights, and \(N_{\mathrm{eff}}\leq m\). The index summarizes concentration relative to equal weighting; it does not indicate whether a donor is substantively comparable or whether the synthetic control is credible.

Interpret the Pattern, Not Just the Range

A useful interpretation separates three questions. First, do the post-treatment gaps retain their sign and approximate magnitude when each donor is removed? Second, does one omission cause a large change, suggesting the result depends on that donor or on the replacements available? Third, do pre-treatment fit and weight concentration change at the same time? A large gap change accompanied by much worse pre-treatment fit is different from a large change among specifications that all fit the pre-treatment data comparably.

For a dynamic analysis, inspect the full sequence \(\widehat{\tau}^{(-k)}_t\), not only its average. For instance, an average near zero could combine positive gaps early and negative gaps later. Conversely, similar averages could hide substantially different timing. A graph with one line for the baseline and one for each donor deletion often makes these patterns easier to see.

The causal interpretation still rests on the design’s assumptions. In particular, the donor units must not receive the treatment or be affected by spillovers, and the chosen donors and weighting procedure must provide a credible approximation to the treated unit’s untreated trajectory after treatment. Good pre-treatment fit is relevant evidence, but it cannot verify the unobserved post-treatment counterfactual. Leave-one-out analysis assesses dependence on donor composition; it does not repair a contaminated donor pool, rule out differential shocks, or establish the identifying assumptions.

Common Pitfall and Reporting Practice

• Wrong reasoning: “Donor A had a large baseline weight, so I removed its weight and kept all other weights unchanged. That is the leave-one-out estimate.” Correction: Refit the weights on the reduced donor pool under the stated fitting rule. The remaining weights must meet the specification’s constraints and should be selected anew.
• Wrong reasoning: “The estimates have the same sign after every deletion, so the causal effect is established.” Correction: Sign stability is evidence of robustness to these particular omissions, not proof of the counterfactual assumptions or absence of other biases.
• Wrong reasoning: “The range of leave-one-out estimates is a confidence interval.” Correction: It is a sensitivity range across alternative donor pools. It has no stated coverage probability and does not measure sampling uncertainty.
• Wrong reasoning: “The donor with the biggest weight must be the only influential donor.” Correction: Influence depends on how removing a donor changes the refitted weights, fit, and post-treatment outcomes. A modestly weighted donor can still matter if it is difficult to replace.

A transparent report states the donor pool and omission rule, explains which fitting and tuning choices were held fixed, and gives the baseline and leave-one-out results. Include pre-treatment fit diagnostics, post-treatment gaps by period, an average gap if relevant, and a weight concentration summary. If one donor’s removal materially changes the conclusion, say so directly and describe the result as donor-sensitive rather than selecting the preferred specification after seeing the answers.