Why Predictor Choice Is Part of the Design
A synthetic control combines untreated comparison units to approximate the treated unit’s outcome in the absence of intervention. The donor weights determine that combination, but the fitted control also depends on which features are matched and how much each feature matters. A fit can look excellent on one chosen measure and poor on another. Predictor selection and tuning therefore shape the counterfactual, rather than merely polishing it.
The central safeguard is temporal: choose predictors, importance weights, and tuning rules using information from before the intervention. Outcomes after treatment cannot be used to select the counterfactual, since they contain the effect the analysis is intended to estimate. Pre-intervention tuning does not by itself establish that the synthetic control is a credible counterfactual; it is a disciplined way to build one using only information available before treatment.
Donor Weights and Predictor Importance Weights
Suppose unit 1 is treated, and units \(j=2,\ldots,J+1\) are untreated donors. The intervention begins after pre-intervention period \(T_0\). Let \(Y_{1t}\) denote the treated unit’s outcome and \(Y_{jt}\) the outcome for donor \(j\). A predictor vector \(X_1\) summarizes selected pre-intervention features for the treated unit; the columns of \(X_0\) are the corresponding vectors for the donors.
For a fixed predictor set and importance matrix, the standard synthetic-control fitting step chooses donor weights to minimize weighted predictor imbalance:
A common choice is a diagonal, positive semidefinite \(V\), with nonnegative diagonal entries. A larger entry gives the associated predictor more influence in the fit. Multiplying all entries of \(V\) by the same positive constant does not change the minimizing \(W\); relative importance is what matters. The donor-weight restrictions make the synthetic control a convex combination, so it cannot extrapolate beyond the donor units by assigning negative weights.
Predictors might include pre-intervention averages of important covariates, outcome levels, or summaries of the pre-intervention outcome path. They should be defined so that the treated and donor quantities refer to the same period and measurement scale. A predictor set should be defensible in context: consider which pre-intervention features are substantively relevant and prognostic for the outcome, rather than searching indiscriminately for a combination that happens to fit one period.
Worked Example: Choosing a Predictor Set
A hypothetical region receives a policy after year 6. The analyst has annual outcomes for the treated region and donors, along with pre-policy unemployment and population. The aim is to fit the synthetic control using years 1–6.
A plausible candidate predictor set includes the mean outcome in years 1–3, the mean outcome in years 4–6, unemployment averaged over years 1–6, and population measured before the policy. These quantities are all available before treatment and represent both the outcome trajectory and relevant regional characteristics. The analyst could also compare a version that includes annual outcome lags, provided the number of pre-intervention periods and validation plan can support the extra flexibility.
The analyst would not include a post-policy unemployment value or an outcome measured in year 7. Those values are not pre-intervention information, and the year 7 outcome may reflect the policy. Nor should a variable be included merely because it happens to make the final treated-versus-synthetic gap small after treatment. The choice here is a design decision grounded in pre-policy information, not a search for the most favorable estimated effect.
Tuning Importance Weights With Pre-Intervention Data
A predictor set and a particular \(V\) determine the donor weights through the imbalance minimization above. The importance matrix can be fixed using substantive knowledge, chosen from a prespecified collection of candidates, or tuned to predict held-out pre-intervention outcomes. Tuning is useful because predictors may be measured on different scales and because emphasizing every feature equally may not yield a good approximation to the treated unit’s outcome path.
A transparent tuning procedure divides the pre-intervention periods into training and validation periods. Fit donor weights using only training-period information. Then compare the resulting synthetic control’s outcomes with the treated unit’s actual outcomes in the held-out pre-intervention periods. For candidate specification \(m\), one possible validation loss is
where \(\mathcal{H}\) is the set of held-out pre-intervention periods, and \(\widehat w_{jm}\) is obtained without using the treated unit’s outcome in those periods as a fitting target. The selected specification has the smallest validation loss among the candidates considered. All outcomes used for this comparison are pre-intervention outcomes; none is a post-intervention outcome.
With short time series, a single holdout can make the choice depend heavily on one unusual year. A rolling-origin or blocked procedure can provide several pre-intervention validation comparisons: fit on an earlier block and validate on a later block, then repeat where the available periods allow. The ordering matters. Fitting on later pre-intervention periods to validate an earlier period is still pre-treatment in the causal sense, but it does not mimic forecasting forward and can let later information influence the fit. Forward-in-time validation is usually easier to interpret as a test of how well the pre-intervention relationship predicts an unseen subsequent period.
Worked Example: Comparing Two Importance-Weight Choices
In a hypothetical analysis, two candidate importance matrices produce donor weights using training-period information. Candidate A gives weights \(0.6\) and \(0.4\) to donors 2 and 3; candidate B gives weights \(0.2\) and \(0.8\). The two candidates are evaluated on two held-out pre-intervention years. Their observed outcomes are:
| Held-out year | Treated outcome | Donor 2 outcome | Donor 3 outcome |
|---|---|---|---|
| Year 5 | 105 | 100 | 110 |
| Year 6 | 108 | 106 | 111 |
For candidate A, the synthetic outcomes are \(0.6(100)+0.4(110)=104\) in year 5 and \(0.6(106)+0.4(111)=108\) in year 6. Its validation mean squared prediction error is
For candidate B, the synthetic outcomes are \(0.2(100)+0.8(110)=108\) in year 5 and \(0.2(106)+0.8(111)=110\) in year 6. Thus
Candidate A has the lower held-out mean squared error, \(0.5\) outcome-units squared versus \(6.5\). This is evidence that A predicted these two pre-intervention years better under this validation design. It is not proof that A identifies the true untreated counterfactual after the intervention.
Comparing Fit-Tuning Strategies
There is no universally best tuning strategy. The choice depends on how many pre-intervention periods are available, how much flexibility the candidate specifications contain, and whether a credible temporal validation split is possible. The key distinction is whether a strategy evaluates prediction on information not used to fit the donor weights.
| Strategy | How it works | Main caution |
|---|---|---|
| Substantive, prespecified weights | Set predictor priorities before fitting, using subject-matter knowledge and measurement considerations. | Priorities may not produce the best pre-intervention outcome prediction; report the rationale. |
| In-sample fit tuning | Choose \(V\) to minimize fit error on the same pre-intervention outcomes used to construct the weights. | Flexible tuning can overfit; the reported fit may understate prediction error in new periods. |
| Held-out or rolling validation | Fit using training pre-periods and compare predictions with one or more later pre-period blocks. | Very short or unusual validation blocks can make the selected specification unstable. |
In-sample tuning is not automatically invalid, but its objective is not an independent test of predictive performance. If many predictor sets and importance matrices are tried, the best in-sample match may partly reflect the search itself. Validation more directly asks whether a specification that fits one part of the pre-intervention record predicts another part. It also uses data less efficiently: holding periods out leaves less information for fitting. In a short panel, the analyst must weigh that cost against the risk of overfitting.
Worked Example: Why a Holdout Can Change the Choice
Suppose two candidate specifications are fitted on two training years, then evaluated on two later pre-intervention years. Candidate A has training residuals \(0\) and \(1\); candidate B has training residuals \(1\) and \(1\). Their training mean squared errors are
Choosing only by training fit would select A. Now suppose their held-out residuals are \(4\) and \(4\) for A, and \(1\) and \(2\) for B. The validation losses are
On this hypothetical split, B predicts the held-out pre-intervention outcomes better, despite its larger training error. If B is selected, the analyst then refits its chosen predictor set and importance matrix using all available pre-intervention periods to obtain the final donor weights. The holdout has served to choose the specification, not to estimate the final weights using a permanently reduced sample.
A Careful Workflow and Its Limits
A useful workflow separates design choices, tuning, and final fitting. First, define candidate predictors and any transformations using pre-intervention information. Second, specify a modest and interpretable set of importance-weight choices, or a tuning procedure, before examining post-intervention effects. Third, evaluate candidates with a pre-intervention validation plan appropriate to the time series. Finally, refit the chosen specification using all pre-intervention information and compute the synthetic control.
Use features measured before intervention, with consistent definitions for the treated unit and donors. Explain their substantive or predictive relevance.
Choose candidate importance weights and pre-intervention training and validation periods. Keep validation outcomes out of the donor-weight fitting step.
Calculate the same validation loss for each candidate and select according to the stated rule, not the post-intervention gap.
Use all pre-intervention data to estimate the final donor weights. Good pre-intervention prediction is useful evidence about fit, but does not alone guarantee a valid causal counterfactual.
Pre-intervention outcome histories are often informative predictors because they directly describe the trajectory the synthetic control is intended to reproduce. However, adding many outcome lags can be problematic when there are few pre-intervention periods: the fit may become sensitive to individual years, and the available validation data may be too limited to distinguish candidates reliably. Predictor scales also matter. If a feature is rescaled, its numerical contribution to the weighted objective changes unless its importance weight is adjusted accordingly. Any standardization or transformation should be specified using pre-intervention data and applied consistently to treated and donor units.
The final causal interpretation still depends on the design: the donor combination must be a credible stand-in for the treated unit’s untreated path, and the intervention must not already have affected the supposedly pre-intervention information. Predictor tuning is not a substitute for those conditions, nor does a small validation error establish them. Its role is narrower and important: to make the construction of the synthetic control transparent and to avoid letting the outcomes of interest determine the counterfactual.
Common Mistake
Wrong reasoning: “I should try predictor sets until the estimated post-intervention effect looks plausible, then keep the one with the smallest post-intervention gap.” This uses the outcome after treatment to choose the counterfactual and can directly steer the estimated effect toward a preferred result.
Correction: Fix or tune the specification using pre-intervention predictors and outcomes only. If tuning is data-driven, state the pre-intervention validation rule and apply it consistently. Post-intervention outcomes may be used afterward to calculate the estimated gap under the selected synthetic control, but not to select the predictors or importance weights that define it.
Check Your Understanding
Answer each question using the distinction between fitting and validating a synthetic control.
- What is the difference between donor weights \(W\) and predictor importance weights \(V\)?
- Why can minimizing in-sample pre-intervention error give an optimistic impression of predictive fit when many specifications are searched?
- A candidate specification has a lower validation mean squared error than another. What does this establish, and what does it not establish about the causal counterfactual?
- Why should an analyst not select predictor weights by choosing the specification that gives the most appealing post-intervention effect?
- After selecting a specification with a pre-intervention holdout, why might the analyst refit the donor weights using all pre-intervention periods?