Tuning Predictor Importance Without Overfitting

Why Predictor Importance Must Be Tuned Carefully

Synthetic control has two distinct kinds of weights. Donor weights combine control units to form a synthetic version of the treated unit. Predictor-importance weights determine which pre-intervention characteristics matter most when choosing that donor combination. The donor weights are constrained to be nonnegative and sum to one; the predictor-importance weights govern how well the treated unit’s predictors should match.

The donor weights depend on the predictor-importance weights. Giving more emphasis to one predictor can change which donor combination appears best, and hence change the synthetic outcome trajectory. The choice of predictor importance is therefore a design decision, not a harmless adjustment after the synthetic control has been built. A defensible approach uses part of the pre-intervention period to construct the synthetic control and a separate, held-out pre-intervention period to compare candidate predictor weights.

Definition: Let \(Y_{1t}\) denote the outcome for the treated unit in period \(t\), and let \(Y_{jt}\) denote the outcome for donor \(j\), for \(j=1,\ldots,J\). A synthetic outcome based on donor weights \(w=(w_1,\ldots,w_J)\) is \(\hat{Y}_{1t}(0;w)=\sum_{j=1}^{J}w_jY_{jt}\), where \(w_j\geq 0\) and \(\sum_{j=1}^{J}w_j=1\). Predictor-importance weights specify how discrepancies between the treated unit’s predictors and the synthetic unit’s predictors are weighted when selecting \(w\).

Separate the Training and Validation Periods

Suppose periods \(1,\ldots,T_0\) precede the intervention. Divide them into a training set \(\mathcal{T}\) and a validation set \(\mathcal{V}\), with neither set containing post-intervention periods. Construct the predictor vector \(X_1\) for the treated unit and predictor matrix \(X_0\) for the donors using training information. In particular, do not put the held-out validation outcomes into the predictors used to fit the donor weights.

Represent a candidate importance scheme by a diagonal matrix \(V\) with nonnegative entries. For each candidate \(V\), choose donor weights using the predictor discrepancy:

\hat{w}(V)=\arg\min_{w\in\mathcal{W}}(X_1-X_0w)'V(X_1-X_0w), \qquad \mathcal{W}=\{w:w_j\geq 0,\ \sum_{j=1}^{J}w_j=1\}.

Then calculate the validation errors \(e_t(V)=Y_{1t}-\hat{Y}_{1t}(0;\hat{w}(V))\) for \(t\in\mathcal{V}\). Before the intervention, the treated outcome is observed without the intervention, so these errors measure how well each candidate’s synthetic control predicts held-out pre-intervention outcomes. Use a pre-specified validation criterion to compare candidates, and retain the candidate with the lowest validation loss.

Only the relative importance of predictors is relevant to the minimization over \(w\): multiplying every entry of \(V\) by the same positive constant leaves the minimizer unchanged. It is therefore convenient to normalize candidate matrices, for example so their diagonal entries sum to one. This prevents arbitrary scale differences from obscuring comparisons. The normalization does not determine which predictors deserve more importance; the validation comparison does.

Key distinction: The training step chooses donor weights for a given predictor-importance scheme. The validation step compares schemes by their predictive performance in held-out, untreated periods. Validation outcomes may guide the choice of \(V\), but they should not also be used as training predictors if they are meant to provide an honest held-out assessment.

Choosing a Validation Criterion

Several loss criteria are standard, and they need not rank candidates in the same order. For \(m=|\mathcal{V}|\) validation periods, mean squared error (MSE), root mean squared prediction error (RMSPE), and mean absolute error (MAE) are:

\operatorname{MSE}(V)=\frac{1}{m}\sum_{t\in\mathcal{V}}e_t(V)^2,\qquad \operatorname{RMSPE}(V)=\sqrt{\operatorname{MSE}(V)},\qquad \operatorname{MAE}(V)=\frac{1}{m}\sum_{t\in\mathcal{V}}|e_t(V)|.

MSE and RMSPE rank candidates identically because the square-root function is increasing: if one candidate has lower MSE, it also has lower RMSPE. MSE is in squared outcome units; RMSPE and MAE are in the same units as the outcome. Squared loss gives relatively large errors substantial influence, while absolute loss is less sensitive to a single unusually large error. Neither criterion is automatically correct in every application. The choice should reflect the intended notion of predictive fit and be made before examining which candidate wins.

Worked Example: One Large Error Versus Several Moderate Errors

Consider three hypothetical validation periods. Candidate D has errors \((0,0,4)\), and candidate E has errors \((2,2,2)\). Suppose the outcome is measured in points.

For D, the squared errors are \(0,0,16\). Thus:

\operatorname{MSE}(D)=\frac{0+0+16}{3}=\frac{16}{3}=5.333,\quad \operatorname{RMSPE}(D)=\sqrt{\frac{16}{3}}=2.309,\quad \operatorname{MAE}(D)=\frac{0+0+4}{3}=1.333.

For E, the squared errors are \(4,4,4\), and the absolute errors are \(2,2,2\):

\operatorname{MSE}(E)=\frac{4+4+4}{3}=4,\quad \operatorname{RMSPE}(E)=\sqrt{4}=2,\quad \operatorname{MAE}(E)=\frac{2+2+2}{3}=2.

MSE and RMSPE prefer E, since \(4<5.333\) and \(2<2.309\). MAE prefers D, since \(1.333<2\). The reason is visible in the errors: D predicts two periods exactly but misses the third by 4 points, whereas E misses every period by 2 points. A criterion cannot be chosen just because its name is familiar; it encodes which pattern of prediction errors is considered worse.

A Nested Selection Example

The following example shows how predictor importance can change donor weights and, in turn, validation predictions. It is deliberately small so that both stages of selection are visible.

Worked Example: Comparing Two Predictor-Importance Choices

Suppose there are two predictors and two donors, A and B. The treated unit has training predictor values \(X_1=(10,4)'\), while A has \((8,2)'\) and B has \((12,8)'\). Write \(w\) for A’s weight, so B’s weight is \(1-w\), with \(0\leq w\leq1\). The synthetic predictor values are:

X_0w=(8w+12(1-w),\ 2w+8(1-w))'=(12-4w,\ 8-6w)'.

Compare two normalized importance choices: \(V_A=\operatorname{diag}(1,0)\), emphasizing predictor 1, and \(V_B=\operatorname{diag}(0,1)\), emphasizing predictor 2. Under \(V_A\), the first-predictor discrepancy is \(10-(12-4w)=4w-2\). Its squared value is minimized at \(4w-2=0\), so \(\hat{w}(V_A)=0.5\). Under \(V_B\), the second-predictor discrepancy is \(4-(8-6w)=6w-4\), minimized at \(6w-4=0\), so \(\hat{w}(V_B)=2/3\). Both solutions satisfy the donor-weight constraints.

Suppose the resulting synthetic predictions and treated outcomes in three held-out, pre-intervention periods are as follows:

Validation period	Treated outcome	Prediction under \(V_A\)	Prediction under \(V_B\)
1	21	20	20.5
2	21	22	20.5
3	26	24	25.5

The errors \(Y_{1t}-\hat{Y}_{1t}(0)\) are \((1,-1,2)\) under \(V_A\) and \((0.5,0.5,0.5)\) under \(V_B\). Therefore:

\operatorname{MSE}(V_A)=\frac{1^2+(-1)^2+2^2}{3}=2,\qquad \operatorname{MSE}(V_B)=\frac{0.5^2+0.5^2+0.5^2}{3}=0.25.

The corresponding RMSPE values are \(\sqrt{2}=1.414\) and \(\sqrt{0.25}=0.5\) outcome units. MAE values are \((1+1+2)/3=1.333\) and \((0.5+0.5+0.5)/3=0.5\) outcome units. All three criteria select \(V_B\) in this example. This means that, among these two candidates and under these validation data and criteria, emphasizing predictor 2 gives better held-out outcome prediction. It does not establish that predictor 2 is intrinsically more important in every period or that the treatment effect has been identified without the design assumptions.

Why Post-Intervention Tuning Invalidates the Design

After intervention, the treated unit’s observed outcome may include the effect of treatment. In potential-outcome notation, \(Y_{1t}=Y_{1t}(1)\) after treatment, while the counterfactual needed for the treatment effect is \(Y_{1t}(0)\), the outcome the treated unit would have had without treatment. The synthetic control is intended to approximate \(Y_{1t}(0)\). Choosing predictor weights to make that synthetic outcome match the observed post-intervention \(Y_{1t}(1)\) uses the outcome being evaluated to construct its own counterfactual.

This creates a direct risk of fitting away the effect. A candidate that moves the synthetic prediction toward the treated post-intervention outcome can look better by a post-intervention prediction criterion even when it makes the counterfactual less credible. The resulting estimated gap can be pulled toward zero. In general, the bias need not always be toward zero, but the post-outcome-driven selection makes the estimated effect depend on the very outcome contrast the design is meant to estimate.

Worked Example: The Cost of Selecting Weights on the Treated Post-Outcome

Suppose, hypothetically, that in a post-intervention period the treated unit’s untreated potential outcome is 10 points and its observed treated outcome is 15 points. The true treatment effect for this period is therefore \(15-10=5\) points. Candidate A gives a synthetic prediction of 10, while candidate B gives a prediction of 14.

Using pre-intervention validation would select between A and B based on their earlier prediction errors. If instead one chose the candidate with the smallest post-period discrepancy from the observed treated outcome, the errors would be \(|15-10|=5\) for A and \(|15-14|=1\) for B, so B would be selected. The estimated treatment-effect gap under B would then be \(15-14=1\) point, not the true 5 points. The example does not imply that a post-period prediction will always move toward the observed outcome, but it shows why using that outcome as a tuning target can contaminate the counterfactual and distort the effect estimate.

A Defensible Tuning Workflow

A practical tuning procedure makes the separation between design and evaluation explicit. It should also limit the opportunity to keep adjusting choices until a validation result looks favorable.

Fix the candidate set and criterion.
Specify the candidate predictor-importance matrices, any normalization, the pre-intervention validation periods, and the loss criterion before comparing results.

Fit donor weights within each candidate.
For each \(V\), solve the constrained predictor-matching problem using training information only. Check that each solution satisfies nonnegativity and the sum-to-one constraint.

Evaluate held-out pre-intervention outcomes.
Compute the synthetic predictions and errors for the validation periods, then calculate the chosen loss using the same periods for all candidates.

Select and report transparently.
Choose the candidate with the lowest pre-specified validation loss and report the criterion, split, candidate set, and validation performance. Do not select weights by matching post-intervention outcomes.

A single validation block may be unusually easy or difficult to predict. If enough pre-intervention data are available, one can compare candidates across multiple held-out blocks or use rolling-origin validation, where each prediction is made using only information available before its validation period. Reusing the same validation outcomes for many rounds of informal changes can itself overfit: the analyst may end up selecting a candidate that performs well on that particular block by chance. More elaborate validation does not replace the substantive identification assumptions of synthetic control, but it can make tuning less dependent on one arbitrary split.

Common Mistakes and Interpretation

Confusing predictor weights with donor weights. Predictor importance changes the matching objective; donor weights combine control units. They play different roles and have different constraints.
Using the training fit as the only basis for tuning. The same data that select donor weights can make a candidate appear to fit well by construction. Held-out pre-intervention outcomes provide a more informative comparison of predictive performance.
Choosing a criterion after seeing which one favors a preferred candidate. MSE/RMSPE and MAE respond differently to large errors. State the criterion in advance and interpret its units correctly.
Tuning to the post-intervention treated outcome. That outcome can include treatment effects. Using it to choose the counterfactual model compromises the separation between outcome prediction and causal effect estimation.
Treating the lowest validation loss as proof of causal validity. Better pre-intervention prediction is useful evidence about fit, but it does not by itself establish that the synthetic control reproduces the untreated post-intervention trajectory.

Key takeaway: Select predictor-importance weights by a transparent comparison of held-out pre-intervention prediction, not by fitting the treated unit’s post-intervention outcomes. The validation criterion, split, and candidate set are part of the design and should be reported.

Check Your Understanding

Use the definitions and examples above to answer the following questions.

What is the difference between predictor-importance weights and donor weights in synthetic control?
Why should held-out validation outcomes not also be included as training predictors when they are being used to compare candidate importance weights?
A candidate has validation errors \((1,1,5)\), and another has errors \((2,2,2)\). Which criterion could favor the first candidate, and why might a squared-error criterion favor the second?
Why do MSE and RMSPE always produce the same ranking when evaluated on the same validation periods?
Explain how tuning on post-intervention outcomes can change the estimated treatment-effect gap.