Why Cross-Validate a Propensity Model?
A propensity model estimates the probability of treatment given measured covariates. Its specification may involve choices such as whether to include nonlinear terms or interactions, and a penalized model may require choosing a penalty strength. Cross-validation provides a systematic way to compare such candidates using observations not used to fit each corresponding model.
The criterion used for that comparison matters. A common choice is held-out log loss, which evaluates the probabilities assigned to the observed treatment assignments. It is a measure of predictive performance, not a direct measure of confounding control or covariate balance. A candidate with the lowest cross-validated loss is therefore not automatically the best model for a causal analysis.
For a fitted probability \(\hat e_i\), the binary log loss is
A confident prediction of the observed assignment has low loss; a confident prediction of the opposite assignment has high loss. If \(k(i)\) is the fold containing observation \(i\), and \(\hat e_i^{(-k(i))}\) is the prediction from a model fitted without that fold, the cross-validated loss is
When folds have unequal sizes, this pooled formula weights observations equally. Averaging fold-specific mean losses equally instead gives each fold the same weight, which can differ if fold sizes differ.
Choosing a Validation Criterion
Log loss is a standard choice when the fitted probabilities themselves matter. It rewards probabilities that assign high probability to the observed treatment and penalizes confident errors strongly. The validation loss is computed on held-out observations, without adding the penalty used during model fitting. This compares predictive performance rather than the model's training objective.
Other criteria answer somewhat different questions. The Brier score is the mean squared error \(\frac{1}{n}\sum_i(A_i-\hat e_i)^2\); it also evaluates probability predictions. Classification accuracy depends on a cutoff, usually 0.5, and discards how far a prediction is from that cutoff. The area under the receiver operating characteristic curve (AUC) measures ranking discrimination, not probability calibration. Neither accuracy nor AUC directly measures covariate balance.
Choose the criterion before comparing candidates and use the same folds and criterion across them. The choice should fit the purpose: log loss is often reasonable for comparing estimated assignment probabilities, while the causal analysis still requires separate checks of balance and overlap. No predictive criterion can establish that all important confounders were measured or correctly modeled.
What Cross-Validation Estimates
For a candidate modeling procedure, cross-validation approximates its expected prediction loss on new observations drawn from the same population and under the same sampling conditions. It does so by repeatedly fitting on a subset and evaluating on observations set aside from that fit. The following justification explains why the held-out losses are informative, while also clarifying their limits.
For a fitted procedure, the target is its expected loss on a new unit, such as \(E\{\ell(A,\hat e(X))\}\), where expectation is over the relevant data-generating population.
The model for a given fold is fitted without the observations in that fold. Its predictions for those observations are therefore not the model's in-sample fitted values.
If the held-out units are representative of the target population, their average loss estimates the loss of a model trained on the remaining data. Pooling across folds uses each observation for validation once.
Using the same folds reduces differences caused merely by different partitions. The candidate with the lowest estimated loss performs best on this criterion, subject to sampling variability.
The estimate is not exact: the training sets within folds are smaller than the full data set, and the estimated losses vary with the sample and fold assignment. Cross-validation also does not test the causal assumptions needed for propensity-score analyses. Its interpretation depends on folds representing the intended prediction setting. For clustered observations, for example, units from the same cluster should generally not be split across training and validation folds when the aim is prediction for new clusters.
Worked Example: Selecting a Penalty by Mean Cross-Validated Log Loss
Suppose an analyst compares three ridge-penalty strengths for a logistic propensity model using the same four folds. The table gives the mean log loss in each held-out fold; smaller values are better.
| Penalty strength | Fold 1 | Fold 2 | Fold 3 | Fold 4 |
|---|---|---|---|---|
| \(\lambda=0\) | 0.51 | 0.72 | 0.55 | 0.70 |
| \(\lambda=0.1\) | 0.54 | 0.61 | 0.57 | 0.60 |
| \(\lambda=1\) | 0.66 | 0.58 | 0.63 | 0.57 |
For equal-sized folds, the mean for \(\lambda=0.1\) is
The corresponding means are \(2.48/4=0.620\) for \(\lambda=0\) and \(2.44/4=0.610\) for \(\lambda=1\). Thus \(\lambda=0.1\) has the smallest mean cross-validated log loss among these candidates. This selects it for predictive performance under the specified criterion; it does not yet show that the resulting matched or weighted groups have adequate covariate balance.
Fitting and Comparing Candidate Specifications
The candidates may differ in penalty strength, functional form, or a prespecified set of interactions. For each candidate, fit the model on each training portion and calculate its held-out loss. Keep the comparison fair: use the same folds, outcome coding, loss definition, and preprocessing rules for every candidate. A result should not be treated as evidence that an unconsidered model is inferior.
Any data-dependent step that affects the fitted model belongs inside the fold-specific training process. For example, if covariate selection, transformation choice, or tuning another parameter is based on the observed data, doing that step once on all observations before cross-validation allows information from validation folds to influence the fit. This is leakage and can make validation performance look better than it should. Prespecified transformations that do not learn from the data can be applied consistently, but learned preprocessing should be estimated using training observations only.
Worked Example: Computing Log Loss for Held-Out Predictions
In a hypothetical validation fold, four units have treatment assignments \(1,1,0,0\). One candidate model gives held-out propensity predictions \(0.80, 0.70, 0.20, 0.30\), respectively. Its average log loss is
For each treated unit, the loss is \(-\log(\hat e_i)\); for each untreated unit it is \(-\log(1-\hat e_i)\). The predictions assign reasonably high probability to the observed assignments, so the losses are relatively small. The value \(0.2899\) is a validation score for this fold, not an estimate of treatment effect and not a balance statistic.
In practice, repeat the calculation for every held-out fold and aggregate the losses. Do not calculate this validation score from predictions produced by a model that was fitted using those same observations: those are training losses and can be overly optimistic.
Cross-Validation Is Not Balance Assessment
The causal purpose of propensity-score adjustment is to make treated and untreated units comparable on measured covariates through a design such as matching or weighting. Cross-validated treatment-prediction loss instead evaluates how well the candidate predicts assignment in held-out data. These goals can point in different directions. A model can predict treatment well without producing the balance needed for the intended analysis; conversely, a modest improvement in predictive loss does not guarantee a meaningful improvement in balance.
After selecting the tuning choice, refit that candidate on the full analysis sample and conduct the balance assessment required by the planned design. Examine covariate balance after matching or weighting, using appropriate summaries such as standardized mean differences, and inspect whether the comparison is supported by the available units. If balance is inadequate, the predictive winner does not override that diagnostic. Reconsider the specification or design and reassess balance; do not describe the cross-validation result itself as proof of causal comparability.
Worked Example: A Predictive Winner That Needs a Balance Check
In a hypothetical study, three penalty strengths have the following mean cross-validated log losses. The last column is the largest absolute standardized mean difference across measured covariates after applying the planned matching procedure with the model refit on the full sample.
| Penalty strength | Mean validation log loss | Largest absolute standardized difference |
|---|---|---|
| \(\lambda=0.1\) | 0.580 | 0.16 |
| \(\lambda=0.5\) | 0.590 | 0.07 |
| \(\lambda=1\) | 0.620 | 0.11 |
The loss criterion selects \(\lambda=0.1\), since \(0.580\) is the smallest validation loss. Yet its largest absolute standardized difference is \(0.16\), larger than the corresponding \(0.07\) for \(\lambda=0.5\). This illustrates why the two columns cannot be treated as interchangeable: one summarizes prediction, the other summarizes a feature of the matched comparison.
The analyst should report the prespecified selection rule and the balance results separately. If the balance assessment does not support the planned comparison, the low predictive loss does not make the analysis acceptable. A balance-informed choice among candidates can be considered when its rule is specified and justified for the design, but it is a different selection strategy from minimizing prediction loss.
Diagnostics, Alternatives, and Limitations
Several checks help identify a misleading comparison. First, inspect fold-specific losses, not only their average. One unusually poor fold may signal that the selected result depends heavily on a particular partition or subset. Second, verify that each fold contains relevant treatment groups and that its structure reflects the intended use of the model. Stratifying folds by treatment can help avoid folds with very few observations in one group, especially in smaller samples. Third, repeat the comparison under more than one reasonable fold assignment when the result is sensitive; a tiny difference in mean loss should not be treated as a decisive ranking.
A single held-out split is simpler but can be highly dependent on which observations happen to be set aside. Cross-validation uses multiple partitions and is often more data-efficient for comparing candidates, though it still has sampling variability. Information criteria such as AIC or BIC offer another way to compare fitted models, but they are not the same as held-out validation, and they do not assess balance. Whatever approach is used to tune or compare models, final balance diagnostics remain necessary for propensity-score design.
- Fold leakage: fitting a selection or preprocessing step once on the full data before cross-validation lets validation information influence the model.
- Wrong target: choosing a model by accuracy or AUC and interpreting that choice as evidence of balance confuses treatment prediction with the causal design objective.
- Overinterpreting small differences: the candidate with the numerically smallest average loss may not be meaningfully better, particularly if fold-specific results vary substantially.
- Unrepresentative folds: random folds may be inappropriate for clustered or time-ordered data when validation should represent new clusters or later time periods.
Cross-validation also cannot correct omitted confounders, establish positivity, or guarantee that a chosen functional form is scientifically adequate. It selects among the candidates supplied to it, for the criterion and fold construction used. These limitations are reasons to interpret it as one part of model development rather than a complete validation of a causal analysis.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- In \(K\)-fold cross-validation, why should a validation observation not be used to fit the model that predicts its treatment assignment?
- Two candidates have mean cross-validated log losses of \(0.48\) and \(0.52\). What can be concluded, and what cannot be concluded?
- Why can AUC be insufficient as the sole criterion for choosing a propensity model?
- A covariate-selection step is performed once using the complete data before cross-validation. What is the concern, and how should the step be handled?
- After tuning, a candidate has the lowest validation loss but the planned matched sample has poor covariate balance. What should the analyst conclude?
Solutions. (1) If the observation is used in fitting, its prediction is in-sample and can reflect information from its own treatment assignment. Holding it out gives a more honest estimate of performance on observations not used to fit that fold-specific model. (2) The first candidate has the lower estimated log loss among these two under the chosen folds and criterion. The difference does not prove better covariate balance, causal validity, or a meaningful advantage beyond sampling variability. (3) AUC measures ranking discrimination, not the accuracy of probability values; propensity analyses need estimated probabilities for design procedures, and AUC does not directly assess balance. (4) Selection has used information from the validation observations, creating leakage. Repeat the selection using only each training portion, then apply that selected step to its held-out fold. (5) The predictive criterion selected the candidate, but the balance diagnostic raises a separate problem. The analyst should not claim adequate comparability based on low validation loss and should reconsider the specification or design, then reassess balance.