Why Cross-Fit Propensity Scores?
A propensity model may use many candidate covariates and regularization to estimate the probability of treatment. When the same observations both determine the model and receive its fitted scores, a fitted score can partly reflect that observation’s own treatment assignment. Cross-fitting separates those roles: each person’s score is predicted by a model fitted without that person. The construction is especially useful when the propensity model is estimated with a high-dimensional procedure, but it is not a substitute for choosing an appropriate causal design or model.
The key object is the true propensity score \(e_0(X_i)=P(A_i=1\mid X_i)\), where \(A_i\) is the binary treatment indicator and \(X_i\) is the vector of pre-treatment covariates. The fitted score is an estimate of this conditional probability. In a regularized logistic model, a training sample is used to estimate coefficients, often with an \(L_1\) penalty; the resulting model predicts treatment probabilities for observations it did not use in fitting.
For a regularized logistic propensity model, one possible training-fold fit is
Here \(n_{-k}\) is the training-set size, \(\lambda_k\) is the penalty level, and \(\operatorname{expit}(u)=1/(1+\exp(-u))\). Other regularization schemes can be used; the defining feature of cross-fitting is the fold-specific exclusion, not a particular penalty. If a penalty is selected from the data, its selection must also avoid using held-out treatment labels in a way that leaks information into their predictions. A safe general approach is to select or tune within the training data for each fold, or to fix the tuning procedure in advance and apply it separately within each training set.
Constructing and Interpreting the Folds
The partition should be specified so that, for the conditional argument below, fold assignment is independent of the held-out treatment assignments. A simple sufficient design is to generate the fold labels randomly, independently of all observed records, before fitting the models. With independent observations, fitting fold \(k\)'s model only on other observations then ensures that the held-out person’s treatment is not used to construct their score.
This condition matters. Independence of the observations and exclusion of person \(i\) from the training set do not alone establish that the treatment probability remains \(e_0(X_i)\) after conditioning on fold membership. If fold membership was assigned using treatment labels, conditioning on that membership may change the treatment distribution. For example, putting every treated person in one fold and every untreated person in another makes treatment status fixed within each fold. A conditional-mean argument that ignores this treatment-dependent assignment is not valid. Treatment-stratified folds can be useful computationally, particularly when treatment is rare, but their conditional justification is not the same as that for treatment-independent random folds.
The following short derivation states exactly what the usual argument provides. Assume independent observations, a split generated independently of the observed records, and a fit for fold \(k\) that uses only observations outside \(I_k\). Conditional on the training data, the split, and \(X_i\), the fitted prediction \(\hat e_i^{(-k)}\) is fixed. Under these assumptions, the held-out \(A_i\) has its original conditional treatment probability:
It follows by subtracting the fixed prediction that
This is not generally zero: it becomes zero only if the fitted prediction equals the true propensity at that covariate value (or under an appropriate averaging condition). Thus cross-fitting prevents the held-out treatment assignment from directly influencing its own fitted score; it does not make propensity predictions unbiased or guarantee correct specification.
Worked Example: Assigning Out-of-Fold Predictions
Suppose six hypothetical patients are assigned to three folds, two patients per fold. An \(L_1\)-penalized logistic model is fitted separately on the four patients outside each fold. The fitted probabilities shown below are predictions from the model that excluded the patient’s fold.
| Patient | Fold | Treatment \(A_i\) | Out-of-fold score \(\hat e_i^{(-k)}\) |
|---|---|---|---|
| 1 | 1 | 1 | 0.62 |
| 2 | 1 | 0 | 0.28 |
| 3 | 2 | 1 | 0.71 |
| 4 | 2 | 0 | 0.43 |
| 5 | 3 | 1 | 0.77 |
| 6 | 3 | 0 | 0.34 |
For patient 3, the score is obtained by fitting on patients 1, 2, 5, and 6, then evaluating that fitted model at \(X_3\). It is not the score from a model fitted on all six patients. The same exclusion rule applies to each row. The vector of scores to use in subsequent calculations is therefore \((0.62,0.28,0.71,0.43,0.77,0.34)\), with one held-out prediction per patient.
The split is appropriate for the conditional argument if the fold assignments were generated independently of the treatment assignments. The table itself does not reveal how the folds were assigned, so that design condition must be checked from the analysis procedure rather than inferred from the predictions.
Using Cross-Fitted Scores for Weighting
For an average treatment effect, an inverse-probability weighted estimator can use each person’s out-of-fold score. A normalized (Hájek) form compares the weighted outcome mean among treated people with the weighted mean among untreated people:
where \(k(i)\) is the fold containing person \(i\), and \(Y_i\) is the observed outcome. The formula normalizes each treatment group’s weights to sum to one. It does not by itself establish a causal interpretation: that also depends on the target estimand, consistency, conditional exchangeability given the adjusted covariates, positivity, and adequate estimation of the propensity scores.
Worked Example: A Normalized Weighted Contrast
Consider six hypothetical patients with outcomes \(Y_i\), treatment indicators, and cross-fitted scores as follows. The scores are assumed to have been generated by independent folds as described above.
| Patient | \(A_i\) | \(Y_i\) (outcome units) | \(\hat e_i^{(-k)}\) |
|---|---|---|---|
| 1 | 1 | 10 | 0.60 |
| 2 | 0 | 6 | 0.30 |
| 3 | 1 | 12 | 0.70 |
| 4 | 0 | 8 | 0.40 |
| 5 | 1 | 11 | 0.80 |
| 6 | 0 | 5 | 0.20 |
For treated patients, the inverse-probability weights are \(1/0.60=1.667\), \(1/0.70=1.429\), and \(1/0.80=1.250\). The normalized treated mean is
For untreated patients, the weights are \(1/(1-0.30)=1.429\), \(1/(1-0.40)=1.667\), and \(1/(1-0.20)=1.250\). Their normalized mean is
The estimated weighted contrast is \(10.95-6.48=4.47\) outcome units. This arithmetic illustrates how the cross-fitted scores enter the estimator; the small hypothetical data set cannot establish exchangeability, positivity, or reliable causal estimation.
Diagnostics, Alternatives, and Limits
Cross-fitting changes how predictions are generated, not the criteria for judging them. Inspect the score distribution and the resulting weights, and assess covariate balance using the weighting procedure intended for the analysis. Check whether any scores are near zero or one, whether a small number of observations dominate the weights, and whether balance remains poor for important covariates or feature blocks. A held-out prediction can still be extreme or inaccurate. Cross-fitting does not repair a lack of overlap, omitted confounding, or a misspecified functional form.
The main practical alternatives have different tradeoffs:
| Approach | How predictions are formed | Main consideration |
|---|---|---|
| Full-sample fitted scores | Fit once using every observation and predict on those same observations | Uses all data for fitting, but each treatment label can affect its own fitted value. |
| Single holdout split | Fit on one subset and predict only for a separate held-out subset | Separates fitting and prediction, but discards the training subset from that held-out prediction task and can be sensitive to the split. |
| \(K\)-fold cross-fitting | Rotate the held-out fold so every observation receives a prediction from a model excluding it | Uses observations for training in some folds and prediction in another, but requires multiple fits and careful fold-specific preprocessing. |
In every approach, data-dependent preprocessing must be considered part of model fitting. For example, if feature screening, imputation rules, or tuning use treatment information, doing that step once on the full data can leak held-out information even if the final regression is refitted within folds. When a step is learned from the data, learn it using the training portion of each fold and then apply it to that fold’s held-out observations. Also verify that each training fold contains enough treated and untreated observations for the chosen fitting procedure to operate reliably.
Cross-fitting is particularly relevant when an analysis uses an estimator whose theoretical behavior benefits from separating nuisance-model fitting from evaluation of the same observations. Its guarantees depend on the estimator and its assumptions; cross-fitting alone does not supply a general guarantee of low bias, valid uncertainty intervals, or causal identification. In particular, a propensity-only weighted contrast does not become robust to misspecification merely because the scores are out of fold. Whether additional nuisance models or orthogonal estimators are appropriate is a separate methodological choice.
Common Mistakes
- Calling a score cross-fitted when the observation helped fit it. A full-sample fitted value is not out of fold. Record which training observations produced each prediction.
- Assuming a held-out treatment has its original conditional probability under any split. That equality needs a split independent of held-out treatment assignments, or a separate argument for the particular assignment design. Treatment-dependent folds require special care.
- Treating cross-fitting as a cure for misspecification. The conditional residual mean is \(e_0(X_i)-\hat e_i^{(-k)}\), not necessarily zero. Examine model adequacy and balance.
- Ignoring unstable weights. A fold-specific score can still be close to zero or one. Diagnose overlap and weight concentration rather than assuming the fold procedure has controlled them.
- Tuning or screening on held-out labels. This lets the held-out treatment assignment influence its prediction indirectly. Keep data-adaptive steps inside the training fold when their use could cause leakage.
Check Your Understanding
Answer each question, then compare with the concise solutions below.
- In four-fold cross-fitting, which observations are used to predict the score for a person in fold 2?
- Why does the conditional equality \(E[A_i\mid X_i,\text{training data},\text{split}]=e_0(X_i)\) require care when fold assignment uses treatment labels?
- A cross-fitted score is \(0.10\) for a treated person. What is that person’s treated inverse-probability weight, and what diagnostic concern does it raise?
- Does exclusion of an observation from its propensity-model fit imply that its score is unbiased for \(e_0(X_i)\)? Explain.
Solutions. (1) Use the other three folds as the training set; fold 2 is held out. (2) Conditioning on treatment-dependent fold membership can change the conditional distribution of \(A_i\), so the original propensity need not remain the conditional mean. (3) The weight is \(1/0.10=10\); it may be influential and signals a need to inspect overlap and weight concentration. (4) No. Exclusion prevents direct reuse of that person’s treatment in fitting, but model misspecification, regularization bias, and sampling variability can still make the prediction differ from the true propensity.