Why Cross-Fit Treatment Probabilities?
Inverse probability weights depend on estimated treatment probabilities. When a flexible model is fitted and evaluated on the same observations, its predictions can partly reflect the treatment assignments it was trained to explain. This can make fitted probabilities look unusually good in-sample and can make weights misleading. Cross-fitting separates model fitting from probability prediction for each observation: the model that predicts an observation’s treatment probability is fitted without using that observation.
Cross-fitting is a way to obtain predictions, not a treatment model in itself. The model can be logistic regression, a random forest, a Super Learner, or another estimator. Whatever procedure is used—including model selection and tuning—must be confined to the training data for the relevant fold. Cross-fitting reduces direct reuse of an observation in its own fitted prediction; it does not guarantee correct probabilities or valid causal conclusions.
Definition and Construction
For binary treatment, write \(e_0(L)=P(A=1\mid L)\) for the true treatment probability and \(\hat e_{-k}(L)\) for the estimate fitted without fold \(k\). For a treated observation \(i\in I_k\), the unstabilized inverse probability of treatment weight is \(1/\hat e_{-k}(L_i)\). For a control observation it is \(1/\{1-\hat e_{-k}(L_i)\}\). Equivalently, the weight for the treatment actually received is \(A_i/\hat e_{-k}(L_i)+(1-A_i)/\{1-\hat e_{-k}(L_i)\}\).
With multiple treatment categories, use the held-out prediction for the category actually received: if \(A_i=a\), that probability is \(\hat P_{-k}(A=a\mid L_i)\), and the corresponding unstabilized weight is its reciprocal. For repeated treatment decisions, fit the relevant conditional treatment mechanism without the held-out individual and multiply the held-out conditional probabilities along that individual’s observed treatment history. In repeated-decision data, all records from the same individual must stay in the same fold.
A practical construction follows four steps:
Partition independent individuals, not rows from the same individual. Use folds large enough to leave adequate training data, and assign folds independently of the observations when relying on the independence argument below.
For fold \(k\), fit the entire treatment-probability procedure using indices outside \(I_k\). Any tuning or selection used to produce the prediction must also use only this training sample.
Evaluate the fitted mechanism at the covariates of observations in \(I_k\), recording the probability of their observed treatment category.
Take the reciprocal of the probability of the treatment actually received, or multiply reciprocals across decision times. Check for failed fits, extreme probabilities, and extreme weights before using them.
Worked Example: Binary Weights From Two Held-Out Folds
Suppose six independent individuals are divided into two folds. The models fitted outside each fold produce the following probabilities for the treatment actually observed. For binary treatment, the table shows \(A_i\), the held-out estimate \(\hat e_{-k}(L_i)\), and the relevant unstabilized weight.
| Fold | Individual | \(A_i\) | \(\hat e_{-k}(L_i)\) | Weight |
|---|---|---|---|---|
| 1 | 1 | 1 | 0.20 | \(1/0.20=5\) |
| 1 | 2 | 0 | 0.50 | \(1/(1-0.50)=2\) |
| 1 | 3 | 1 | 0.80 | \(1/0.80=1.25\) |
| 2 | 4 | 0 | 0.40 | \(1/(1-0.40)\approx1.667\) |
| 2 | 5 | 1 | 0.60 | \(1/0.60\approx1.667\) |
| 2 | 6 | 1 | 0.25 | \(1/0.25=4\) |
In fold 1, the model was fitted on individuals 4–6; in fold 2, it was fitted on individuals 1–3. The sum of the six weights is \(5+2+1.25+1.667+1.667+4\approx15.584\), so their sample average is approximately \(15.584/6=2.597\). This large average is not an arithmetic error: inverse probability weights need not average to one in a small sample, particularly when the displayed weights are unstabilized.
Why Held-Out Predictions Have a Useful Separation
Proof. Conditional on the training observations, \(\hat f_{-k}\) is fixed. Independence of the fold assignment and the iid observations means \(O_i\) is independent of those training observations and retains the population distribution. Thus, for every fixed fitted rule \(f\), the conditional expectation of \(g(f,O_i)\) is the integral of \(g(f,o)\) with respect to the population distribution of \(O\). Substituting the conditionally fixed \(\hat f_{-k}\) gives the displayed equality. Measurability and integrability ensure these conditional expectations are defined. \(\square\)
The theorem does not say that every out-of-fold probability is accurate, or that an average of dependent fold-specific predictions is an independent sample. It says something narrower: an observation is not in the training data for the rule used to predict it. This is the separation that ordinary in-sample evaluation lacks.
A Consistency Result for Cross-Fitted Weighted Averages
The next result makes precise one circumstance in which cross-fitting supports an ordinary weighted average. It concerns a binary treatment and an unstabilized treated weight; it does not establish a causal effect without the additional assumptions required for a causal interpretation.
Proof. Compare the cross-fitted sum to the same sum using the true probability. The absolute difference is bounded by $$ \frac{1}{n}\sum_{k=1}^K\sum_{i\in I_k}|Y_i|A_i \left|\frac{1}{\hat e_{-k}(L_i)}-\frac{1}{e_0(L_i)}\right|. $$ Since \(\hat e_{-k}\ge c\) and \(e_0\ge\eta\), the reciprocal difference is at most \(|\hat e_{-k}(L_i)-e_0(L_i)|/(c\eta)\). The displayed bound is therefore at most $$ \frac{B}{c\eta}\sum_{k=1}^K\frac{|I_k|}{n} \left\{\frac{1}{|I_k|}\sum_{i\in I_k}|\hat e_{-k}(L_i)-e_0(L_i)|\right\}. $$ Conditional on the training data for fold \(k\), the held-out covariates are iid from the population distribution. The conditional expectation of the average in braces is the conditional \(L_1\) error in the theorem assumption. Its conditional variance is at most \(1/|I_k|\), because the absolute differences are bounded by one. Since \(|I_k|\) grows proportionally to \(n\), conditional Chebyshev’s inequality and the assumed \(L_1\) convergence imply that each fold average converges to zero in probability. There are finitely many folds, so their weighted sum converges to zero in probability. Finally, the iid law of large numbers gives $$ \frac{1}{n}\sum_{i=1}^n\frac{A_iY_i}{e_0(L_i)} \xrightarrow{p}E\left[\frac{AY}{e_0(L)}\right], $$ which proves the result. \(\square\)
The expectation on the right has a useful algebraic interpretation: $$ E\left[\frac{AY}{e_0(L)}\right] = E\left[\frac{E[AY\mid L]}{e_0(L)}\right] = E\left[E[Y\mid A=1,L]\right]. $$ The second equality follows from \(E[AY\mid L]=e_0(L)E[Y\mid A=1,L]\). If \(Y\) is an observed outcome and a causal interpretation is intended, further identification assumptions are needed; cross-fitting alone supplies none of them.
Worked Example: Fold-Specific Training and Held-Out Probabilities
There are eight individuals in two folds of four. A simple treatment model uses a constant probability equal to the treatment proportion in its training sample. Fold 1 contains two treated and two control individuals; fold 2 contains one treated and three control individuals.
To predict fold 1, train on fold 2. The fitted probability is \(1/4=0.25\). The two treated individuals in fold 1 each receive weight \(1/0.25=4\); the two controls each receive weight \(1/(1-0.25)=4/3\). The total fold 1 weight is \(2(4)+2(4/3)=32/3\).
To predict fold 2, train on fold 1. The fitted probability is \(2/4=0.50\). The one treated individual has weight \(1/0.50=2\); the three controls each have weight \(1/(1-0.50)=2\). The total fold 2 weight is \(2+3(2)=8\).
The total is \(32/3+8=56/3\), and the average weight is \((56/3)/8=7/3\). If instead the full sample proportion \(3/8\) were used for every individual, treated weights would all be \(8/3\) and control weights would all be \(8/5\). That calculation uses each individual’s treatment in the probability used to weight that same individual. The example does not prove that the cross-fitted weights are better in this small sample; it shows exactly how the training sample differs.
Worked Example: Cross-Fitting a Two-Decision Weight
Suppose one individual belongs to fold 2, and both of that individual’s treatment decisions are predicted by models trained without fold 2. The held-out model assigns probability \(0.40\) to the observed first decision, \(A_1=1\). Given the observed history at the second decision, it assigns probability \(0.75\) to \(A_2=1\). The individual instead received \(A_2=0\), whose estimated conditional probability is \(1-0.75=0.25\).
The estimated probability of the observed treatment history is the product \(0.40\times0.25=0.10\). The corresponding unstabilized history weight is \(1/0.10=10\). Both conditional probabilities were predicted without using any records from this individual. Splitting the two decisions into different folds would violate the required individual-level separation and could expose one decision record to training while another record from the same individual is held out.
Diagnostics, Alternatives, and Limitations
Cross-fitting addresses one source of dependence between a fitted model and its evaluation observation. It does not cure positivity problems, misspecification, inadequate sample size, or data leakage through an incorrect fold design. A treatment model can predict probabilities out of fold and still assign probabilities near zero to observed treatments, producing extreme weights.
- Check the partition. Confirm that no individual appears in both the training and held-out data. For clustered observations, assign whole clusters to folds.
- Check the fitting pipeline. Preprocessing, feature selection, tuning, and ensemble selection must not use held-out treatment outcomes. Otherwise the prediction is not genuinely held out.
- Inspect predicted probabilities and weights by fold. Look for probabilities at or near zero for observed treatment categories, very large weights, failed fits, and folds with little treatment variation.
- Check stability across partitions. Substantial changes when the random fold assignment changes can indicate a small sample, unstable fitting, or sparse treatment patterns. A stable partition is not proof of a correct model.
A single holdout also separates fitting from evaluation, but it uses only that holdout for prediction or evaluation and can waste data or yield a noisy result. Ordinary in-sample fitting uses all observations for model fitting but does not provide the same separation. Cross-fitting lets each observation be held out once while most observations train each fold-specific model. The fold-specific fits overlap, so the resulting predictions and weights are not mutually independent; the procedure is not a substitute for a valid uncertainty analysis.
A common mistake is to claim, “These are cross-fitted probabilities, so the weights are unbiased and extreme weights are solved.” The reasoning is wrong: excluding an observation from its own model fit does not make an estimated probability equal to the true conditional probability, nor does it ensure that the true probability is bounded away from zero. The correction is to treat cross-fitting as a prediction-separation strategy. Model adequacy, positivity, the construction of the treatment history, and the assumptions for a causal estimand still require separate scrutiny.
Check Your Understanding
Answer each question, then check the concise solutions below.
- For a treated individual in fold 3, which observations may be used to fit the probability used in that individual’s weight?
- A binary treatment model predicts \(\hat e=0.20\) for a held-out treated individual. What is the unstabilized weight?
- In repeated-decision data, why should all records from one individual be assigned to the same fold?
- Does cross-fitting by itself establish that estimated treatment probabilities are correct or that positivity holds?
Solutions. (1) Use observations outside fold 3 only, including for tuning and model selection. (2) The weight is \(1/0.20=5\). (3) Otherwise, records from the held-out individual could enter the training data for another decision, defeating the individual-level separation. (4) No. Cross-fitting separates prediction from fitting for each held-out observation; it does not guarantee model correctness, adequate overlap, or causal identification.