Why Predict Each Observation From a Model That Excludes It?
A propensity score is the conditional probability of receiving treatment given measured pretreatment covariates. Write \(A_i\in\{0,1\}\) for the treatment indicator, \(X_i\) for the covariate vector, and \(e(X_i)=P(A_i=1\mid X_i)\) for the true propensity score. An estimated score is produced by fitting a model to observed treatment assignments and covariates.
When a flexible model is evaluated on the same observations used to fit it, its predictions can look more accurate than they will be on new observations. The model has had an opportunity to adapt to those observations. Out-of-fold prediction addresses this particular form of optimism: each observation receives a prediction from a model that was fitted without that observation. It does not, by itself, ensure a correct propensity model, remove confounding, or guarantee that any later analysis is valid.
In contrast, an in-sample fitted score for observation \(i\) is produced by a model whose training data included \(i\). OOF describes how a prediction was generated; it does not describe a different definition of the true propensity score. Both kinds of estimate aim to predict \(P(A=1\mid X)\), but they use different fitted rules and can have different prediction errors.
The Fold-Based Algorithm
Choose the number of folds \(K\), assign each observation to one fold, and use the same model-building procedure for each training subset. A standard implementation is:
Partition the \(n\) observations into \(K\) nonoverlapping sets. Each observation must be in exactly one held-out fold.
For each \(k\), estimate the propensity model using all observations not in \(F_k\). The model-building steps that use treatment information must also be confined to this training subset.
Apply the fitted rule to \(X_i\) for each \(i\in F_k\), storing its estimated probability of treatment.
After all folds have been held out, each observation has exactly one OOF estimate, obtained from a model that did not use its treatment assignment to fit that rule.
The held-out observation's covariates are used to make its prediction, as they should be; what is excluded is that observation's contribution to fitting the prediction rule. If a model is tuned or selected using treatment outcomes, that selection must also avoid the held-out fold. One approach is to perform tuning within each outer training subset, then fit the chosen model on that subset and predict its outer fold. Tuning once on the full sample and then calling the resulting predictions OOF can leak held-out treatment information into model selection.
What Does an OOF Prediction Estimate?
A useful justification requires an explicit sampling and fold-assignment condition. Suppose the observations \((X_i,A_i)\) are independent and identically distributed, and the fold assignment is generated independently of the observations. Conditional on the fold memberships, the held-out observations are then independent draws from the population and are independent of the training observations outside their fold. This condition matters: a fold assignment based on covariates can select a different covariate distribution for the held-out fold.
Let \(\ell(A,\hat p)\) be a prediction loss, such as the log loss for a binary treatment. For a fixed fold \(F_k\), conditional on the training data \(D_{-k}\), the fitted rule \(\hat f^{(-k)}\) is fixed. Each held-out pair has the population distribution under the stated independent-assignment condition. Therefore:
The right side is the population prediction risk of this particular fitted rule, evaluated on a new population draw. Averaging across folds gives a cross-validated estimate of the average risk of rules trained on roughly \(n(1-1/K)\) observations. It is not exactly the risk of a rule fitted to all \(n\) observations: each fold has a different rule, and each rule uses less data than the full-sample fit.
If fold assignment depends on \(X_i\), the equality to the population risk need not hold. A held-out loss may instead estimate risk under the covariate distribution conditional on belonging to that fold. If assignment depends on treatment labels, it can similarly change the target distribution of held-out treatment-covariate pairs. Such designs may be useful for specific purposes, but their evaluation target must be stated rather than assumed to be the general population.
Worked Example: Computing OOF Log Loss
Consider six hypothetical observations assigned to three folds. The table gives each treatment indicator and the prediction made by a model trained without that observation's fold. For binary treatment, the log loss for observation \(i\) is \(-\log(\hat p_i)\) when \(A_i=1\), and \(-\log(1-\hat p_i)\) when \(A_i=0\).
| Observation | Fold | \(A_i\) | OOF estimate \(\hat p_i\) | Log loss |
|---|---|---|---|---|
| 1 | 1 | 1 | 0.70 | \(-\log(0.70)=0.3567\) |
| 2 | 1 | 0 | 0.30 | \(-\log(0.70)=0.3567\) |
| 3 | 2 | 1 | 0.65 | \(-\log(0.65)=0.4308\) |
| 4 | 2 | 0 | 0.40 | \(-\log(0.60)=0.5108\) |
| 5 | 3 | 1 | 0.80 | \(-\log(0.80)=0.2231\) |
| 6 | 3 | 0 | 0.25 | \(-\log(0.75)=0.2877\) |
The average OOF log loss is
Thus, the mean held-out log loss in this hypothetical sample is 0.3610 per observation. A smaller log loss indicates better probabilistic predictions on this loss scale when comparing models on the same evaluation data. This one number does not establish that the estimated scores are adequate for a causal analysis; it summarizes predictive performance under the fold design and loss used.
When OOF Prediction Can Help
OOF predictions are particularly useful when a flexible learner can adapt closely to the training observations. Trees, forests, boosting, and ensembles can yield predictions that appear unusually confident on the data used to fit them. Excluding each observation from its prediction reduces the direct self-influence that can make in-sample predictions optimistic. This is a prediction-evaluation advantage, not a guarantee that the OOF scores will be less extreme for every observation or more accurate under every metric.
OOF predictions are also useful when a downstream procedure needs a prediction for each observed unit but it is important that the unit's own treatment assignment did not help train the rule that generated its score. For example, when constructing a prediction-based quantity for further analysis, this separation can limit a particular source of reuse of the same data. The validity of that later analysis still depends on its own assumptions and on how all model-selection steps were carried out.
Worked Example: The Fold-Assignment Condition
Suppose hypothetical covariates \(X_i\) are independent draws from a uniform distribution on \([0,1]\). Consider assigning observations with \(X_i<0.5\) to one fold and those with \(X_i\geq0.5\) to another. This assignment uses no treatment labels, but it is not independent of the observations.
For an observation known to be in the first fold, \(X_i\) is uniform on \([0,0.5)\), not on \([0,1]\). For instance, the population mean of \(X\) is \(0.5\), whereas its conditional mean in the first fold is \(0.25\). Consequently, the average held-out loss in that fold evaluates predictions over a covariate distribution concentrated below \(0.5\); it cannot be identified as the ordinary population risk merely because the treatment labels were not used to assign folds.
For population-style evaluation, generate fold membership independently of the observed covariates and treatment assignments (often by random allocation). If folds are deliberately formed using covariates or treatment, describe the conditional target and avoid interpreting the resulting held-out average as an unconditional new-draw population risk without additional justification.
Worked Example: Training-Set Size and the Meaning of the Estimate
Suppose a hypothetical data set has \(n=120\) observations and is divided into \(K=6\) equal folds. Each fold contains \(120/6=20\) observations, so each model is trained on \(120-20=100\) observations and predicts 20 held-out observations. The pooled OOF loss consequently summarizes predictions from six rules, each trained on 100 observations.
If the same data were divided into \(K=10\) equal folds, each model would use \(120-12=108\) training observations and predict 12 held-out observations. That can make each fitted rule closer in sample size to a full-data fit, but it does not make the OOF estimate identical to the risk of the final rule fitted on all 120 observations. The folds still yield distinct training samples and distinct fitted rules. The choice of \(K\) is therefore a tradeoff in the training size of each rule and the construction of the evaluation, not a way to remove all estimation error.
Diagnostics, Alternatives, and Limitations
Before using OOF predictions, check that every observation has one and only one held-out prediction, and that no held-out observation contributed to its model's fitting, preprocessing, tuning, or treatment-informed feature selection. Confirm that each training fold contains enough treated and untreated observations for the chosen estimator to be fitted. Sparse categories or small treatment counts can make some fold-specific fits unstable even when a full-sample fit appears workable.
Compare held-out predictive performance across folds, not just as one pooled average: a large difference between folds can indicate instability or a fold-specific shift. Inspect whether predicted probabilities are systematically too high or too low relative to observed treatment frequencies in held-out data. Such checks diagnose prediction problems; they do not establish covariate balance or resolve causal identification assumptions. OOF probabilities can still be extreme, and a flexible or poorly specified model can still predict badly.
| Approach | What it produces | Main caution |
|---|---|---|
| In-sample fitted scores | Predictions from a rule fitted using the same observations | Can reflect overfitting and may understate new-observation prediction error |
| Out-of-fold scores | One prediction per observation from a rule excluding its fold | Rules use fewer observations; valid population-risk interpretation requires an appropriate fold design |
| Full-sample refit | A final prediction rule fitted using all available observations | Its predictions on those same observations are not OOF predictions |
The full-sample refit is often useful when a final prediction rule is needed, while OOF estimates are useful for held-out evaluation or analyses that require predictions separated from each unit's own treatment information. Do not silently substitute one for the other: state which scores were used and how they were generated. OOF estimation also does not repair unmeasured confounding, lack of overlap, poor measurement of covariates, or misspecification. It controls one aspect of prediction reuse, not every source of bias.
Common Pitfall
A second frequent mistake is to fit folds correctly but select the model using all treatment labels first. If the held-out labels influenced which model, variables, or tuning settings were chosen, those predictions are not fully out of fold with respect to the complete estimation procedure. Keep treatment-informed decisions inside the training data for each held-out fold.
Check Your Understanding
Answer each question, then compare your reasoning with the brief solutions.
- In a five-fold procedure, what data are used to predict an observation in fold 3?
- State a condition on fold assignment that supports interpreting held-out loss as population prediction risk.
- A researcher randomly forms folds, but chooses the tuning parameter using treatment labels from the full sample. What is the concern?
- With \(n=90\) observations and \(K=3\) equal folds, how many observations train each fold-specific model?
- Does an OOF propensity estimate guarantee covariate balance or eliminate unmeasured confounding?
Solutions. (1) Fit the rule on folds 1, 2, 4, and 5, then predict the observations in fold 3. (2) Fold assignment should be independent of the observations, so that conditional on the assignment the held-out data have the population distribution and are independent of the training data. (3) Full-sample tuning uses held-out treatment information, so the complete prediction procedure is not independent of those labels; tune within each outer training set instead. (4) Each fold has \(90/3=30\) observations, leaving \(90-30=60\) for training. (5) No. OOF construction limits direct reuse of each observation in fitting its prediction rule, but it does not ensure balance or address unmeasured confounding.