Calibration and Validation of CATE Predictions

Why Validate Conditional Effect Predictions?

A model can produce a conditional average treatment effect (CATE) prediction for every covariate profile and still be poorly calibrated. It may rank people reasonably well while systematically overstating effect sizes, or it may look accurate in the data used to fit it but fail on new observations. Validation asks whether predicted effects correspond to observed effect differences in a population not used to construct those predictions.

The target is a conditional mean effect, not an observable individual treatment effect. Write \(\tau(x)=E[Y(1)-Y(0)\mid X=x]\), where \(Y(1)\) and \(Y(0)\) are potential outcomes and \(X\) denotes observed pretreatment covariates. A fitted model, evaluated at \(X_i\), gives a prediction \(\hat{\tau}(X_i)\). Validation compares these predictions with an appropriately adjusted estimate of the conditional effect, not with the individual’s observed outcome or an unadjusted treated-versus-control difference.

Identification and a Validation Pseudo-Outcome

For observed data \((Y,W,X)\), let \(W\) be a binary treatment indicator and define \(e(x)=P(W=1\mid X=x)\) and \(m_w(x)=E[Y\mid W=w,X=x]\) for \(w\in\{0,1\}\). To interpret \(m_1(x)-m_0(x)\) as \(\tau(x)\), the usual identification conditions are needed: consistency, no interference between units, conditional exchangeability \((Y(1),Y(0))\perp W\mid X\), and positivity \(0<e(X)<1\) in the target population. These conditions connect observed conditional outcome means to potential-outcome means; they do not guarantee precise estimation where treatment probabilities are near zero or one.

A useful validation quantity is the doubly robust, or augmented inverse-probability-weighted, pseudo-outcome. If the nuisance functions \(e,m_1,m_0\) are known, define

In practice, nuisance functions are estimated. For validation, use estimates obtained without fitting to the validation observation—for example, fit nuisance models on training folds and predict them for held-out folds. This cross-fitting helps prevent overfitting from making the pseudo-outcome appear artificially aligned with the CATE prediction. The CATE prediction itself should also be produced without using the validation outcome.

The identity follows by conditioning on \(X\). The treated residual term has conditional expectation zero:

The same argument gives zero for the control residual term, because \(E[Y-m_0(X)\mid W=0,X]=0\) and \(P(W=0\mid X)=1-e(X)\). The remaining term is \(m_1(X)-m_0(X)=\tau(X)\) under the identification conditions. Thus \(\Gamma\) is an unbiased, conditional-mean target for calibration. It is not a measured treatment effect for one person: it can be noisy, and its value depends on the observed outcome and treatment assignment.

Calibration-in-the-Large and Calibration Slope

On held-out observations, let \(\hat{\tau}_i\) denote the predicted effect and \(\Gamma_i\) its pseudo-outcome. A simple calibration-in-the-large summary is the mean prediction error:

Here \(n_{\mathrm{val}}\) is the number of validation observations. A value near zero indicates little average offset in the validation sample, while a positive value indicates that the pseudo-outcomes average above predictions. This summary can conceal important variation: overprediction in one region may cancel underprediction in another.

A calibration slope summarizes whether differences in predictions track differences in pseudo-outcomes. Fit the validation regression

The ideal linear calibration relationship has \(a=0\) and \(b=1\). A slope below one often indicates predictions that vary more than the corresponding validation effects, while a slope above one often indicates predictions that vary less. Those interpretations are conditional on the fitted intercept and on the linear summary being useful. A slope is not a test of all possible calibration failures: nonlinear departures can remain even when the slope is one. Since \(\Gamma_i\) is noisy, uncertainty in the slope can also be substantial, especially with limited validation data or weak overlap.

Grouped Calibration Plots

A grouped plot offers a visual check that does not force the entire calibration relationship to be linear. Sort held-out observations by \(\hat{\tau}_i\), divide them into groups (often prediction quantiles), and calculate within each group the mean prediction and mean pseudo-outcome. Plot the latter against the former, with the 45-degree line as the ideal reference. Group sizes and uncertainty intervals should be shown or reported where feasible; intervals should account for the validation design, including clustering when relevant.

The plot estimates average behavior within prediction ranges, not accuracy for each observation. Grouping smooths noisy individual pseudo-outcomes, but it also hides within-group variation and can make results sensitive to the number and boundaries of groups. A calibration curve that departs from the reference line may reveal systematic over- or underprediction in a region even when the overall mean error is near zero.

Held-Out Design and Overlap Diagnostics

A credible assessment separates model development from validation. Fit the CATE prediction model using training data, then evaluate predictions on observations whose outcomes did not inform that model. In cross-validation, each observation’s prediction should be made by a model that excluded that observation. If nuisance models are also estimated, their predictions for the validation observations should likewise avoid using those observations’ outcomes. Repeatedly adjusting a model in response to the same validation results turns that set into development data; a final assessment then requires a fresh holdout or a clearly defined resampling procedure.

The validation sample should represent the population for which the predictions are intended. If the covariate distribution or treatment assignment mechanism differs, calibration summaries may change even if the fitted model is unchanged. Report the target population, the validation construction, and the distribution of predicted effects. For observational data, inspect treatment probabilities and the size of inverse-probability contributions; positivity in theory does not prevent extreme weights in finite samples.

Weak overlap can make pseudo-outcomes highly variable because residuals are divided by \(e(X)\) for treated observations and by \(1-e(X)\) for controls. In such regions, a few observations may drive a slope or group mean. Report where predictions are supported by data, inspect group counts and propensity distributions, and consider whether the estimand should be restricted to a population with adequate overlap. Trimming or restricting changes the target population and must be stated; it is not merely a numerical repair.

Common Pitfalls and Interpretation

• Validating on training outcomes. A close fit on observations used to learn the CATE model can reflect overfitting. Use held-out or properly cross-fitted predictions.
• Treating a pseudo-outcome as an individual effect. \(\Gamma_i\) is a noisy score with a conditional-mean property, not the unobserved \(Y_i(1)-Y_i(0)\).
• Using raw outcome differences as validation. In observational data, differences by treatment status can reflect confounding. Use an adjustment that targets the conditional effect under stated assumptions.
• Calling a slope of one proof of calibration. A linear slope summarizes one feature of the relationship and can miss nonlinear miscalibration or offsetting errors across groups.
• Ignoring weak overlap. Large inverse-probability factors can make validation summaries unstable. Report this limitation rather than interpreting an extreme pseudo-outcome as a large treatment effect.
• Overlooking the target population. Calibration in one population does not automatically transfer to a population with a different covariate or treatment distribution.

Calibration and validation complement model-building diagnostics: they ask whether predicted conditional effects correspond to effect differences in data not used to create the predictions. No single summary answers that question completely. A useful assessment combines held-out evaluation, a mean-error summary, a slope, grouped plots, and explicit checks of overlap and target-population relevance.