Tutorials › Causal Inference › Diagnosing CATE Estimates with Calibration and Overlap

Causal Learning and Policy Decisions · Tutorial 959 of 972

Diagnosing CATE Estimates with Calibration and Overlap

Use calibration plots, subgroup comparisons, and propensity-score overlap checks to diagnose the magnitudes and support of conditional treatment-effect estimates.

Advanced 10 min read

What You'll Learn

  • Define calibration of predicted conditional average treatment effects.
  • Build a calibration plot from predicted-effect groups and observed subgroup estimates.
  • Interpret calibration intercepts and slopes as diagnostic summaries.
  • Use a doubly robust score to estimate subgroup effects in observational data.
  • Inspect propensity-score overlap within the regions where effects are predicted.
  • Distinguish poor calibration, weak overlap, and uncertainty in subgroup estimates.

Why Diagnose Conditional Effect Estimates?

A conditional average treatment effect (CATE) learner can assign each unit a predicted treatment effect, but a detailed set of predictions is not automatically informative. A learner may rank units plausibly while systematically overstating the magnitude of effects. It may also report strong effects for covariate profiles that are rarely observed under one treatment condition. Calibration and overlap diagnostics address these two distinct concerns: whether predicted magnitudes agree with subgroup evidence, and whether the data support estimating effects in the regions where predictions are made.

These checks are diagnostics, not guarantees. Good calibration on observed groups does not establish that a model is correct for every covariate profile, and apparent overlap in a plot does not prove that all identification assumptions hold. The interpretation depends on how predictions and subgroup effects were obtained, the target population, and the uncertainty in the estimates.

Definition: Let \(X\) denote measured pretreatment covariates, \(Y(1)\) and \(Y(0)\) the potential outcomes under treatment and control, and \(W\in\{0,1\}\) the observed treatment. The CATE is \(\tau(X)=E[Y(1)-Y(0)\mid X]\). A fitted learner produces a prediction \(\hat{\tau}(X)\). The predictions are calibrated in the mean if \(E[\tau(X)\mid\hat{\tau}(X)]=\hat{\tau}(X)\) in the target population. In words, among units assigned the same predicted effect, the average true effect equals that prediction.

The definition concerns averages among units with a given prediction; it does not claim that an individual unit’s unobserved treatment effect equals its prediction. Since \(\tau(X)\) is not observed, calibration must be assessed indirectly. A common approach is to group units by their predicted effects and compare each group’s average prediction with an estimate of that group’s average treatment effect.

Calibration Plots and Subgroup Comparisons

To construct a calibration plot, first produce predictions for an evaluation sample that was not used to fit the effect learner, or use an appropriate sample-splitting or cross-fitting procedure. Divide the evaluation units into a small number of groups, often quantile groups based on \(\hat{\tau}(X)\). For each group, plot its mean predicted effect against an estimate of the group’s causal effect. Perfect calibration corresponds to points lying on the 45-degree line through the origin.

The subgroup effect estimate must respect the study design. In a randomized experiment, a difference in treatment and control means within a subgroup can estimate that subgroup’s average effect. In observational data, an unadjusted difference may be confounded. One option, under the identification conditions below, is to average a doubly robust score within each subgroup.

Identification and score: Write \(e(X)=P(W=1\mid X)\) for the propensity score and \(m_w(X)=E[Y\mid W=w,X]\) for the outcome regression under \(w\). If consistency, conditional exchangeability \((Y(1),Y(0))\perp W\mid X\), and positivity \(0<e(X)<1\) hold in the target population, then the CATE is identified. Given estimates \(\hat e\) and \(\hat m_w\), define the score \(\hat{\Gamma}= \hat m_1(X)-\hat m_0(X)+\frac{W}{\hat e(X)}\{Y-\hat m_1(X)\}-\frac{1-W}{1-\hat e(X)}\{Y-\hat m_0(X)\}\). Under the identification conditions and suitable estimation conditions, its conditional mean targets \(\tau(X)\) when the nuisance estimates are consistent in the appropriate doubly robust sense. A subgroup average of this score estimates the subgroup’s average effect; an individual score is noisy and is not an observed individual treatment effect.

For a subgroup \(G\), a score-based estimate is \(\hat{\tau}_G=|G|^{-1}\sum_{i\in G}\hat{\Gamma}_i\), while its mean prediction is \(\bar{\hat{\tau}}_G=|G|^{-1}\sum_{i\in G}\hat{\tau}(X_i)\). A calibration plot compares these quantities across groups. Standard errors or intervals matter: a point’s distance from the diagonal should be considered relative to uncertainty, not judged only by visual distance.

Worked Example: A Calibration Plot in a Randomized Study

Suppose a hypothetical randomized study measures a health score in points, with higher scores better. A learner was fit on separate data. In an evaluation sample, three groups have mean predicted effects of 0.8, 2.0, and 3.5 points. In each group, 100 people were assigned treatment and 100 control. The observed group means are 10.4 versus 9.2 points in the low-prediction group, 12.4 versus 10.0 in the middle group, and 14.5 versus 11.5 in the high group.

Random assignment within these groups supports estimating each group effect by its treatment-minus-control mean difference. The estimates are \(10.4-9.2=1.2\), \(12.4-10.0=2.4\), and \(14.5-11.5=3.0\) points. Thus the plotted points (mean prediction, estimated effect) are (0.8, 1.2), (2.0, 2.4), and (3.5, 3.0).

Assume, for illustration, that the outcome standard deviation is 4 points in both arms of each group. The standard error of each difference is \(SE=\sqrt{4^2/100+4^2/100}=\sqrt{0.32}=0.566\) points. An approximate 95% interval for the low-group effect is \(1.2\pm1.96(0.566)\), or 0.09 to 2.31 points. For the middle group it is \(2.4\pm1.96(0.566)\), or 1.29 to 3.51 points; for the high group it is \(3.0\pm1.96(0.566)\), or 1.89 to 4.11 points.

The low and middle estimates exceed their mean predictions, while the high estimate is below its prediction. But each interval is wide enough that the group-level evidence does not sharply distinguish these discrepancies from sampling variation. The plot suggests where magnitude miscalibration might occur; it does not establish it from point estimates alone. The standard errors here rely on the stated randomization and variance assumptions.

Calibration Intercept and Slope

A plot can be summarized by a calibration regression. Conceptually, regress estimated effects on mean predictions across evaluation units or prediction groups: \(\text{estimated effect}=\alpha+\beta\,\text{predicted effect}+\text{error}\). The ideal reference values are \(\alpha=0\) and \(\beta=1\). A nonzero intercept indicates an overall offset, while a slope below 1 is consistent with predictions that vary more than the estimated effects. A slope above 1 is consistent with predicted variation that is too small. These interpretations are diagnostic: uncertainty, grouping, estimation error, and the chosen regression weights affect the fitted values.

Worked Example: Summarizing Three Calibration Points

Suppose a hypothetical evaluation gives three equally weighted groups with mean predictions \(x=(1,2,4)\) outcome units and estimated subgroup effects \(y=(1.4,2.2,3.4)\) units. Fit the ordinary least-squares line \(y=\hat{\alpha}+\hat{\beta}x\) to these three group summaries. This is a compact descriptive summary, not a substitute for uncertainty estimates or a full calibration analysis.

The means are \(\bar{x}=7/3=2.333\) and \(\bar{y}=7/3=2.333\). The slope is \[ \hat{\beta}=\frac{\sum_j(x_j-\bar{x})(y_j-\bar{y})}{\sum_j(x_j-\bar{x})^2} =\frac{3.067}{4.667}=0.657. \] The intercept is \(\hat{\alpha}=\bar{y}-\hat{\beta}\bar{x}=2.333-0.657(2.333)=0.800\) outcome units.

The fitted slope is below 1 and the intercept is positive. In this small example, that pattern describes estimated effects as having less spread and a higher baseline than the predictions would imply under the ideal line. With only three group summaries, and no accounting for their different precision, this regression should not be treated as a decisive test of calibration.

The conditional calibration definition is stronger and more precise than a single regression summary. A line can conceal nonlinear departures: predictions may be too low in one range and too high in another while producing an overall slope near 1. The plot and group comparisons therefore remain useful even when a calibration intercept and slope are reported. Group boundaries should be set using predictions, not chosen after inspecting subgroup outcomes, to reduce outcome-guided selection.

Overlap and Extrapolation

Calibration asks whether predicted magnitudes agree with subgroup evidence. A separate question is whether the data contain treated and control units with comparable covariates in the region of interest. Positivity requires \(0<e(X)<1\) for covariate profiles in the target population. In finite samples, propensities very close to 0 or 1 can also make estimates unstable: the inverse-probability factors \(1/e(X)\) or \(1/\{1-e(X)\}\) become large, and one treatment arm may have little empirical representation.

Inspect propensity-score distributions separately by treatment status and, importantly, within groups or regions of \(\hat{\tau}(X)\). A global plot can look reassuring even if a high-predicted-effect subgroup contains almost no treated units, or almost no controls. Check the range and distribution of estimated propensities, arm-specific counts, and the amount of the target population lying near the extremes. A propensity plot cannot prove positivity at every covariate value, particularly when \(X\) is high-dimensional, and a fitted propensity score can itself be misspecified.

Worked Example: Weak Overlap in a High-Prediction Group

In a hypothetical observational evaluation sample, a learner’s high-effect group contains 40 people. Their estimated propensity scores range from 0.01 to 0.12, and 28 of the 40 scores are below 0.05. Only 3 people in the group received treatment. Elsewhere in the sample, propensity scores range more broadly, so a plot of the entire sample might obscure this concentration near zero.

For a person with \(\hat e(X)=0.01\), the treated-arm inverse-probability factor is \(1/\hat e(X)=1/0.01=100\). This means a treated observation at that score can receive very large leverage in an inverse-probability component of an estimator. The three treated people also provide sparse direct evidence about outcomes in this predicted-effect region.

The diagnosis is weak empirical overlap for the high-prediction group, not proof that its CATE is zero or that the estimate must be biased in a particular direction. The prediction may depend heavily on outcome-model extrapolation from other covariate profiles. Report the limited support and uncertainty; consider whether the target population should be restricted to a region with adequate support, while recognizing that changing the target changes the estimand.

Justification: Why the Score Can Support Group Checks

The score’s key property follows by conditioning on \(X\). For clarity, suppose the propensity and outcome regressions in the score equal their true values. Conditional exchangeability and consistency imply \(E[Y\mid W=w,X]=m_w(X)\). Thus the treated residual term has conditional expectation zero after accounting for treatment assignment:

$$ E\left[\frac{W}{e(X)}\{Y-m_1(X)\}\mid X\right] = \frac{P(W=1\mid X)}{e(X)}E[Y-m_1(X)\mid W=1,X] =1\cdot 0=0. $$

Likewise, the control residual term has conditional expectation zero because \(P(W=0\mid X)=1-e(X)\) and \(E[Y-m_0(X)\mid W=0,X]=0\). Therefore,

$$ E[\Gamma\mid X]=m_1(X)-m_0(X)=\tau(X). $$

Taking conditional expectations again gives \(E[\Gamma\mid\hat{\tau}(X)] = E[\tau(X)\mid\hat{\tau}(X)]\), provided the prediction is a well-defined function of the covariates and the evaluation procedure preserves the relevant conditioning. Consequently, comparing score averages with prediction averages across prediction-defined groups targets the calibration comparison. In practice, nuisance functions and effect predictions are estimated, so sample splitting or cross-fitting and uncertainty calculations help limit overfitting effects; neither replaces the identification assumptions.

Common Pitfalls and Practical Interpretation

  • Treating calibration as proof of identification. A prediction can agree with subgroup estimates even when confounding or selection bias affects both. Calibration does not verify exchangeability, consistency, or the outcome and propensity models.
  • Reading noisy group estimates as exact truth. A point away from the diagonal may reflect sampling error. Show uncertainty and consider the precision of each subgroup estimate.
  • Checking overlap only overall. Support may be poor exactly where predicted effects are largest. Examine propensity distributions and treatment counts within prediction-relevant regions.
  • Calling a small fitted propensity a causal effect. A propensity score describes treatment assignment given covariates, not the treatment effect. Its role here is to diagnose support and potential instability.
  • Assuming a good calibration slope settles the plot. An overall line can hide nonlinear discrepancies. Inspect grouped comparisons across the prediction range.
  • Ignoring a changed target after restricting support. Excluding poorly supported units may improve credibility for the retained population, but the resulting average concerns that population rather than the original target.
Key takeaway: Calibration compares predicted CATE magnitudes with estimated effects in prediction-defined groups; overlap checks whether both treatment conditions are represented where those predictions are made. Interpret both with uncertainty and identification assumptions in view.

Check Your Understanding

Answer each question using the definitions and reasoning above.

  1. What does mean calibration assert about the average true effect among units with the same predicted effect?
  2. In a randomized evaluation subgroup, the treatment mean is 8.6 points and the control mean is 7.1 points. What is the estimated subgroup effect, and what comparison would appear on a calibration plot?
  3. Why is an unadjusted treatment-control difference not generally sufficient for a subgroup effect in observational data?
  4. A subgroup’s fitted propensity scores are concentrated near 0. What does this imply about treated-unit support and the inverse-probability factor?
  5. Why can an overall calibration slope near 1 fail to reveal important miscalibration?
Need help applying this method to your own data?

Our team supports biostatistics, statistical programming and pharmacometric analyses. Send us a request and we will reply by email.

Request an analysis
Clinical Biostats