Tutorials › Causal Inference › Validation of Conditional Effect Learners

Causal Learning and Policy Decisions · Tutorial 960 of 972

Validation of Conditional Effect Learners

Use independent held-out data to compare conditional-effect predictions, assess their calibration, and examine the value of rules they imply.

Advanced 10 min read

What You'll Learn

  • Define a held-out doubly robust score for conditional treatment effects
  • Compare learners with squared error against the score and interpret the comparison
  • Assess calibration using averages within prediction groups
  • Estimate and compare treatment-rule values using held-out data
  • Recognize the assumptions, uncertainty, and limits of each diagnostic

Why Validate a Conditional-Effect Learner?

A conditional-effect learner estimates how treatment effects vary with observed characteristics. A model can predict outcomes accurately overall yet estimate those differences poorly: strong prediction of the outcome level does not ensure good prediction of the treatment effect. Validation should therefore target causal-effect predictions and the decisions they may inform, rather than rely only on ordinary outcome prediction error.

This tutorial develops three complementary checks on data not used to fit the candidate learners: comparison against doubly robust effect scores, calibration of predicted effects, and estimated value of treatment rules derived from the predictions. These checks answer different questions. None, by itself, proves that a learner has recovered the true individual effects or that a treatment rule will work in every target population.

Let \(X\) denote observed covariates, \(A\in\{0,1\}\) treatment, and \(Y\) the observed outcome. Write \(Y(1)\) and \(Y(0)\) for the potential outcomes under treatment and control, and define the conditional average treatment effect as \(\tau(x)=E[Y(1)-Y(0)\mid X=x]\). For the causal interpretations below, assume consistency, conditional exchangeability \((Y(1),Y(0))\perp A\mid X\), and positivity \(0<e(x)<1\), where \(e(x)=P(A=1\mid X=x)\). Under these conditions, the conditional effect is identified by the difference in conditional outcome means.

Definition: A held-out validation sample consists of observations not used to fit the candidate effect learners or the nuisance models used to score them. For a held-out individual \(i\), a doubly robust effect score is $$ \hat\Gamma_i = \hat m_1(X_i)-\hat m_0(X_i) +\frac{A_i}{\hat e(X_i)}\{Y_i-\hat m_1(X_i)\} -\frac{1-A_i}{1-\hat e(X_i)}\{Y_i-\hat m_0(X_i)\}, $$ where \(\hat m_a(x)\) estimates \(E[Y\mid A=a,X=x]\), and \(\hat e(x)\) estimates \(e(x)\). The score is conditionally unbiased for \(\tau(X_i)\) if the propensity model is correct, or if both outcome models are correct, subject to the identification assumptions and adequate overlap.

The nuisance estimates should be obtained without using the held-out outcomes being scored. They can be fitted on a separate training sample or through cross-fitting, in which each observation is scored using nuisance estimates trained without that observation. The learner being evaluated must also be fitted without the validation observations. These separations help prevent overly optimistic comparisons caused by evaluating a model on data it has already adapted to.

The phrase “doubly robust” does not mean that every estimate is reliable when nuisance models are poor, nor does it remove the need for positivity. Extreme estimated propensities can produce large residual corrections and unstable scores. The overlap diagnostics and calibration concepts introduced in the earlier material are useful context: validation results should be interpreted in the covariate region represented by the data and supported by treatment comparisons.

Compare Learners Against Held-Out Effect Scores

Suppose candidate learner \(k\) predicts \(\hat\tau_k(X_i)\) for each held-out individual. A direct score-based comparison is the mean squared error against the doubly robust score:

$$ \widehat R_k = \frac{1}{n_{\mathrm{val}}} \sum_{i=1}^{n_{\mathrm{val}}} \{\hat\tau_k(X_i)-\hat\Gamma_i\}^2. $$

This is not ordinary prediction error for \(Y\). It compares predicted treatment effects with an observed-data score whose conditional mean targets the treatment effect. Under the conditions above, the score can be noisy even when the nuisance estimates are good. Consequently, \(\widehat R_k\) is not an estimate of the learner’s pure squared error relative to the unobserved \(\tau(X_i)\). But, under conditional score unbiasedness, the extra score noise has the same expected contribution for competing learners evaluated on the same observations.

Worked Example: Comparing Two Effect Learners

Suppose four hypothetical held-out records produce doubly robust scores \(6,2,4,8\) outcome units. Learner A predicts \(5,3,4,6\); learner B predicts \(4,4,5,7\). Assume the scores were constructed using nuisance estimates fitted without these four records, and assume the causal identification conditions and the required nuisance-model condition are plausible in the represented covariate region.

For learner A, the squared score errors are \((5-6)^2=1\), \((3-2)^2=1\), \((4-4)^2=0\), and \((6-8)^2=4\). Thus

$$ \widehat R_A=\frac{1+1+0+4}{4}=1.500 $$

squared outcome units. For learner B, the squared errors are \((4-6)^2=4\), \((4-2)^2=4\), \((5-4)^2=1\), and \((7-8)^2=1\), giving

$$ \widehat R_B=\frac{4+4+1+1}{4}=2.500 $$

squared outcome units. On this held-out sample, learner A has a lower mean squared error against the doubly robust scores by \(2.500-1.500=1.000\) squared outcome units. This is evidence favoring A on this comparison, not proof that its true CATE error is lower in every population or that the difference is statistically distinguishable from sampling variation.

The reason a shared score-noise component can cancel in a comparison follows from a short calculation. If \(E[\hat\Gamma\mid X]=\tau(X)\), write \(\hat\Gamma=\tau(X)+\epsilon\), where \(E[\epsilon\mid X]=0\). For a prediction \(f(X)\),

$$ E[(f(X)-\hat\Gamma)^2\mid X] = (f(X)-\tau(X))^2+E[\epsilon^2\mid X]. $$

The cross-product vanishes because \(E[\epsilon\mid X]=0\). The second term does not depend on the candidate learner. Therefore, when two fixed learners are evaluated on the same validation distribution with the same score, the difference in their expected score risks equals the difference in their conditional-effect squared-error risks. This justification depends on score unbiasedness and does not make the observed finite-sample difference free of uncertainty.

Calibration: Do Predictions Match Average Effects?

A learner is calibrated when individuals assigned a given predicted effect have, on average, an effect equal to that prediction. Since individual treatment effects are not observed, calibration can be assessed using doubly robust scores as noisy proxies. A population calibration condition is

$$ E[\tau(X)\mid \hat\tau(X)]=\hat\tau(X). $$

When the held-out score satisfies \(E[\hat\Gamma\mid X]=\tau(X)\), it follows that \(E[\hat\Gamma\mid\hat\tau(X)]=E[\tau(X)\mid\hat\tau(X)]\). Thus, comparing average scores with average predictions within groups of similar predictions provides a practical calibration check. This check assesses average agreement at the chosen grouping resolution; it does not establish that predictions are correct for each individual.

Worked Example: Checking Calibration in Prediction Groups

A hypothetical validation set is divided into two groups by a learner’s predicted effect. In the low-prediction group, the four predictions are all \(1\) outcome unit and the four doubly robust scores are \(0,2,1,1\). In the high-prediction group, the four predictions are all \(3\) outcome units and the scores are \(2,4,3,3\). Suppose the scores were formed with held-out observations and the identification and score conditions are credible.

For the low-prediction group, the mean prediction is \(1\), while the mean score is \((0+2+1+1)/4=1\) outcome unit. For the high-prediction group, the mean prediction is \(3\), and the mean score is \((2+4+3+3)/4=3\) outcome units. The grouped averages agree in both groups, so this sample shows no grouped calibration discrepancy.

That finding does not prove perfect calibration. With only four observations per group, score noise may hide a real discrepancy, and grouping can conceal overprediction in one part of a group and underprediction in another. A calibration plot should therefore include group sizes and uncertainty, and its visual pattern should be read alongside overlap and score stability.

Calibration and discrimination are distinct. A learner that predicts one constant equal to the overall average effect can be calibrated overall while failing to distinguish people with meaningfully different effects. Conversely, a learner may rank individuals well but systematically overstate effect magnitudes. Report calibration together with a score-based risk comparison and the distribution of predictions; do not treat one as a substitute for the others.

Evaluate the Value of Rules Implied by Learners

A conditional-effect prediction may be used to construct a treatment rule \(d(X)\in\{0,1\}\), which assigns treatment according to observed characteristics. For validation, compare candidate rules using held-out outcomes and nuisance estimates, rather than assuming that smaller effect-score error automatically means greater practical value. Define \(m_a(x)=E[Y\mid A=a,X=x]\), \(p_1(x)=e(x)\), and \(p_0(x)=1-e(x)\). A doubly robust contribution for the rule’s value is

$$ \hat\phi_i(d) = \hat m_{d(X_i)}(X_i) + \frac{\mathbb{1}\{A_i=d(X_i)\}}{\hat p_{d(X_i)}(X_i)} \{Y_i-\hat m_{d(X_i)}(X_i)\}, \qquad \hat V(d)=\frac{1}{n_{\mathrm{val}}}\sum_{i=1}^{n_{\mathrm{val}}}\hat\phi_i(d). $$

Here \(\mathbb{1}\{\cdot\}\) is one when the condition holds and zero otherwise. Under the same causal identification conditions, and with a correct propensity model or correct outcome models, the expectation of the contribution targets the mean outcome if the rule were applied to the validation population. This calculation is a diagnostic estimate of value, not a guarantee of future performance. It is sensitive to overlap, nuisance estimation, the target population, and sampling uncertainty.

Worked Example: Comparing Two Rules With Held-Out Scores

Consider four hypothetical held-out individuals. The table gives observed treatment \(A\), outcome \(Y\), estimated propensity \(\hat e\), outcome predictions \(\hat m_0,\hat m_1\), and the treatment assigned by rules P and Q. These values are illustrative, and the nuisance predictions are assumed to have been fitted independently of the four held-out outcomes.

Individual\(A\)\(Y\)\(\hat e\)\(\hat m_0\)\(\hat m_1\)Rule PRule Q
1050.55500
2140.55510
3050.55500
4080.56601

For individual 1, both rules choose the observed control treatment, and the residual \(Y-\hat m_0=0\); each contribution is \(5\). For individual 2, Rule P assigns the observed treatment, so its contribution is \(5+(1/0.5)(4-5)=3\). Rule Q assigns control, which was not observed, so its residual correction is zero and its contribution is \(\hat m_0=5\). For individual 3, both contributions are \(5\), because the observed control residual is zero.

For individual 4, Rule P assigns the observed control treatment. Its contribution is \(6+(1/0.5)(8-6)=10\). Rule Q assigns treatment, but the observed treatment is control. Its indicator \(\mathbb{1}\{A=d(X)\}\) is therefore zero, so its contribution is \(\hat m_1=6\), with no residual correction. Averaging gives

$$ \hat V(P)=\frac{5+3+5+10}{4}=5.750, \qquad \hat V(Q)=\frac{5+5+5+6}{4}=5.250. $$

The estimated held-out value difference is \(5.750-5.250=0.500\) outcome units in favor of Rule P. This small hypothetical sample is far too limited to establish a reliable advantage; the calculation illustrates the estimator and the importance of applying the treatment-match indicator correctly.

For a fair comparison, evaluate both rules on the same held-out individuals and examine the individual differences \(\hat\phi_i(P)-\hat\phi_i(Q)\). Their sample mean is the estimated value difference; their variability indicates how uncertain that comparison may be. If rules were repeatedly tuned using the same validation outcomes, the reported values can be optimistic. A dedicated evaluation sample or a design that accounts for adaptation is needed for a confirmatory assessment.

What the Diagnostics Establish—and What They Do Not

DiagnosticMain questionImportant limitation
Squared error against held-out doubly robust scoresWhich learner better predicts the score on this validation distribution?Scores are noisy; the ranking is uncertain and depends on score validity.
Calibration by prediction groupsDo average scores agree with average predicted effects in these groups?Grouping can hide within-group errors and weakly populated groups are imprecise.
Held-out rule valueWhich specified rule has the larger estimated mean outcome in this population?Value estimates depend on overlap, nuisance quality, target population, and evaluation design.

None of these checks verifies unmeasured confounding control, consistency, or the relevance of the validation population to a future deployment setting. Those are substantive identification and generalization questions. Nor does a favorable diagnostic establish that the estimated heterogeneity is real rather than a product of noise or extensive searching; that issue requires careful accounting for how candidate learners and comparisons were selected.

Common Mistakes

  • Using ordinary outcome prediction error as a proxy for CATE accuracy. Predicting \(Y\) well can reflect accurate baseline outcome prediction while missing treatment-effect variation. Compare effect predictions with a suitable held-out causal score and report what assumptions support that score.
  • Calling the score-based loss the true CATE mean squared error. The score contains irreducible noise. Under conditional unbiasedness, shared noise cancels in expected comparisons, but an individual loss value still includes that noise.
  • Claiming that grouped calibration proves individual accuracy. Agreement of group means supports only average calibration at the chosen resolution. It cannot establish correct effects for each member of a group.
  • Applying a residual correction when the rule’s treatment was not observed. The value-score residual is multiplied by \(\mathbb{1}\{A=d(X)\}\). If the observed treatment differs from the rule’s assignment, the correction is zero; use the model-predicted outcome for the assigned treatment.
  • Ignoring validation reuse and uncertainty. Repeatedly choosing models or rules based on the same validation outcomes can make their apparent performance too favorable. Keep evaluation data separate from fitting and selection, and quantify uncertainty for the comparison.
Key takeaway: Validate conditional-effect learners with complementary held-out diagnostics: score-based comparisons address relative effect-prediction performance, calibration checks average agreement, and rule-value estimates assess implications for decisions. Interpret each only under its identification, nuisance-estimation, overlap, and validation-design conditions.

Check Your Understanding

Use the assumptions and distinctions in this tutorial to answer the following questions.

  1. Why can squared error against a doubly robust score compare two CATE learners even though the score is not the true conditional effect?
  2. State the two alternative nuisance-model correctness conditions that make the doubly robust effect score conditionally unbiased.
  3. A learner’s predictions and mean doubly robust scores agree in each of three prediction groups. What does this establish, and what can it fail to reveal?
  4. For a policy-value contribution, what happens to the residual correction when the observed treatment does not match the rule’s assigned treatment?
  5. Why should two candidate rules be evaluated on the same held-out individuals, and why does that not alone guarantee a reliable value comparison?
Need help applying this method to your own data?

Our team supports biostatistics, statistical programming and pharmacometric analyses. Send us a request and we will reply by email.

Request an analysis
Clinical Biostats