Tutorials › Causal Inference › Sample Splitting for Adaptive Policy Evaluation

Causal Learning and Policy Decisions · Tutorial 969 of 972

Sample Splitting for Adaptive Policy Evaluation

Separate policy selection from policy evaluation to reduce adaptive optimism and quantify the value of a learned treatment rule.

Advanced 12 min read

What You'll Learn

  • Explain why selecting the best-looking policy on its training data biases its reported value upward.
  • Define policy value and the evaluation target conditional on a training sample.
  • Use an independent holdout and doubly robust scores to estimate policy value.
  • Calculate a standard error and interval from held-out evaluation scores.
  • Distinguish cross-fitting nuisance models from cross-fitting policy selection.
  • Identify assumptions and limits of uncertainty statements after adaptive selection.

Why Policy Evaluation Needs Separation

A treatment policy is often chosen after comparing many candidate rules: assign treatment to everyone, treat only certain subgroups, or use a rule learned from covariates. If the same observations both determine which rule looks best and estimate its value, chance variation can favor the selected rule twice. Its apparent value can therefore be too high, even when every candidate’s estimate is unbiased before selection.

Sample splitting addresses this problem by separating the data used to choose a policy from the data used to evaluate it. The key is not that the evaluation data are intrinsically better; it is that, conditional on the training data, the selected policy is fixed and the evaluation observations have not helped select it. Cross-fitting can also reduce overfitting in nuisance-model estimates, but it does not automatically make a policy selected on all the data independent of those data.

Definition: A treatment policy \(\pi\) maps baseline covariates \(X\) to an action, here \(\pi(X)\in\{0,1\}\). Its value is the expected outcome under the action the policy assigns: \(V(\pi)=E[Y(\pi(X))]\). A policy’s value is defined for a target population and an outcome scale; larger values are preferable when the outcome is beneficial.

The target in a split-sample analysis is usually the value of the particular policy selected using the training sample, evaluated in the target population. It is not automatically the value of the best possible policy, nor a guarantee that the selected rule will be optimal in a new population. Inference is commonly interpreted conditional on the training data: after selection, treat the learned rule as fixed and quantify uncertainty in its population value.

The Source of Adaptive Optimism

Suppose candidate policies \(\pi_1,\ldots,\pi_J\) have true values \(V_1,\ldots,V_J\), and the training data produce estimates \(\widehat V_j=V_j+\varepsilon_j\). Even if each estimation error has mean zero, selecting the policy with the largest estimate selects partly on favorable noise. For two candidates with equal true value and independent, identically distributed, mean-zero errors, the expected maximum error is positive unless the errors are always identical:

$$ E[\max(\varepsilon_1,\varepsilon_2)] = \frac{1}{2}E[|\varepsilon_1-\varepsilon_2|] \geq 0. $$

This identity follows from \(\max(a,b)=(a+b+|a-b|)/2\). Taking expectations and using \(E[\varepsilon_1]=E[\varepsilon_2]=0\) leaves the expression shown. With multiple candidates, the selected maximum can be still more optimistic. The problem is adaptive selection, not necessarily a biased estimator for any one policy considered in advance.

Worked Example: The Apparent Winner and an Independent Check

Suppose, for illustration, two candidate policies truly each yield an average of 10 outcome units in the target population. On a training sample, their estimated values are 10.8 and 9.7 units. The first policy is selected, and its apparent advantage over the second is \(10.8-9.7=1.1\) units. Because the winner was chosen for having the larger training estimate, that 10.8 is not an independent estimate of its value.

Now suppose the selected policy is evaluated on a separate randomized evaluation sample. Its six? No: consider four evaluation scores of 9, 10, 10, and 11 units. The estimated value is

$$ \widehat V_{\mathrm{eval}} = \frac{9+10+10+11}{4} = 10\text{ units}. $$

The training estimate of 10.8 and the holdout estimate of 10 answer different statistical questions: the first helped choose the policy, while the second estimates the selected policy’s value without having selected it. This constructed example assumes the true values are known only to make the selection effect transparent; in an actual evaluation, the population value is unknown.

Holdout Evaluation with Doubly Robust Scores

For observational data, causal interpretation requires assumptions. Let \(Y(1)\) and \(Y(0)\) denote potential outcomes, and let \(A\) be the observed treatment. Assume consistency, so \(Y=Y(A)\); conditional exchangeability, \((Y(0),Y(1))\perp A\mid X\); and positivity, so each action the policy might assign has positive probability at the covariate values in the target population. These assumptions allow observed data, with adjustment for \(X\), to identify the policy’s value. In a randomized trial, random assignment supplies exchangeability, and known assignment probabilities simplify evaluation.

Write \(e_a(x)=P(A=a\mid X=x)\) for the treatment propensity and \(m_a(x)=E[Y\mid A=a,X=x]\) for the outcome regression. On an evaluation observation, define the doubly robust score for policy \(\pi\) using estimates \(\widehat e_a\) and \(\widehat m_a\):

$$ \widehat\psi_i(\pi) = \widehat m_{\pi(X_i)}(X_i) + \frac{\mathbf{1}\{A_i=\pi(X_i)\}}{\widehat e_{\pi(X_i)}(X_i)} \left[Y_i-\widehat m_{\pi(X_i)}(X_i)\right]. $$

The first term predicts the outcome under the policy’s action. The second corrects that prediction using the observed outcome when the assigned action matches the policy. With suitable identification assumptions, the score has conditional expectation equal to the policy’s expected outcome if either the propensity model is correct or the outcome regression is correct. This protection does not remove the need for exchangeability, consistency, positivity, or appropriate estimation and inference.

A simple sample-splitting plan fits the policy and nuisance models using training data, then computes the scores only on independent evaluation observations. Conditional on the training data, these fitted objects are fixed. The evaluation estimate and its standard error are:

$$ \widehat V_{\mathrm{eval}}(\pi) = \frac{1}{n_{\mathrm{eval}}} \sum_{i\in\mathrm{eval}}\widehat\psi_i(\pi), \qquad \widehat{SE}\{\widehat V_{\mathrm{eval}}(\pi)\} = \frac{s_{\widehat\psi}}{\sqrt{n_{\mathrm{eval}}}}, $$

where \(s_{\widehat\psi}\) is the sample standard deviation of the evaluation scores. A large-sample interval is the estimate plus or minus an appropriate normal critical value times this standard error. A \(t\)-interval can be used under a normal-score model, especially as a small-sample illustration. In either case, the interval concerns the value of the selected policy, not the probability that the policy is optimal.

Worked Example: Computing a Holdout Value and Interval

Consider a hypothetical randomized evaluation sample of four people. The assignment probability for either action is \(0.5\), and the policy was fixed before these evaluation observations were used. For each person, the observed action happens to match the policy’s recommendation. Suppose the observed outcomes under the recommended actions are \(7\), \(5/3\), \(4\), and \(31/6\) units. Use the inverse-probability score, the doubly robust score with the prediction term set to zero:

$$ \widehat\psi_i = \frac{\mathbf{1}\{A_i=\pi(X_i)\}Y_i}{0.5}. $$

Because the match indicator is 1 in each row, the four scores are \(14\), \(10/3\), \(8\), and \(31/3\) units. Their mean is

$$ \widehat V_{\mathrm{eval}} = \frac{14+10/3+8+31/3}{4} = \frac{107}{12} = 8.9167\text{ units}. $$

The sum of squared deviations from this mean is \(8620/144\). Thus the sample variance and sample standard deviation are

$$ s_{\widehat\psi}^{\,2} = \frac{8620/144}{4-1} = 19.9537, \qquad s_{\widehat\psi} = \sqrt{19.9537} = 4.467\text{ units}. $$

The estimated standard error is \(4.467/\sqrt{4}=2.233\) units. Under an independent, identically distributed normal-score model, a 95% \(t\)-interval with \(df=3\) uses \(t_{0.975,3}=3.182\):

$$ 8.9167\pm 3.182(2.233) = [1.81,\ 16.02]\text{ units}. $$

Under the stated model, this procedure gives intervals that cover the fixed policy’s population value in 95% of repeated evaluation samples. It does not mean there is a 95% probability that this particular interval contains the value. Four observations provide very little information, and the interval is wide; the normal-score assumption should not be treated as established by these four values.

Cross-Fitting: What It Does and Does Not Do

Cross-fitting divides data into \(K\) folds. For each fold, nuisance models are fitted without that fold, and scores for observations in the fold are computed using those out-of-fold predictions. This reduces the risk that flexible nuisance models overfit the same observations on which their scores are evaluated. Under standard regularity conditions, suitable positivity and identification assumptions, and adequate nuisance-model performance, the average cross-fitted score can support asymptotically valid inference.

There is an important distinction. Cross-fitting nuisance models for a policy fixed independently of the evaluation sample does not mean that the policy itself may safely be selected using every evaluation observation. If each fold instead selects a policy from the other folds, the resulting average estimates the performance of the fold-specific learning procedure, on observations excluded from each corresponding fit. It is not automatically an estimate of the value of one final policy refitted on all observations. The dependence created by overlapping training folds also means that a generic standard error cannot be justified merely by calling the procedure cross-fitting.

Worked Example: Cross-Fitting Nuisance Predictions

Suppose a policy has already been selected on an independent training sample. A separate randomized evaluation sample is divided into two folds. The propensity is known to be \(0.5\); for simplicity, the out-of-fold outcome prediction for the policy-recommended action is 10 units in each case, and all six observed assignments match the policy. The score is therefore \(10+2(Y-10)\). Suppose the observed outcomes in the two evaluation folds are \(9.5, 10.5, 10\) and \(9, 11, 10\) units, respectively. The corresponding cross-fitted scores are \(9, 11, 10\) and \(8, 12, 10\) units.

The mean score across the six observations is

$$ \widehat V_{\mathrm{cf}} = \frac{9+11+10+8+12+10}{6} = 10\text{ units}. $$

The squared deviations from 10 sum to \(1+1+0+4+4+0=10\), so \(s_{\widehat\psi}=\sqrt{10/5}=1.414\) units and the score-based standard error is \(1.414/\sqrt{6}=0.5774\) units. A large-sample normal interval would be \(10\pm1.96(0.5774)=[8.868,11.132]\) units. These six observations are too few to make a normal approximation persuasive; the calculation illustrates the procedure, not a reliable small-sample guarantee. Cross-fitting protects the nuisance predictions from being trained on their own scored observations. The policy remains fixed because it was selected on independent training data.

Practical Design and Common Pitfalls

A defensible evaluation begins by specifying what is being evaluated: the target population, outcome and units, policy, and whether the aim is the value of one selected rule or the average performance of a learning algorithm. Then reserve observations for evaluation, or use a design that explicitly targets out-of-sample performance. Keep policy selection separate from those evaluation outcomes. If nuisance models are flexible, cross-fit them within the evaluation stage and calculate scores only from out-of-fold predictions.

A holdout estimate is not immune to all problems. A small holdout has high sampling variability; a policy may be poorly supported by the observed treatment probabilities; and confounding or measurement error can invalidate the causal interpretation. In observational data, near-zero propensities can make inverse-probability scores unstable even when positivity holds mathematically. Report the design, assumptions, score-based uncertainty, and target population so readers can judge the result in context.

  • Mistake: “The largest training value is an unbiased evaluation because each candidate estimator was unbiased.” Correction: selection by the largest noisy estimate generally makes the selected estimate optimistic. Evaluate the selected rule on data not used to choose it.
  • Mistake: “Cross-fitting the outcome model means the policy is independent of evaluation data.” Correction: cross-fitting nuisance predictions and separating policy selection are different safeguards. The policy itself must not use the holdout outcomes if the goal is an independent holdout evaluation.
  • Mistake: “A narrow standard error proves the policy is optimal.” Correction: a standard error describes uncertainty in an estimated value under the stated assumptions. It does not establish optimality or compare the policy with every alternative.
Key takeaway: Select the policy without using the observations reserved to evaluate its value. Use valid causal scores, account for nuisance estimation appropriately, and interpret uncertainty as uncertainty about the value of the specified policy—not as evidence that it is the best possible policy.

Check Your Understanding

Answer each question by distinguishing policy selection, value estimation, and uncertainty.

  1. Why can the maximum of several unbiased policy-value estimates be optimistic for the selected policy?
  2. What is the conditional target of a holdout evaluation when the policy was selected using the training sample?
  3. Name the causal assumptions needed to interpret observational-data policy scores as values under intervention.
  4. What does cross-fitting nuisance models protect against, and what does it not guarantee about policy selection?
  5. If a 95% interval for a policy’s value is wide, what does that say about precision, and what does it not say about the policy’s optimality?
Need help applying this method to your own data?

Our team supports biostatistics, statistical programming and pharmacometric analyses. Send us a request and we will reply by email.

Request an analysis
Clinical Biostats