What Does It Mean to Evaluate a Learned Policy?
A treatment policy assigns an action to each person based on their observed characteristics. A policy might recommend treatment for people predicted to benefit, subject to cost or capacity constraints. After learning such a rule, the central question is not whether its predictions look persuasive, but what outcome would result if the rule were used in the target population? This is the policy’s value.
Evaluation requires care because each person reveals an outcome under only the treatment they actually received. The outcome under the other treatment is counterfactual. The methods here use treatment assignment probabilities and outcome models to estimate the average outcome that would arise under the policy. To assess performance without optimism from policy selection, the policy should be evaluated on observations that did not help choose or tune it.
The value is an average outcome in the units of \(Y\), such as dollars, symptom-free days, or a score. A policy’s value is not itself a treatment effect: it describes the expected outcome under that policy. To compare two policies, one may estimate their value difference, \(V(\pi_1)-V(\pi_0)\), using the same evaluation population.
Identification and Evaluation Scores
For observational data, identification requires assumptions connecting observed treatment groups to the counterfactual outcomes. Write \(Y(d)\) for the potential outcome under action \(d\), and let \(e(X)=P(D=1\mid X)\). The assumptions are consistency, conditional exchangeability, and positivity. Consistency means the observed outcome equals the potential outcome for the treatment received. Conditional exchangeability means \(\{Y(1),Y(0)\}\) is independent of \(D\) given \(X\). Positivity means \(0<e(X)<1\) for covariate values in the target population. In a randomized trial, treatment probabilities are determined by its randomization design; exchangeability follows from randomization, subject to appropriate handling of the design.
These assumptions do not make the unobserved counterfactuals visible. Instead, they allow the policy value to be represented by expectations of functions of observed data. An evaluation sample should consist of independent, identically distributed observations from the target population, or have a sampling and dependence structure accounted for in the variance calculation. Define \(m_d(X)=E[Y\mid D=d,X]\) and \(e_1(X)=e(X)\), \(e_0(X)=1-e(X)\). The inverse-probability score and the doubly robust score for a policy are:
Here \(\mathbf{1}\{\cdot\}\) is one when its condition is true and zero otherwise. Both scores require the policy and nuisance predictions to be specified without using the evaluation observation’s outcome to select the policy. The inverse-probability score weights an outcome by the inverse probability of receiving the action the policy would assign. The doubly robust score starts with the predicted outcome under the policy’s action, then adds a weighted residual correction for observations who received that action.
The doubly robust result follows by conditioning on \(X\). If the outcome regressions are correct, then within either treatment arm the conditional mean of \(Y-m_D(X)\) is zero, so the correction has conditional expectation zero and the first term has expectation \(E[Y\{\pi(X)\}\mid X]\). If the propensity is correct, the correction’s conditional expectation is \(E[Y\{\pi(X)\}-m_{\pi(X)}(X)\mid X]\), which added to the first term again gives the conditional policy outcome. Averaging over \(X\) gives the stated value.
Worked Example: Inverse-Probability Evaluation
Worked Example: Inverse-Probability Evaluation
Suppose an independently held-out evaluation sample has five people. The policy, propensity scores, treatments, and outcomes are shown below. Assume for this illustration that the listed treatment probabilities are known and satisfy positivity.
| Person | \(\pi(X)\) | \(e(X)\) | \(D\) | \(Y\) |
|---|---|---|---|---|
| 1 | 1 | 0.50 | 1 | 12 |
| 2 | 0 | 0.25 | 0 | 5 |
| 3 | 1 | 0.80 | 0 | 4 |
| 4 | 0 | 0.40 | 1 | 9 |
| 5 | 1 | 0.60 | 1 | 8 |
For each person, the policy score is zero if the observed treatment does not match the policy action. Otherwise, use the outcome divided by the probability of receiving the assigned action. The five scores are \(12/0.50=24\), \(5/(1-0.25)=6.667\), \(0\), \(0\), and \(8/0.60=13.333\).
Thus the estimated value is
The estimate is an average outcome of 8.800 units under the policy, on this illustrative scale. The two zero scores do not mean those people would have had zero outcomes under the policy. Their observed treatment did not reveal the outcome under the action the policy would assign; inverse-probability weighting accounts for this through the other observations. With such a small sample, this estimate is highly variable and is not a reliable precision claim.
Doubly Robust Estimation in Practice
For evaluation observation \(i\), calculate the doubly robust score using the policy action, the predicted outcome for that action, the observed outcome, and the appropriate treatment probability. Then average the scores. The outcome predictions and propensity estimates should be obtained without using that observation to fit the corresponding nuisance model. Cross-fitting can help accomplish this by producing out-of-fold nuisance predictions. It does not, by itself, protect against selecting the policy using the same evaluation outcomes.
Worked Example: Calculating Doubly Robust Scores
Consider four held-out observations with the following hypothetical values. The columns \(m_1\) and \(m_0\) are outcome-model predictions, and \(e\) is the estimated probability of treatment.
| Person | \(\pi(X)\) | \(e(X)\) | \(m_1(X)\) | \(m_0(X)\) | \(D\) | \(Y\) |
|---|---|---|---|---|---|---|
| 1 | 1 | 0.50 | 10 | 6 | 1 | 12 |
| 2 | 0 | 0.40 | 8 | 5 | 0 | 4 |
| 3 | 1 | 0.25 | 7 | 4 | 0 | 6 |
| 4 | 0 | 0.70 | 9 | 3 | 0 | 5 |
For person 1, the policy selects treatment and the person received it, so the score is \(10+(12-10)/0.50=14\). Person 2’s policy selects control, which was received; the score is \(5+(4-5)/(1-0.40)=3.333\). Person 3’s policy selects treatment, but the person received control, so the residual correction is zero and the score is \(7\). Person 4’s policy selects control, which was received; the score is \(3+(5-3)/(1-0.70)=9.667\).
Averaging these values gives
This estimate is 8.500 outcome units. The example demonstrates the calculation, not evidence that either nuisance model is correct. In real analyses, very small estimated probabilities can make the residual correction unstable; report and inspect the propensity-score distribution and the implied weights rather than treating the estimator’s name as a safeguard against poor overlap.
Quantifying Uncertainty
Let \(\widehat\psi_i\) be the selected policy-value score for evaluation observation \(i\), either inverse-probability or doubly robust. The sample estimate is \(\widehat V=n^{-1}\sum_{i=1}^n\widehat\psi_i\). With independent evaluation observations and regularity conditions that yield an approximately normal sampling distribution, estimate its standard error by the sample standard deviation of the scores divided by \(\sqrt n\). A large-sample 95% interval is \(\widehat V\pm1.96\,SE(\widehat V)\). Its repeated-sampling interpretation concerns the value of the policy being evaluated; it is not a 95% probability statement about a fixed parameter after observing the interval.
Worked Example: An Illustrative Standard Error
Use the four doubly robust scores from the previous example: \(14,\ 3.333,\ 7,\ 9.667\), with mean \(8.500\). Their squared deviations from the mean sum to approximately \(30.250+26.694+2.250+1.361=60.555\). Therefore, the sample variance, standard deviation, and standard error are
Applying the large-sample formula mechanically gives \(8.500\pm1.96(2.247)\), or approximately \((4.096,\ 12.904)\) outcome units. This is only an illustration of the calculation: four observations are far too few for the large-sample approximation to be persuasive. In a substantive analysis, use a sufficiently large independent evaluation sample and an uncertainty procedure appropriate to the sampling design. If comparing two policies on the same observations, form each observation’s difference between policy scores and estimate uncertainty from those paired differences, rather than treating the two value estimates as independent.
Adaptive Evaluation: Where Optimism Enters
A policy may be selected from many candidates, tuned to maximize estimated value, or modified after inspecting evaluation results. If those same outcomes then supply the reported value estimate, the estimate is generally optimistic: random noise helped select the winner and is counted again as evidence of its performance. A conventional standard error computed after that selection usually does not capture all the uncertainty created by the search. The issue is not limited to a large number of policies; repeated informal changes in response to validation results can create the same problem.
Use training data to learn and tune the policy, and reserve an independent sample for a prespecified evaluation. Do not alter the policy after examining that sample’s outcomes and still describe it as an untouched test.
Fit propensity and outcome models on separate folds or external training data, then produce predictions for held-out observations. This addresses overfitting in nuisance estimation, not policy selection on the evaluation set.
Record how many policies were compared, which outcomes were inspected, whether thresholds or constraints changed, and whether results were repeatedly checked. Repeated use of a nominal test set makes it part of the adaptive process.
For a confirmatory estimate, evaluate the finalized policy on new data. If fresh data are unavailable, use a method designed for the actual selection procedure and clearly describe the resulting inferential target and limitations.
When the policy is trained on a separate sample and then fixed, an evaluation interval typically quantifies uncertainty in its value conditional on that learned policy. It does not automatically include variation from drawing a different training sample and learning a different policy. If the goal is to characterize the full learning procedure, that additional variation must be incorporated through a design that repeats or otherwise accounts for policy learning, not merely by adding a standard error to a single held-out estimate.
Common Pitfall
Wrong reasoning: “I used cross-fitting, so my policy estimate is unbiased even though I chose the best policy after comparing all candidates on the same data.” Cross-fitting can reduce overfitting in nuisance predictions, but it does not undo selection of a policy based on evaluation outcomes. Correction: protect the evaluation sample from policy choice and tuning, or use an inference strategy that explicitly addresses the adaptive selection. Also inspect overlap: a nominally doubly robust estimate can still be unstable when the policy often chooses actions that were very unlikely for the people receiving them.
Check Your Understanding
Answer these questions using the definitions and procedures above.
- In words, what does \(V(\pi)\) measure, and why is it not generally the same quantity as an average treatment effect?
- State the positivity assumption in terms of \(e(X)\). What practical problem can arise when the policy selects treatment for covariate profiles with very small \(e(X)\)?
- For a person whose policy-assigned action differs from the observed treatment, what part of the doubly robust score remains?
- Why does cross-fitting nuisance models not, on its own, correct for choosing a policy using the evaluation outcomes?
- What does a standard error computed on a held-out sample quantify when the learned policy is treated as fixed?