Tutorials › Causal Inference › Doubly Robust Off-Policy Evaluation with Observational Data

Causal Learning and Policy Decisions · Tutorial 968 of 972

Doubly Robust Off-Policy Evaluation with Observational Data

See how outcome predictions and propensity weights combine to evaluate a treatment rule from observational data, and work through the assumptions and calculations.

Advanced 10 min read

What You'll Learn

  • Define policy value using potential outcomes and state the assumptions needed to identify it from observational data
  • Construct the augmented inverse-probability score for a fixed treatment rule
  • Derive why the estimator is consistent when either the propensity model or the outcome model is correct
  • Calculate a candidate policy’s estimated value from observed outcomes and fitted nuisance models
  • Compare estimated values for two policies and recognize the limits of the comparison

Why Evaluate a Policy With Observational Data?

A treatment policy assigns a treatment using a person’s observed characteristics. To decide whether a candidate policy is useful, we need to estimate its expected outcome, including outcomes that were not observed for each person. In randomized data, known treatment probabilities can help with this task. In observational data, treatment assignment may depend on measured characteristics, so evaluation also requires adjustment for that assignment process.

The augmented inverse-probability estimator combines two models: a propensity model for the probability of treatment, and outcome models for expected outcomes under each treatment. Its key property is called double robustness: under the identification assumptions below, it can consistently estimate the value of a fixed policy if either the propensity model or the outcome models are correct, subject to suitable estimation and regularity conditions. This is not a guarantee against unmeasured confounding or poor overlap.

Policy Value and Identification

Let \(X\) denote measured baseline characteristics, let \(A\in\{0,1\}\) be the treatment received, and let \(Y\) be the observed outcome, measured so that larger values are preferred. Write \(Y(1)\) and \(Y(0)\) for the potential outcomes under treatment and control. A fixed policy \(d\) maps characteristics to an action: \(d(X)\in\{0,1\}\). Its value is the mean outcome that would result if everyone in the target population followed that policy.

Definition: The value of policy \(d\) is \(V(d)=E[Y(d(X))]\). For observational data to identify this value from the observed distribution, assume consistency (the observed outcome is \(Y=Y(A)\)), no interference between individuals, conditional exchangeability \((Y(0),Y(1))\perp A\mid X\), and positivity: each action the policy may assign has positive probability at the relevant covariate values. A common, stronger positivity condition is that \(0<e(X)<1\) throughout the target population, where \(e(x)=P(A=1\mid X=x)\).

Consistency and no interference connect each person’s observed treatment and outcome to that person’s potential outcomes. Conditional exchangeability says that, after conditioning on \(X\), treatment assignment carries no additional information about the potential outcomes. Positivity ensures that outcomes under the policy’s chosen action are represented in the data rather than requiring unsupported extrapolation.

Define the propensity for action \(a\) as \(e_a(x)=P(A=a\mid X=x)\), and define the outcome regression \(m_a(x)=E[Y\mid A=a,X=x]\). Under the assumptions above, \(m_a(x)=E[Y(a)\mid X=x]\). Therefore,

$$ V(d)=E[m_{d(X)}(X)]. $$

This expression identifies the policy value, but estimating it using only fitted outcome predictions can be sensitive to outcome-model error. The augmented estimator adds a weighted residual correction, using observed outcomes for individuals who received the action prescribed by the policy.

The Augmented Inverse-Probability Estimator

Let \(\hat e_a(x)\) be an estimated probability of receiving action \(a\), and let \(\hat m_a(x)\) be an estimated mean outcome under that action. For a fixed policy, the estimated value is

$$ \widehat V_{\mathrm{AIPW}}(d) = \frac{1}{n}\sum_{i=1}^{n} \left[ \hat m_{d(X_i)}(X_i) + \frac{\mathbb{1}\{A_i=d(X_i)\}}{\hat e_{d(X_i)}(X_i)} \left\{Y_i-\hat m_{A_i}(X_i)\right\} \right]. $$

Here \(\mathbb{1}\{\cdot\}\) is 1 when its condition is true and 0 otherwise. The first term predicts each individual’s outcome under the policy’s action. The second term corrects that prediction using the observed residual \(Y_i-\hat m_{A_i}(X_i)\), but only when the person actually received the action the policy would assign. The residual is weighted by the inverse estimated probability of receiving that action.

For binary treatment, \(\hat e_1(x)=\hat e(x)\) and \(\hat e_0(x)=1-\hat e(x)\). Thus the denominator is the estimated probability of the policy-selected action, not always the probability of treatment. An equivalent expression sums over both actions, selecting the term for which \(a=d(X_i)\).

Why the Estimator Is Doubly Robust

The robustness claim concerns the population expectation of the score, and then consistency of its sample average as the fitted nuisance models approach their limits. It is not a claim that an arbitrary finite-sample estimate is unbiased, nor that a model can be badly misspecified in every respect. The policy must be fixed independently of the evaluation observations, or the evaluation procedure must otherwise account for policy selection. Nuisance estimation must also satisfy conditions that make estimation error negligible; simply fitting highly flexible models on the same observations does not automatically ensure this.

Result: Under consistency, no interference, conditional exchangeability, and positivity for the policy’s actions, the AIPW score has expectation \(V(d)\) if either (i) the outcome regressions for the policy-selected actions are correct or (ii) the propensity model is correct. With suitable convergence and regularity conditions, this yields a consistent estimator if either nuisance component is consistently estimated, even when the other is misspecified.

To see the first part, condition on \(X=x\) and let \(a=d(x)\). If the outcome model is correct, then \(E[Y-m_a(x)\mid A=a,X=x]=0\). Consequently the expected residual correction is zero for any valid, finite weight, and the expected score is \(m_a(x)=E[Y(a)\mid X=x]\).

For the second part, suppose instead that the propensity model is correct but the outcome prediction \(\hat m_a(x)\) may be wrong. Conditional on \(X=x\), the probability of receiving action \(a\) is \(e_a(x)\), and the mean observed residual among those receiving it is \(m_a(x)-\hat m_a(x)\). Therefore the conditional expected score is

$$ \hat m_a(x) + \frac{e_a(x)}{e_a(x)} \left\{m_a(x)-\hat m_a(x)\right\} = m_a(x). $$

Averaging over \(X\) gives \(V(d)\). This algebra explains why the residual correction can remove outcome-model error when the assignment probabilities are right. It does not make either model optional in practice: both can improve precision, and poor overlap can make the correction unstable.

Worked Example: Seeing Both Robustness Cases

Suppose that in one covariate group the true probability of treatment is \(e(x)=0.25\), and the true treated mean outcome is \(m_1(x)=12\) outcome points. Consider a policy that treats everyone in this group. First suppose the fitted treated outcome mean is \(\hat m_1(x)=10\), but the propensity is correct. The conditional expected score is

$$ 10+\frac{0.25}{0.25}(12-10)=10+2=12. $$

The outcome prediction is wrong by 2 points, but the correct propensity weighting repairs that error in expectation. Now suppose instead that \(\hat m_1(x)=12\) is correct, while the fitted treatment probability is wrong and equals \(0.50\). Then the conditional expected score is

$$ 12+\frac{0.25}{0.50}(12-12)=12+0=12. $$

The residual has conditional mean zero when the outcome model is correct, so the wrong propensity does not shift the expected score in this calculation. These are population-expectation calculations, not promises that a small sample will produce exactly 12 points.

Worked Observational Evaluation

The following hypothetical data illustrate evaluation of a fixed rule. The candidate policy treats individuals classified as high-risk and does not treat individuals classified as low-risk. The fitted outcome predictions and treatment probabilities are assumed to come from models specified before evaluating this policy. The outcome is measured in points.

PersonRisk group\(A\)\(Y\)\(\hat e(X)\)\(\hat m_0(X)\)\(\hat m_1(X)\)Policy action
1High1120.608101
2High070.508101
3High190.408101
4Low050.25690
5Low1110.50690
6Low070.20690

Worked Example: Calculating the Candidate Policy’s Value

For a treated person, the estimated probability of the selected action is \(\hat e(X)\); for an untreated person, it is \(1-\hat e(X)\). Apply the AIPW score row by row. Person 1 received the policy’s action, so the score is

$$ 10+\frac{12-10}{0.60}=13.333. $$

Person 2 did not receive the policy’s treated action, so the correction is zero and the score is \(10\). Person 3 received treatment, giving \(10+(9-10)/0.40=7.500\). Person 4 received the policy’s control action, giving \(6+(5-6)/0.75=4.667\). Person 5 received treatment rather than the policy’s control action, so the score is \(6\). Person 6 received control, giving \(6+(7-6)/0.80=7.250\).

The estimated policy value is the average of these six scores:

$$ \widehat V_{\mathrm{AIPW}}(d) = \frac{13.333+10+7.500+4.667+6+7.250}{6} = \frac{48.750}{6} = 8.125\text{ outcome points}. $$

This is an estimated mean outcome under the candidate rule for the target population represented by these data, given the identification assumptions and fitted models. It is not the observed mean outcome among only the people who happened to follow the rule: some individuals’ policy-selected actions were not observed, and their contributions use the model predictions.

Comparing Policies and Checking the Estimate

Policy evaluation is often comparative. For a second fixed policy, calculate its value with the same evaluation observations and nuisance predictions, then subtract the estimates. Because both estimates use the same sample, the difference is an estimate of a value contrast; the point estimates alone do not establish that one policy is better with statistical certainty.

Worked Example: Comparing the Rule With Treating No One

Use the same hypothetical data to evaluate the policy that assigns control to everyone. The score for person 1 is \(8\), since that person received treatment rather than the policy’s control action. Person 2 received control, so the score is

$$ 8+\frac{7-8}{1-0.50}=6. $$

Person 3’s score is \(8\). Person 4’s score is \(6+(5-6)/0.75=4.667\). Person 5’s score is \(6\), and person 6’s score is \(6+(7-6)/0.80=7.250\). Therefore,

$$ \widehat V_{\mathrm{AIPW}}(\text{treat none}) = \frac{8+6+8+4.667+6+7.250}{6} = \frac{39.917}{6} = 6.653\text{ outcome points}. $$

The estimated value difference is \(8.125-6.653=1.472\) outcome points. Equivalently, the candidate rule’s score differs from the treat-none score only for the three high-risk people: their score differences are \(5.333\), \(4\), and \(-0.500\). Their sum is \(8.833\), and \(8.833/6=1.472\). The positive estimated contrast favors the candidate rule in this hypothetical calculation, but a decision would also require valid uncertainty assessment and consideration of costs or other outcomes.

Practical Limits and Common Mistakes

The weighting step makes overlap especially important. If the policy selects an action with a very small estimated probability for some individuals, the corresponding residual can receive a very large weight. A few such scores can dominate the average and make it unstable. Check the distribution of estimated probabilities for the actions the policy selects, and be cautious about evaluating a rule in regions with little or no observed support.

  • Mistake: “Doubly robust means confounding does not matter.” The robustness result still requires conditional exchangeability given measured \(X\). Unmeasured confounding can invalidate the causal interpretation.
  • Mistake: “Either model can be arbitrary without consequences.” Double robustness is a consistency result when one nuisance component is correct and regularity conditions hold. Extreme weights, finite-sample noise, or estimation problems can still harm performance.
  • Mistake: using the treatment probability for every policy action. The denominator must be the probability of the action the policy selects: \(\hat e(X)\) for treatment and \(1-\hat e(X)\) for control.
  • Mistake: treating a positive estimated contrast as proof of superiority. A point estimate has sampling uncertainty, and evaluating a policy chosen using the same data can create additional bias. The value calculation does not by itself settle either issue.

The practical strength of the AIPW estimator is that it combines outcome prediction with a correction grounded in the observed assignment process. Its causal interpretation, however, comes from the identification assumptions—not from the estimator’s name. A credible evaluation therefore reports the target population, policy, outcome units, overlap diagnostics, and assumptions alongside the estimated value.

Check Your Understanding

Use the definitions, assumptions, and calculations above to answer these questions.

  1. State the conditional exchangeability and positivity assumptions used to identify the value of a policy from observational data.
  2. For a policy that assigns control, what estimated probability belongs in the AIPW denominator when \(\hat e(X)=0.30\)?
  3. In the robustness derivation, why does a correct propensity model recover the outcome regression when the fitted outcome model is wrong?
  4. Explain why a policy-selected action with a very small estimated probability can make the AIPW estimate unstable.
  5. The candidate policy’s estimated value exceeds the treat-none estimate in the worked example. What can and cannot be concluded from that point-estimate comparison alone?
Need help applying this method to your own data?

Our team supports biostatistics, statistical programming and pharmacometric analyses. Send us a request and we will reply by email.

Request an analysis
Clinical Biostats