Double Robustness Is Not Protection Against Every Error

What Double Robustness Does—and Does Not—Promise

The double-robustness property of augmented inverse-probability weighting (AIPW) is useful but conditional: under the assumptions that identify a causal effect, the estimator can remain consistent if either the propensity-score model or the outcome-regression model is correctly specified. It does not make the effect identifiable when some treatment groups are absent, repair incorrect uncertainty calculations for dependent data, change the estimand to the one a researcher intended, or guarantee consistency when both nuisance models are wrong.

This tutorial examines those boundaries for the average treatment effect (ATE). The earlier material on the double-robustness property of AIPW establishes why correctness of either nuisance model can suffice. Here the central task is to identify what must already be true for that result to apply.

Definition: Let \(A\in\{0,1\}\) be treatment, \(X\) measured pre-treatment covariates, and \(Y\) the observed outcome. Let \(Y(1)\) and \(Y(0)\) be the potential outcomes under treatment and control. The ATE is \(\tau=E[Y(1)-Y(0)]\). Define the true propensity score \(e_0(X)=P(A=1\mid X)\) and the true outcome regressions \(\mu_a^0(X)=E[Y\mid A=a,X]\), for \(a\in\{0,1\}\).

For the usual ATE interpretation from observational data, assume consistency (the observed outcome is \(Y=Y(A)\)), no interference between units, conditional exchangeability \( (Y(1),Y(0))\perp A\mid X\), and positivity \(0<e_0(X)<1\) for covariate values in the target population. These assumptions have distinct roles: consistency and no interference define the treatment and outcome relation; exchangeability addresses confounding given \(X\); positivity requires both treatment options to be represented at each relevant \(X\). For standard independent-observation inference, also assume the sampled units are independent and drawn from the population defining the estimand.

With fitted nuisance functions \(\hat e(X)\) and \(\hat\mu_a(X)\), the AIPW estimator is

\hat\tau_{\mathrm{AIPW}} = \frac{1}{n}\sum_{i=1}^n \left[ \hat\mu_1(X_i)-\hat\mu_0(X_i) +\frac{A_i}{\hat e(X_i)}\{Y_i-\hat\mu_1(X_i)\} -\frac{1-A_i}{1-\hat e(X_i)}\{Y_i-\hat\mu_0(X_i)\} \right].

The two nuisance components are the propensity score and the treatment-specific outcome regressions. Double robustness concerns their correctness in estimating the target population’s functions. It does not remove the identification assumptions above, nor does it guarantee that a particular finite-sample estimate or its standard error is reliable.

Why Positivity Is Not a Nuisance-Model Error

If \(e_0(X)=0\) for a covariate pattern, no one with that pattern receives treatment. The treated outcome mean for that pattern is not learned from the observed data. If \(e_0(X)=1\), the control outcome mean is not learned. This is a failure of identification for the full-population ATE, not merely a propensity-model fitting problem. An outcome model can produce a number in an unsupported group, but that number is an extrapolation rather than information supplied by observed treatment comparisons. AIPW cannot turn the extrapolation into identification.

Worked Example: A Covariate Group With No Treated Units

Suppose a hypothetical population has two equally common covariate groups, \(X=0\) and \(X=1\). In group \(X=0\), both treatment groups are observed, and their mean outcomes are \(E[Y\mid A=1,X=0]=6\) and \(E[Y\mid A=0,X=0]=4\). In group \(X=1\), nobody is treated, and the observed control mean is 5. The treatment effect in group \(X=0\) is therefore \(6-4=2\).

For \(X=1\), the treated mean is unobserved. If its potential treated mean were 5, the group effect would be \(5-5=0\), and the population ATE would be \((2+0)/2=1\). If instead that mean were 9, the group effect would be \(9-5=4\), and the ATE would be \((2+4)/2=3\). Both values of the missing mean are compatible with the observed data, so the ATE is not identified by these data.

Here \(e_0(1)=0\). The inverse-probability term for treated units in this group has no observations to contribute, and the treated outcome regression cannot be checked against observed treated outcomes there. A fitted regression may return a prediction, but the two possible ATEs show why double robustness does not resolve the missing information.

Positivity should be considered relative to the target population: a subgroup outside the support of one treatment arm cannot contribute an empirically identified treatment contrast to the full-population ATE. In finite samples, very small estimated scores may also cause instability, but that is a separate issue from structural nonpositivity. Do not interpret a model’s ability to return fitted probabilities as evidence that the data contain the comparisons required for identification.

Dependence Can Break Inference Without Breaking the Formula

The ordinary AIPW standard error based on independent influence-function contributions relies on independent observations, or on an appropriate sampling model that justifies that independence. If units are clustered, repeated, or otherwise dependent, treating all observations as independent can understate uncertainty. Double robustness is a statement about consistency under nuisance-model conditions; it is not a correction for a variance estimator that ignores dependence.

Dependence can also threaten causal identification. For example, if one person’s treatment changes another person’s outcome, the no-interference condition fails and the individual potential outcomes \(Y(1)\) and \(Y(0)\) may not adequately represent the intervention. Clustered sampling with no interference is a different situation: the causal estimand might still be meaningful, while standard errors must account for the sampling dependence.

Worked Example: Cluster Dependence and an Understated Standard Error

Consider a hypothetical randomized study with four clinics, two assigned to treatment and two to control, and two patients observed per clinic. Suppose the outcome model has a clinic-level random component with variance 9 and an individual-level component with variance 1. The two patients within a clinic share the clinic component. The marginal variance of one patient’s outcome is \(9+1=10\), and the variance of a clinic mean is \(9+1/2=9.5\).

The treatment effect estimate is the difference between the mean of the two treated-clinic means and the mean of the two control-clinic means. Under independent clinics, its variance is \(9.5/2+9.5/2=9.5\), giving standard error \(\sqrt{9.5}=3.082\) outcome units.

If the eight patients are incorrectly treated as independent, there are four patients per arm. Using the marginal variance 10 gives the naive variance \(10/4+10/4=5\), or standard error \(\sqrt{5}=2.236\) outcome units. The estimate may still target the randomized treatment contrast, but the naive standard error is too small in this example because it fails to account for shared clinic variation. Choosing AIPW rather than a simple difference in means would not, by itself, fix that dependence error.

The diagnostic question is not just whether a fitted propensity score and outcome model look adequate. It is also whether the independent unit in the analysis matches the design and dependence structure. When treatment is assigned by clinic, for example, the number of independently assigned clinics—not simply the number of patients—matters for uncertainty. The appropriate variance method depends on the sampling and assignment design.

The Estimand Must Match the Question

AIPW is not a single estimand independent of its formula and target distribution. The displayed expression estimates an ATE averaged over the distribution of \(X\) in the sampled target population. An effect among treated people, an effect in a restricted subgroup, and an effect in a population with a different covariate distribution are different targets. Correct nuisance models do not make these estimands interchangeable.

Worked Example: The ATE and the Effect Among Treated People Differ

Suppose a hypothetical population has equal proportions with \(X=0\) and \(X=1\). The conditional treatment effects are 1 outcome unit for \(X=0\) and 5 units for \(X=1\). The ATE is the equally weighted average:

\tau = \tfrac{1}{2}(1)+\tfrac{1}{2}(5)=3 \text{ outcome units}.

Now suppose treatment probabilities are 0.2 for \(X=0\) and 0.8 for \(X=1\). In a population of 100 people, 10 of the 50 people with \(X=0\) and 40 of the 50 people with \(X=1\) are treated. Thus 20% of treated people have \(X=0\), and 80% have \(X=1\). The average treatment effect among treated people is

\tau_{\mathrm{ATT}} = 0.2(1)+0.8(5)=4.2 \text{ outcome units}.

Both values are consistent with the specified conditional effects; they average those effects over different populations. An AIPW procedure set up for the ATE does not answer the ATT question merely because its propensity and outcome models are correct. The target population and estimand must be stated before fitting or interpreting the estimator.

Other estimand mismatches can arise when the analysis silently changes the target population, restricts the sample after examining treatment patterns, or uses an outcome or treatment definition that differs from the scientific question. State explicitly whether the target is the full eligible population, the treated population, or a defined subgroup, and specify the outcome scale and follow-up period. The formula and the interpretation must refer to the same target.

When Both Nuisance Models Are Wrong

The double-robustness guarantee has two alternatives: the propensity score is correct, or the outcome regressions are correct. It does not say that two incorrect models “average out” to a correct estimate. The following calculation makes the limitation explicit. Let \(e^*(X)\) and \(\mu_a^*(X)\) be fixed candidate functions, with \(0<e^*(X)<1\). Under the identification assumptions, the expected AIPW score minus the ATE is

E\left[ \{e^*(X)-e_0(X)\} \left\{ \frac{\mu_1^*(X)-\mu_1^0(X)}{e^*(X)} + \frac{\mu_0^*(X)-\mu_0^0(X)}{1-e^*(X)} \right\} \right].

This expression is a bias formula for the score using those candidate functions. It is zero if \(e^*=e_0\), or if both outcome regressions are correct. If both components are misspecified, the product need not be zero. Particular errors can cancel, but such cancellation is not a general guarantee and should not be treated as a design strategy.

A Short Derivation of the Bias Formula

Condition on \(X\), write \(e_0=e_0(X)\), and let \(\mu_a^0=\mu_a^0(X)\). By consistency and conditional exchangeability, the conditional mean of \(Y\) among those with \(A=a\) is \(\mu_a^0\). For the treatment residual term, its conditional expectation is \(e_0(\mu_1^0-\mu_1^*)/e^*\); for the control residual term, including its minus sign, it is \(-(1-e_0)(\mu_0^0-\mu_0^*)/(1-e^*)\). Adding the regression contrast \(\mu_1^*-\mu_0^*\), then subtracting the true contrast \(\mu_1^0-\mu_0^0\), gives

\left(1-\frac{e_0}{e^*}\right)(\mu_1^*-\mu_1^0) + \left(\frac{1-e_0}{1-e^*}-1\right)(\mu_0^*-\mu_0^0) = (e^*-e_0) \left\{ \frac{\mu_1^*-\mu_1^0}{e^*} + \frac{\mu_0^*-\mu_0^0}{1-e^*} \right\}.

Taking expectation over \(X\) yields the stated expression. The derivation also shows why correct specification of either component removes the bias under the identifying assumptions, and why neither component alone can guarantee that result when both are wrong.

Worked Example: Both Nuisance Models Misspecified

Consider a hypothetical population with no variation in \(X\), true treatment probability \(e_0=0.4\), and true outcome means \(\mu_1^0=10\) and \(\mu_0^0=4\). The true ATE is \(10-4=6\) outcome units. Suppose the candidate propensity is \(e^*=0.5\), and the candidate outcome means are \(\mu_1^*=12\) and \(\mu_0^*=3\). Both nuisance components are wrong, while the true score is positive and below one.

The bias formula gives

(e^*-e_0) \left\{ \frac{\mu_1^*-\mu_1^0}{e^*} + \frac{\mu_0^*-\mu_0^0}{1-e^*} \right\} = (0.5-0.4) \left\{ \frac{12-10}{0.5}+\frac{3-4}{0.5} \right\} =0.1(4-2)=0.2.

The expected AIPW score is therefore \(6+0.2=6.2\) outcome units, not the true ATE of 6. This example illustrates a nonzero bias when both models are wrong. Other numerical errors could cancel, but that would be accidental and does not restore the general double-robustness guarantee.

Comparison, Diagnostics, and Practical Limits

Outcome regression (OR) estimates the treatment-specific outcome means and averages their contrast over the target covariate distribution. Inverse-probability weighting (IPW) reweights observed outcomes using the propensity score. AIPW combines these ingredients. The approaches differ in what model misspecification threatens their consistency, but none overrides identification or a poorly defined target.

Approach	Model-based protection	Important limitation
Outcome regression	Consistent when the relevant outcome regressions are correctly specified, under the identification assumptions.	Wrong outcome regressions can bias the result; unsupported groups require extrapolation.
IPW	Consistent when the propensity score is correctly specified, under the identification assumptions.	Wrong scores can bias the result; positivity and the target population still matter.
AIPW	Can be consistent if the propensity score or the outcome regressions are correctly specified, under the identification assumptions.	Does not guarantee consistency if both are wrong, or repair violations of identification, estimand choice, or dependence-aware inference.

A useful review therefore checks several distinct issues rather than asking only whether the estimator is “doubly robust”:

Identification: Are consistency, no interference, conditional exchangeability, and positivity plausible for the target population and measured covariates?
Support: Are there covariate groups with only one treatment level? A fitted score cannot establish support where treatment comparisons are absent.
Dependence: Do repeated observations, clusters, or the assignment design require a variance method that accounts for dependence?
Estimand: Does the formula average over the population of interest, and is that population the one named in the conclusion?
Nuisance models: Is there a defensible reason to regard at least one of the propensity or outcome components as adequately specified? Double robustness is not evidence that either model is correct.

These checks clarify the scope of the method. AIPW can reduce reliance on getting both nuisance models exactly right, but it cannot verify the substantive assumptions that connect observed outcomes to counterfactual outcomes. Nor can an estimator’s robustness property substitute for transparent reporting of the target, design, diagnostics, and uncertainty method.

Common Mistake

Common mistake: “The outcome model may be wrong and the propensity model may be wrong, but AIPW is still safe because it is doubly robust.” Correction: The standard guarantee is conditional on the causal identification assumptions and requires at least one nuisance component to be correctly specified. If both are wrong, the bias expression need not vanish. Even with a correct component, positivity, a correctly stated estimand, and valid inference for the dependence structure remain necessary.

Check Your Understanding

Answer each question, then compare with the solutions.

If no members of a covariate subgroup receive treatment, which assumption fails, and can a correct outcome model make the subgroup’s treated mean empirically identified?
In a clustered study, why might an AIPW point estimate and an independence-based standard error have different validity?
A researcher wants the effect among treated people but reports an estimator targeting the population ATE. Does double robustness resolve this discrepancy?
Suppose the true propensity is \(0.6\), the candidate propensity is \(0.5\), and both candidate outcome means equal their true means. What does the bias formula imply?

Solutions. (1) Positivity fails because the treatment probability is zero in that subgroup. A correct model cannot make the missing treated outcome mean empirically identified; any prediction there relies on extrapolation. (2) Dependence can leave the point estimate meaningful for its target while making an independence-based variance calculation wrong. The standard error must reflect the actual dependence and design. (3) No. ATE and ATT average effects over different populations, so the target must be changed to answer the stated question. (4) Both outcome-model errors are zero, so the bracketed term in the bias formula is zero and the score has zero bias under the identification assumptions, despite the incorrect candidate propensity.