Simulation Workshop on Nuisance-Model Misspecification

Why Simulate Nuisance-Model Misspecification?

AIPW and TMLE combine estimates of the treatment propensity and the conditional outcome mean. Both methods can be consistent when either nuisance model is correct, subject to identification, positivity, and appropriate estimation conditions. A simulation makes the scope of that protection concrete: it lets us compare repeated-sample behavior when both models are correct, when just one is wrong, and when both are wrong. It also shows why robustness of bias does not by itself guarantee a narrow confidence interval or good finite-sample performance.

The central design principle is to specify a data-generating process whose true treatment effect is known, then deliberately fit nuisance models with controlled degrees of misspecification. Repeating this process across independent data sets allows bias, spread, and interval coverage to be assessed against that known target.

A Hypothetical Data-Generating Process

Consider a continuous outcome \(Y\), binary treatment \(A\), and two baseline covariates \(X_1\) and \(X_2\), each independently taking values \(-1\) and \(1\) with probability one-half. Generate treatment according to

where \(\operatorname{expit}(z)=1/(1+\exp(-z))\). The four possible linear predictors, in the order \((-1,-1),(-1,1),(1,-1),(1,1)\), are \(-0.8,-1.0,1.4,-0.4\). Thus the true propensity scores range from approximately \(\operatorname{expit}(-1)=0.269\) to \(\operatorname{expit}(1.4)=0.802\). All four covariate patterns have a nonzero probability of receiving either treatment; there is no deterministic treatment assignment in this finite covariate support.

Generate the outcome from

with the error independent of treatment and covariates. The conditional outcome mean under treatment \(a\) is \(m_0(a,X)=1+0.5X_1-0.25X_2+a(2+0.6X_1X_2)\). The individual treatment effect conditional on \(X\) is \(2+0.6X_1X_2\). Since the covariates are independent and balanced, \(E(X_1X_2)=0\), so the true average treatment effect is \(\psi_0=2\) outcome units. This known value lets us measure estimation error directly.

What “Single” and “Double” Misspecification Mean

In each simulated data set, estimate the propensity \(e(X)=P(A=1\mid X)\) and the two outcome means \(m(a,X)=E(Y\mid A=a,X)\). The correctly specified propensity model includes \(X_1\), \(X_2\), and \(X_1X_2\) in a logistic regression. A deliberately misspecified version omits \(X_1X_2\). The correctly specified outcome model includes the covariates and the treatment-by-product term \(A X_1X_2\); a misspecified version omits that term. These choices create four scenarios: both fits correct, only the propensity fit misspecified, only the outcome fit misspecified, and both fits misspecified.

The omitted interaction matters: the true propensity changes with \(X_1X_2\), and the true treatment effect changes with \(X_1X_2\). A model that omits an interaction is not automatically wrong in every application; here it is wrong because the data-generating process explicitly includes a nonzero interaction. Correctness is defined relative to that process, not by whether a model seems familiar or simple.

Why Double Robustness Can Fail When Both Fits Are Wrong

The key bias structure can be seen by conditioning on \(X\). Let \(e_0(X)\) and \(m_0(a,X)\) be the true nuisance functions, define \(\delta_a(X)=\tilde m(a,X)-m_0(a,X)\), and consider the conditional expectation of the AIPW summand minus the true conditional treatment effect. Directly taking expectations over treatment and outcome gives

For example, the treated residual contributes \(-e_0\delta_1/\tilde e\), while the plug-in treated mean contributes \(\delta_1\). Together they give \((\tilde e-e_0)\delta_1/\tilde e\). The control terms give \((\tilde e-e_0)\delta_0/(1-\tilde e)\). Adding them yields the displayed expression. If the propensity is correct, its first factor is zero. If both outcome means are correct, both \(\delta\) terms are zero. If both nuisance models are misspecified, neither condition is assured, so bias can remain.

This identity is a population statement for fixed nuisance functions, not a promise that every finite-sample estimate will be unbiased when just one model is correct. Estimation error, extreme fitted propensities, and sampling variability still matter. TMLE uses a targeted update rather than the AIPW residual correction, but its protection also depends on suitable nuisance estimation and regularity conditions. It is therefore informative to simulate both methods rather than assume identical finite-sample results.

Targeting and a Finite-Sample TMLE Calculation

For a continuous outcome and identity-link fluctuation, one TMLE construction starts with fitted outcome means and updates the observed-treatment prediction along a clever covariate. Define \(H_i=A_i/\tilde e(X_i)-(1-A_i)/\{1-\tilde e(X_i)\}\). Set \(\tilde Q_\epsilon(A_i,X_i)=\tilde m(A_i,X_i)+\epsilon H_i\), where the scalar \(\epsilon\) is chosen to solve the empirical targeting equation \(\sum_i H_i\{Y_i-\tilde Q_\epsilon(A_i,X_i)\}=0\). The updated plug-in estimate averages \(\tilde Q_\epsilon(1,X_i)-\tilde Q_\epsilon(0,X_i)\). This is a continuous-outcome targeting illustration; implementations can vary in their fluctuation specification.

Designing and Reading a Simulation

Choose a sample size, such as \(n=500\), and a number of independent replications, such as \(R=1000\). In each replication, generate covariates, treatment, and outcomes from the specified process; fit nuisance models according to the scenario; calculate both estimators and their standard errors; and form nominal 95% Wald intervals. For each method and scenario, summarize bias \(\frac{1}{R}\sum_{r=1}^R(\hat\psi_r-\psi_0)\), empirical standard deviation across \(\hat\psi_r\), root mean squared error, and coverage, the proportion of intervals containing \(\psi_0\).

The following is a hypothetical illustrative output, not a result from a reported clinical study. It describes what a single 1,000-replication run might look like under this design; small differences between methods or scenarios should not be treated as universal rankings.

The table illustrates the expected pattern, not a theorem about exact finite-sample results. In the singly misspecified scenarios, average bias is small relative to the empirical spread, while simultaneous misspecification produces substantial bias and poor coverage. For the both-misspecified AIPW row, the approximate RMSE from the displayed bias and SD is \[ \sqrt{0.286^2+0.242^2}\approx0.375 \] outcome units. Its 0.704 coverage means that 704 of the 1,000 intervals in this hypothetical run contain the true effect of 2.00. It does not mean there is a 70.4% probability that the fixed parameter is in any particular interval.

With \(R=1000\), the Monte Carlo standard error of a coverage estimate near 0.95 is approximately \[ \sqrt{\frac{0.95(1-0.95)}{1000}}\approx0.0069. \] Consequently, coverage values such as 0.948 and 0.951 are indistinguishable at this simulation precision. The Monte Carlo standard error of an estimated bias is approximately the empirical SD divided by \(\sqrt R\); for an SD near 0.24, this is \(0.24/\sqrt{1000}\approx0.0076\) outcome units. More replications or repeated simulation runs are useful when comparing small differences.

Diagnostics, Comparisons, and Limitations

Before interpreting simulation results, verify the generated propensity probabilities and fitted probabilities. Check that treatment is represented in every covariate pattern, and record how often fitted propensities approach zero or one. Such extremes can inflate inverse-probability weights and variance even when the true assignment mechanism has overlap. Also check that the intended model is actually correct or misspecified under the chosen data-generating process; a coding error in the interaction term can invalidate the comparison.

AIPW and TMLE share a doubly robust motivation, but their finite-sample estimates and variance estimates need not agree. AIPW adds residual corrections directly to a plug-in contrast. TMLE modifies the outcome predictions through a targeted fluctuation and then uses the updated predictions for its plug-in contrast. Either can be more variable in a particular scenario; neither is guaranteed to dominate in bias, variance, or coverage in every finite sample. A correctly specified propensity model paired with a misspecified outcome model is not the same as having both models correct, and the reverse is also true.

Simulation conclusions depend on the chosen sample size, covariate distribution, effect heterogeneity, nuisance estimators, and interval method. A simulation with a few discrete covariate patterns cannot establish performance for every high-dimensional or data-adaptive setting. Nor does a well-behaved result under one process rule out unmeasured confounding in an applied analysis: the simulation assumes its specified treatment and outcome mechanisms. Its purpose is to expose behavior under transparent conditions, not to certify an estimator universally.