Simulation-Based Comparison Under Known Missingness Mechanisms

Why Simulate When the Missingness Mechanism Is Known?

With missing outcomes, observed data alone generally cannot establish which MNAR extrapolation is correct. A simulation offers a different kind of evidence: the analyst specifies a complete-data outcome model and a dropout process, generates data from them, and then knows the mechanism that produced each replicate. Competing analysis models can then be judged against that known truth. This does not prove which model is right for an actual study; it shows how the candidates behave under the simulated conditions.

The central design task is to specify both parts of the data generation. The outcome model determines the distribution of outcomes before dropout. The missingness model determines which outcomes are observed. Their combination creates the observed data on which candidate models must operate. A useful simulation checks not only which candidate ranks best by a stated criterion, but also how its treatment-effect estimates behave.

Specify the Generating Mechanisms First

Consider a simple trial with randomized treatment \(A_i\), where \(A_i=1\) denotes treatment and \(A_i=0\) control. Let \(Y_{i0}\) be a fully observed baseline outcome and \(Y_{i1}\) the follow-up outcome. Let \(R_i=1\) if the follow-up is observed and \(R_i=0\) otherwise. One possible outcome mechanism is

Here the coefficient of treatment is 2 outcome points at follow-up conditional on baseline. If treatment is randomized and baseline has the same distribution in both arms, the corresponding full-data marginal mean difference is also 2 points. A possible MNAR dropout mechanism is

Because \(\varepsilon_i=Y_{i1}-(10+2A_i+0.5Y_{i0})\), retention depends on the current outcome after accounting for treatment and baseline. This is MNAR: even conditional on the observed treatment and baseline, the probability that follow-up is observed depends on the follow-up value itself. The simulated data generator knows each \(\varepsilon_i\), including for participants whose outcome is subsequently hidden.

A simulation should vary features that matter to the question, rather than rely on one convenient setting. For example, scenarios might vary the strength of outcome-dependent dropout, the sample size, the outcome variance, or whether the true mechanism depends on the current outcome or only on observed history. The treatment effect and the outcome scale should remain clearly defined in every scenario. If a parameter is varied, state its values and why they represent relevant conditions.

Candidate analyses should also be explicit. They might include a selection model with current-outcome dependence, a model with dependence only on observed history, and a model that assumes missing at random. These are analysis assumptions, not descriptions of what the simulator knows. To fit a selection model when some outcomes are missing, its observed-data likelihood must account for those unobserved values. In the one-follow-up illustration, write \(f(y\mid A,Y_0)\) for the outcome density and \(\pi(y,A,Y_0)\) for the retention probability. The contribution to the likelihood is

For an observed outcome, both its outcome density and its retention probability contribute. For a missing outcome, its possible values are integrated out. This is why an MNAR selection-model fit requires assumptions about the unobserved outcomes, not merely a regression of the observed missingness indicator on available variables.

What to Measure Across Replicates

Before running the simulation, choose the performance measures and the model-ranking rule. For treatment-effect estimation, let \(\hat{\tau}_b\) be the estimate in replicate \(b\), and let \(\tau\) be the known generating effect. Across \(B\) replicates, the empirical bias, Monte Carlo standard deviation, and root mean squared error (RMSE) are

Bias describes average displacement from the target, while the standard deviation describes variation across generated data sets. RMSE combines both. If interval estimates are a goal, empirical coverage is the fraction of replicates whose reported interval contains \(\tau\); it should be considered alongside interval width. A ranking criterion such as AIC can be recorded separately. Apply the same prespecified rule to all candidates fitted to the same observed replicate, and interpret ranking frequencies as properties of that scenario and criterion, not as proof of truth.

Design, Run, and Check the Simulation

Diagnostics, Alternatives, and Limitations

A simulation can go wrong before any model comparison is interpreted. First check the generated data: verify treatment allocation, outcome means and variances, dropout rates by treatment and outcome range, and the direction and strength of the dependence specified in the mechanism. For example, if low residual outcomes were intended to have higher dropout, a table of generated retention rates should show that pattern. Compare empirical summaries across replicates with the values implied by the generator.

Next examine fitting behavior. Record nonconvergence, boundary estimates, implausibly large standard errors, and whether failures cluster in particular scenarios or candidates. Report how failures are handled. Silently dropping failed fits can bias ranking frequencies and performance summaries, especially if a model fails most often in difficult replicates. A candidate that is computationally unstable may be less useful in practice even if its successful fits look accurate.

MNAR parameters are often weakly identified from observed data. Even when the simulator knows the true dropout dependence, the fitted candidates do not see the hidden outcomes. A correctly specified candidate may therefore estimate the treatment effect imprecisely, and different MNAR candidates may yield similar observed-data fits but different extrapolations. This is not evidence that the simulation is defective by itself; it is a reason to report uncertainty and to examine multiple plausible generating scenarios.

Simulation complements, rather than replaces, other comparison approaches. Observed-data residual checks, posterior predictive checks, artificial-masking calibration, cross-validation, information criteria, and Bayesian model evidence assess different aspects of candidate fit or prediction using available data and their respective assumptions. A known-mechanism simulation adds controlled evaluation against a specified truth, including the error of treatment-effect estimates. It cannot establish which generating scenario applies in a real application, and its conclusions can change when the scenario changes.

A common mistake is to conclude, “The model that generated the simulated data won most often, so it is the correct model for the actual study.” The result supports only a narrower statement: under the particular generating parameters, sample size, fitting procedures, and ranking rule, that candidate ranked most often. A second mistake is to report only the winner frequency. The example shows why that is insufficient: a model can win the ranking often without being the least biased or having the lowest RMSE, and an estimate can have low bias but substantial variability.