Missing Outcomes In Operating-Characteristic Simulations

Why Missing Outcomes Belong in Calibration

A trial’s decision rule is applied to the data that will actually be available, not to an idealized data set in which every participant has an observed outcome. If a fixed Bayesian analysis uses only observed outcomes, missingness can reduce precision and, depending on its mechanism, can also change the apparent treatment effect. Calibration simulations should therefore represent missingness explicitly: generate each participant’s outcome under the trial scenario, generate whether that outcome is observed, and then apply the prespecified analysis to the observed data.

The key is to keep the analysis fixed while varying the data-generation conditions of interest. For each simulated trial, record whether the analysis produces the target decision, as well as quantities such as bias, interval coverage, or interval width when those are part of the calibration objective. A simulation that removes observations but does not specify how they became missing may not represent the intended trial.

Specify the Mechanism Before Simulating

Three standard categories describe how missingness relates to data. Missing completely at random (MCAR) means that observation status is independent of the outcome and the other trial variables. Missing at random (MAR) means that, conditional on observed variables included in the mechanism, observation status does not additionally depend on the unobserved outcome. Missing not at random (MNAR) means that observation status still depends on the outcome after conditioning on the observed variables. These are assumptions about the process that generated missingness, not labels that can generally be confirmed from the observed data alone.

A useful simulation represents the process in two stages. First generate the complete outcomes from the scenario’s outcome model. Then generate \(R_i\) using the chosen mechanism. For example, if \(X_i\) is a recorded baseline covariate, a MAR mechanism might use \(P(R_i=0\mid A_i,X_i,Y_i)=P(R_i=0\mid A_i,X_i)\). An MNAR mechanism could allow this probability to vary with \(Y_i\) even after conditioning on \(A_i\) and \(X_i\). Here \(A_i\) denotes assigned treatment.

Specify whether the target missingness rate is overall, within treatment arm, or conditional on a covariate or outcome. An overall rate alone does not determine the mechanism: identical overall rates can correspond to MCAR, MAR, or MNAR processes and can yield different observed data. The data-generation mechanism is known in a simulation, which makes it possible to compare the analysis with the scenario that generated the trial.

What Missingness Does to an Observed Outcome Rate

Under MCAR, the observed participants are, in expectation, representative with respect to the outcome. Missingness nevertheless reduces the number of observed outcomes and typically lowers precision. Under MAR, an observed-only analysis can also be distorted if the variables associated with missingness and outcome are not appropriately handled. Under MNAR, even conditioning on measured covariates may not remove the selection induced by the missingness process.

For a binary outcome, suppose the true event probability is \(p=P(Y=1)\), and the observation probabilities for participants with and without an event are \(r_1=P(R=1\mid Y=1)\) and \(r_0=P(R=1\mid Y=0)\). The observed event rate is obtained by conditioning on being observed:

When \(r_1=r_0\), this equals \(p\); when they differ, it generally does not. This derivation explains why a stated overall missingness percentage is insufficient to assess bias. The relationship between missingness and outcomes matters, as does whether the fixed analysis can use variables that explain the missingness.

Estimate the Operating Characteristic, Not a Single Trial’s Result

For a binary decision, let \(D_b=1\) if the fixed rule declares success in simulated replicate \(b\), and \(D_b=0\) otherwise. With \(B\) independent replicates, the estimated probability of success under the specified scenario is \(\hat{\theta}=B^{-1}\sum_{b=1}^{B}D_b\). Its estimated Monte Carlo standard error is \(\sqrt{\hat{\theta}(1-\hat{\theta})/B}\). This standard error quantifies simulation error in estimating the operating characteristic; it is not the trial-level standard error of a treatment-effect estimate.

A Bayesian decision rule might declare success when a posterior probability exceeds a prespecified cutoff. For example, the analysis may use independent \(\operatorname{Beta}(1,1)\) priors for the two event risks and declare treatment beneficial when \(P(p_T<p_C\mid\text{observed data})>0.975\). In every simulated replicate, use the same prior, likelihood, handling of missing outcomes, and cutoff. The resulting success proportion estimates the rule’s operating characteristic under the chosen outcome and missingness scenario.

Diagnostics, Alternatives, and Limitations

Check the simulated missingness rates by treatment arm and by any variables that define the mechanism. Under outcome-dependent mechanisms, also inspect rates by the fully generated outcome; this diagnostic is available to the simulator even though the outcome would be hidden when missing in an actual trial. Over many replicates, compare simulated rates with their prespecified targets. A mismatch can reveal a coding error, an incorrectly parameterized mechanism, or confusion between a conditional and marginal rate.

For observed trial data, differences in missingness across arms or covariate groups can be detected, but such checks do not establish MCAR or rule out MNAR: the outcomes that are missing cannot be directly compared with their unobserved values. In simulations, the mechanism is known, so the main diagnostic question is whether the code implements that mechanism and whether the fixed analysis receives only the information it is intended to use.

Complete-case analysis is a simple fixed alternative that uses only observed outcomes. It can be reasonable under suitable assumptions, including MCAR for many standard estimands, but it loses information and can be biased when selection changes the observed outcome distribution. Likelihood-based analyses and multiple-imputation approaches can use measured covariates under appropriate MAR assumptions and correctly specified models. Weighting approaches can also address selection under their own assumptions. For MNAR scenarios, results generally require explicit sensitivity assumptions; no observed-data procedure can learn the distribution of missing outcomes from the observed outcomes alone without assumptions.

In calibration, the purpose is not to select whichever analysis looks best after seeing a scenario. It is to measure the performance of the prespecified analysis over plausible missingness conditions. If the trial’s actual analysis includes covariates, imputation, or other handling of missing outcomes, the simulation should implement that same procedure. A complete-case analysis should not be substituted merely because it is easier to simulate.

• Common mistake: treating a missingness rate as a complete mechanism. A 20% missing rate could be MCAR, MAR, or MNAR. Specify what missingness depends on as well as how often it occurs.
• Common mistake: generating only observed outcomes. This makes an outcome-dependent missingness rule impossible to implement correctly. Generate the complete outcomes first, then generate observation indicators.
• Common mistake: interpreting one replicate as an operating characteristic. A single trial’s decision is one binary realization. Estimate the operating characteristic from repeated independent replicates and report Monte Carlo uncertainty.
• Common mistake: assuming equal missingness rates imply equal bias. The worked example shows that equal overall rates can conceal different outcome-dependent selection.