Why Missingness Belongs in the Design Simulation
A MAMS design makes interim decisions using outcomes available by particular stages. If some outcomes are missing, the analysis may use fewer observations than planned, have different precision, or be impossible to calculate. The resulting effect is not merely a smaller final sample: missingness can change which arms continue, how often a boundary is crossed, and whether the planned analysis can be completed at all.
A simulation should therefore represent both the outcome that would be generated under the trial's outcome model and whether that outcome is observed. It should also represent the pattern of missingness. Independent patient-level dropout, a site-wide recording failure, and dropout related to a patient's unobserved outcome can have the same marginal missing percentage but very different consequences.
The target effect and the analysis rule must be specified before choosing a missingness mechanism. For example, a simulation might generate a treatment effect for all randomized participants, then use the planned complete-case analysis at each look. That evaluates how that analysis behaves under the assumed missingness; it does not establish that complete-case analysis estimates the all-randomized treatment effect without bias.
Specify the Missingness Mechanism
A mechanism is a rule for generating the availability indicators \(R\), possibly conditional on treatment, baseline characteristics, site, earlier outcomes, or the outcome that is missing. Write \(Y_{\mathrm{obs}}\) for observed data and \(Y_{\mathrm{mis}}\) for unobserved outcomes. Three standard categories help describe the assumptions:
- Missing completely at random (MCAR): missingness is independent of both observed and unobserved outcomes and covariates. For a simple independent outcome-level mechanism, \(P(R_{iaj}=1)=1-q\) for every participant.
- Missing at random (MAR): conditional on the observed data included in the mechanism, missingness does not additionally depend on the unobserved outcomes. For instance, dropout might depend on recorded baseline risk and an observed earlier outcome.
- Missing not at random (MNAR): even after conditioning on the observed data, missingness depends on an outcome that is itself unobserved. For example, participants with an unobserved poor outcome might be more likely to drop out.
These labels describe assumptions about the joint data-generating process, not properties that can usually be confirmed from observed trial data alone. In a design simulation, they are prespecified scenarios. MAR or MNAR does not by itself dictate an analysis: the simulated analysis must still match the method planned for the trial.
For repeated assessments, dropout may be monotone: after a participant drops out, later outcomes are unavailable. A simple discrete-time specification uses a dropout hazard \(q_{iaj}\), the probability of dropping out at assessment \(j\) conditional on remaining under observation up to that assessment. If dropout occurs, set the participant's later availability indicators to zero. Intermittent missingness is different: an assessment may be missing while a later assessment is observed. Simulate that pattern only if it is plausible and part of the prespecified scenario.
Worked Example: Independent MCAR Dropout and Approximate Power
Consider a hypothetical two-arm interim comparison with 100 participants randomized to each arm. The outcome is normally distributed with standard deviation 10 points in both arms, and the true treatment-minus-control mean difference is 5 points. Suppose each outcome is independently observed with probability 0.90, regardless of arm or outcome. At this look, use a one-sided \(Z\) test with known standard deviation and threshold 2.00.
The mechanism is MCAR by assumption: each participant has an independent Bernoulli availability indicator with \(P(R=1)=0.90\). The expected observed sample size in each arm is \(100(0.90)=90\). For a calculation using those expected sample sizes, the standard error is
The noncentrality parameter under the specified treatment effect is \(5/1.491\approx3.354\). With a one-sided critical value of 2.00, the approximate power at these sample sizes is
With all 100 outcomes observed per arm, the standard error would be \(10\sqrt{2/100}=1.414\) points and the corresponding power approximation would be \(\Phi(5/1.414-2.00)=\Phi(1.536)\approx0.938\). Thus, even this simple MCAR mechanism reduces the approximate power. These are fixed-sample approximations using expected observed counts, not the exact power under random observed sample sizes; a MAMS design should estimate its operating characteristics by simulating the complete stage-wise decision process.
Represent Dependence at the Correct Level
The level at which missingness is generated determines its dependence structure. Independent Bernoulli draws for each outcome represent independent measurement-level failures. They do not represent a shared event that makes multiple measurements unavailable together. For a site-wide failure, generate a site-level indicator and apply it to every measurement affected by that event. If participants can also have individual dropout, model that separately.
Worked Example: Shared Site-Wide Recording Failure
Suppose a hypothetical trial has five sites, with 20 participants per arm at each site. At each site, a shared recording failure occurs with probability 0.10 and, if it occurs, removes all outcomes in both arms at that site. Let \(G_s=1\) denote a failure at site \(s\). Generate one \(G_s\) per site and set \(R_{ias}=0\) for every affected participant at that site when \(G_s=1\).
For any one participant, the probability of a missing outcome is 0.10. But for two participants at the same site whose outcomes are both affected, the probability both are missing is also 0.10:
If instead independent Bernoulli draws with missing probability 0.10 were used for the two outcomes, the joint probability would be \(0.10(0.10)=0.01\). That implementation has the correct individual missingness rate but fails to reproduce the shared failure.
For either arm, the number observed is 20 times the number of sites without a failure. The expected observed count is
Writing \(F\) for the number of failed sites, \(F\sim\mathrm{Binomial}(5,0.10)\), and \(n_{\mathrm{obs}}=20(5-F)\). Therefore,
so the standard deviation is \(\sqrt{180}\approx13.42\) participants. The same failed sites affect both arms, so their observed counts also vary together. An implementation using independent participant-level failures would miss that feature and could misrepresent the variability of the treatment contrast and the chance that an analysis is unavailable.
The same principle applies to shared data-system outages, assessment-day disruptions, and other common causes: draw the event at the level where it occurs, then apply its consequences to all affected records. A separate individual-level mechanism can be layered on top, provided the combined rule is explicit. For example, a participant's outcome is observed only if the site has no recording failure and the participant has not dropped out.
Connect Missingness to the Interim Analysis
At each scheduled look, generate the outcomes and availability indicators that would exist by that time, then run the prespecified analysis on the available data. For a monotone dropout model, a participant who leaves before a later look contributes earlier observed outcomes but not later ones. For an outcome measured once at the end of a stage, that outcome cannot contribute before it is observed. The simulation should respect the trial's timing and data cut-off rules.
If an interim statistic is a treatment-control difference, both the estimates and their standard error must be calculated according to the planned analysis using observed outcomes. Do not substitute the complete-data sample size into the standard error after outcomes have been removed. Nor should unavailable outcomes be silently replaced with zero, the control mean, or another value unless that replacement is part of the prespecified analysis.
A simulation should define what happens if the available data do not support the planned calculation. For example, an analysis might require at least one observed outcome in each comparison group, or a variance estimate might require more than one observation per group. Record whether the look-specific analysis was completed. If the trial has a prespecified alternative when a calculation cannot be made, implement that rule; otherwise, do not invent one inside the simulation. Keep analysis completion distinct from a statistical decision such as stopping for lack of efficacy.
Worked Example: Outcome-Dependent Missingness and Complete-Case Bias
Suppose a hypothetical binary outcome is favorable with probability 0.60 in the treatment arm and 0.50 in control. In both arms, outcomes are observed with probability 0.90 when favorable and 0.60 when unfavorable. This is an MNAR mechanism because the probability of observation depends directly on the outcome that may be missing.
In the treatment arm, the probability of both a favorable outcome and an observed record is \(0.60(0.90)=0.54\). The total probability of an observed record is
Thus, the favorable proportion among observed treatment outcomes is
For control, the observed favorable probability is \(0.50(0.90)=0.45\), and the overall observation probability is \(0.50(0.90)+0.50(0.60)=0.75\). The observed favorable proportion is \(0.45/0.75=0.6000\). The complete-case difference is therefore about \(0.6923-0.6000=0.0923\), whereas the difference in the full randomized populations is \(0.60-0.50=0.1000\). The complete-case contrast is not the same quantity as the full-population contrast in this example. In other scenarios the direction or size of the discrepancy can differ; simulations should assess the specified mechanism rather than assume a universal direction of bias.
Why Selection Can Change an Estimate
A short identity explains why outcome-dependent observation matters. For an outcome \(Y\) with finite mean and an availability indicator \(R\), the observed-case mean is \(E(Y\mid R=1)\). Since \(E(YR)=P(R=1)E(Y\mid R=1)\), the covariance definition gives
Consequently, when \(P(R=1)>0\),
Under MCAR, \(Y\) and \(R\) are independent, so this covariance is zero and the observed-case mean has the same expectation as the full-data mean. Under outcome-dependent missingness, the covariance need not be zero. This identity concerns the mean in one group; it does not alone establish whether a treatment contrast is biased, since that also depends on the mechanisms in both groups and on the analysis.
Assess Operating Characteristics and Completion
For every simulated replicate, apply the same missingness rules at all arms, sites, participants, and stages as specified in the scenario. Then execute the actual decision logic using the data available at each look. Relevant summaries may include the probability of rejecting a null hypothesis, the probability of selecting an effective arm, expected sample size, and probability that each planned analysis is completed. These quantities should be reported by missingness scenario, alongside the assumed rates and dependencies.
The denominator matters. For example, an estimated rejection probability among all simulated trials answers a different question from a rejection proportion calculated only among trials whose analysis completed. Report the completion proportion as well as any conditional performance summary, and state the denominator. If an uncompleted analysis is assigned a decision by a prespecified trial rule, apply that rule consistently; otherwise, do not count it as an ordinary non-rejection without explanation.
Useful checks compare simulated missingness with its specification: overall rates by arm and stage, rates by site or baseline groups, persistence after dropout, and joint missingness for outcomes affected by shared events. For MNAR scenarios, check the relationship between missingness and the generated full outcome inside the simulation, even though the full outcome would not be available to the trial analyst. These checks verify the mechanism; they do not prove that it is realistic. Plausibility must be considered using design knowledge and sensitivity scenarios.
| Simulation choice | What it represents | Key consequence to assess |
|---|---|---|
| Independent outcome-level draws | Measurement failures independent across outcomes, conditional on specified factors | Reduced observed counts and precision without shared-event clustering |
| Shared site or system indicator | One event removes multiple outcomes together | Correlated availability and variable losses across sites |
| Complete-case analysis | Analysis uses records with required outcomes observed | May lose precision; can target a selected subset under non-MCAR mechanisms |
| Likelihood-based or imputation analysis | An analysis that uses a model for observed and/or missing data | Performance depends on model assumptions and whether they match the simulated mechanism |
Complete-case, likelihood-based, and imputation analyses are alternatives only when they are genuinely available and specified for the trial. A simulation must not use a more favorable analysis than the one intended merely because outcomes are missing. Conversely, a sensitivity analysis can compare prespecified analysis methods under the same data-generating scenarios. The simulation evaluates those methods under stated assumptions; it cannot establish that an unverifiable MAR or MNAR assumption is true.
Common Pitfalls and Limitations
- Using independent draws for a shared event. Independent draws reproduce independent failures, not a site-wide failure. Generate the shared event once at its proper level and apply it to all affected measurements.
- Applying dropout only at the final analysis. This can give the wrong data at interim looks. Generate availability according to when outcomes are measured and when dropout occurs.
- Treating missingness as an outcome value. Setting a missing binary outcome to zero, for example, changes the data and implicitly assigns an outcome rather than representing its absence.
- Using only an overall missing percentage. Equal percentages can conceal different arm imbalance, timing, clustering, or dependence on outcome. Specify these features, not just a marginal rate.
- Calling an analysis failure a design decision. If an estimate or test cannot be calculated, record completion separately unless a prespecified rule maps that situation to a decision.
- Assuming the mechanism is learned from the observed data. MAR and MNAR assumptions are not generally distinguishable from observed outcomes alone. Scenario analysis explores consequences; it does not identify the true mechanism.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A participant's later outcomes are unavailable after dropout, but earlier scheduled outcomes remain observed. What missingness pattern is this, and how should availability be represented?
- A recording outage affects every participant at one site. Why are independent participant-level Bernoulli draws not an adequate representation?
- In a simulated arm, \(P(Y=1)=0.40\), \(P(R=1\mid Y=1)=0.80\), and \(P(R=1\mid Y=0)=0.50\). Calculate the favorable proportion among observed outcomes.
- Why should the simulation report analysis completion separately from the probability of rejecting a null hypothesis?
- Give one diagnostic for an MNAR mechanism that uses information unavailable to the trial analyst but available to the simulator.
Solutions. (1) This is monotone dropout. Once dropout occurs, set the participant's later availability indicators to zero while retaining earlier observed records. (2) The outage creates dependence: all affected records share one site-level cause. Generate one site indicator and apply it to those records; independent draws would allow outcomes at the same site to fail separately. (3) The observed favorable proportion is \(0.40(0.80)/\{0.40(0.80)+0.60(0.50)\}=0.32/0.62\approx0.5161\). (4) An unavailable analysis is not automatically a non-rejection; separate reporting preserves the distinction between inability to calculate the test and a completed test that did not reject. (5) Check whether observation rates differ by the generated full outcome, such as comparing \(P(R=1\mid Y=1)\) with \(P(R=1\mid Y=0)\). The analyst would not know all missing outcomes, but the simulator does.