Stress-Testing Missingness and Nonadherence

Why Missingness and Nonadherence Belong in Design Assurance

A design can perform well when every participant follows the assigned treatment and supplies an outcome, yet behave differently when either condition fails. Missing outcomes reduce information and may bias an analysis; nonadherence changes the treatment contrast that randomization creates. In an adaptive trial, these changes can also affect interim statistics and the probability of reaching a particular decision. Design assurance should therefore simulate plausible patterns of both, rather than treating them as a single percentage reduction in sample size.

The purpose is not to predict exactly who will miss a visit or stop treatment. It is to find out whether the design's conclusions and feasibility depend strongly on assumptions about those events. The simulation should preserve the planned randomization and adaptive rules, generate the events in a specified order, and apply the analysis exactly as intended.

Specify the Data-Generating Scenario

A useful scenario states how outcomes, adherence, and missingness arise. For example, a simple continuous-outcome model might generate a baseline covariate \(X_i\), randomize \(A_i\), generate an outcome \(Y_i\) conditional on assignment and adherence, and then generate \(R_i\) conditional on observed or unobserved information. In a real adaptive simulation, the allocation and interim decisions are also generated according to the design's rules. The same rules must be applied in every scenario being compared.

Missingness mechanisms are commonly described by the information that predicts \(R_i\). Missing completely at random (MCAR) means observation is independent of both observed and unobserved outcome information. Missing at random (MAR) means that, conditional on included observed variables, missingness does not depend on the unobserved outcome. Missing not at random (MNAR) means that missingness still depends on the outcome or other information not observed for those with missing data, even after conditioning on the observed variables. These are assumptions about the data-generating process, not labels that can usually be confirmed from the observed data alone.

For example, a common MAR simulation might make the chance of a missed outcome depend on recorded baseline severity and earlier observed symptoms. An MNAR scenario might additionally make participants with worse unobserved current outcomes more likely to miss the visit. An MCAR scenario might use a fixed missing probability unrelated to participant characteristics or outcomes. The chosen probabilities and predictors should be justified as plausible stress conditions for the trial, rather than selected only because they produce a preferred result.

Nonadherence needs its own specification. It can depend on assignment, adverse effects, baseline characteristics, observed response, or unobserved prognosis. The outcome model should then state how actual treatment exposure affects \(Y_i\). For instance, an assigned participant who does not receive the active treatment might have an outcome distribution closer to the control distribution. Assuming adherence is independent of prognosis is a substantive simplification; it should be stated if used.

What to Measure Across Simulated Trials

For each simulated trial, record more than whether the final decision was positive. Useful summaries include the probability of each planned decision, estimator bias, variability, root mean squared error, and the fraction of trials meeting recruitment or usable-outcome targets. If \(\hat\theta_b\) is the estimate from replicate \(b\), and the scenario's true parameter is \(\theta\), the empirical bias and root mean squared error across \(B\) replicates are

A decision rate is the proportion of simulated trials that meet a specified decision condition. Under a null scenario it may be a false-positive decision rate; under an alternative it may be power or the probability of selecting the correct option. State the condition and denominator explicitly, particularly if some simulations stop early or fail to produce an estimate. A high decision rate is not reassuring if the estimator is biased or the scenario has not represented the intended missingness and adherence patterns.

The estimator must match the estimand. In a randomized trial, an intention-to-treat (ITT) contrast compares participants according to randomized assignment, regardless of subsequent adherence. It targets the effect of assignment under the trial's treatment-policy definition. A contrast restricted to adherent participants is not automatically an estimate of the causal effect of receiving treatment: adherence can be related to prognosis, and conditioning on it can undo the comparability created by randomization.

Why Outcome-Dependent Missingness Can Bias an Estimate

MCAR attrition mainly reduces information in the simple setting above. If missingness depends on outcome, an observed-case mean can instead describe a selected subset. The following calculation shows the mechanism directly. Suppose the target is the mean outcome difference in all randomized participants, but the analysis compares only observed outcomes. The calculation assumes the stated group means exactly; it is an illustration of selection bias, not a claim about a particular study.

A short justification makes the source of bias clear. If the observed fraction is \(q\), the observed mean is \(\mu_{\mathrm{obs}}\), and the missing-group mean is \(\mu_{\mathrm{mis}}\), then the full mean is \(q\mu_{\mathrm{obs}}+(1-q)\mu_{\mathrm{mis}}\). The observed mean equals the full mean only when the observed and missing groups have the same mean, or when no outcomes are missing. Under MCAR, equality holds in expectation; under outcome-dependent missingness it need not. This is why scenarios should vary not only the missing percentage but also the relationship between missingness and outcome.

Nonadherence Changes the Assignment Contrast

A simple treatment-effect model clarifies how adherence can affect an ITT decision rate. Assume treatment has a constant effect \(\tau\) on the outcome for every participant who receives it, no effect for those who do not receive it, and no treatment exposure in the control arm. If adherence is independent of prognosis, and a fraction \(r\) of those assigned active treatment receive it, the expected outcome contrast by assignment is \(r\tau\). This follows because the active-assignment group averages a fraction \(r\) with effect \(\tau\) and fraction \(1-r\) with effect 0, while the control group has effect 0. These assumptions are deliberately simple; real adherence and treatment effects may vary across participants.

Build and Check the Simulation

A clear simulation sequence helps prevent inconsistent assumptions. In an adaptive design, missingness before an interim analysis can affect both the interim statistic and later allocation decisions. Generate events in the order they would occur in the trial, and apply the same analysis and stopping rules as the planned design.

Several diagnostics can reveal a simulation that is not testing what it claims to test. Check that the simulated missing proportion and adherence proportion match their specified probabilities on average; check them by treatment arm and by the covariates or outcomes intended to predict them. Confirm that the analysis receives missing values in the correct form and does not inadvertently use outcomes that should be unavailable at an interim. Test limiting cases: with no missingness and full adherence, the simulation should recover the corresponding complete-data behavior; with a constant outcome effect and no adherence, the treatment-exposure contribution should disappear under the simple model above.

Also inspect whether the estimator targets the intended quantity. Complete-case analysis generally requires more than a large sample to be unbiased; under MCAR it can estimate the same mean contrast in expectation, while under MAR it may require an appropriately specified model using observed predictors. Likelihood-based or multiple-imputation analyses likewise rely on a correctly specified model and suitable observed information; neither automatically solves MNAR. Compare the planned primary analysis with justified alternatives, but do not label a per-protocol contrast as causal merely because it includes adherent participants.

Feasibility, Alternatives, and Limits

Feasibility should be evaluated as a probability, not only through an expected number of observed outcomes. If each of 120 randomized participants independently supplies an outcome with probability 0.75, the expected number observed is \(120(0.75)=90\), and the standard deviation of that count is \(\sqrt{120(0.75)(0.25)}=4.743\). A normal approximation with continuity correction gives the chance of at least 85 observed outcomes as approximately \(P(Z\ge(84.5-90)/4.743)=P(Z\ge-1.160)\), or about 0.877. Thus an expected count of 90 does not mean the target of 85 is virtually guaranteed. In a trial with adaptive decisions, simulate the feasibility event using the actual stopping and follow-up rules.

The main alternatives answer different questions. A complete-case analysis is simple but discards incomplete records and can be biased when missingness is related to outcomes. A model-based analysis can use observed covariates and repeated outcomes under its assumptions, but it depends on model specification and adequate data. An adherence-adjusted or per-protocol analysis targets a different quantity and requires additional assumptions to address selection into adherence. Simulation does not choose among these approaches by itself: it reveals how each behaves under explicitly stated scenarios.

There are limits to what stress-testing can establish. The true missingness mechanism is usually unknowable from the observed data alone, and plausible scenario ranges can be uncertain. Simulated results are conditional on the data-generating assumptions, analysis implementation, and adaptive rules encoded. Report those assumptions and examine more than one credible pattern, including mechanisms that differ in severity and in their dependence on observed or unobserved outcomes. The aim is to make vulnerabilities visible, not to certify that an unknown future pattern has been covered.

Common Mistake

Wrong reasoning: “Twenty percent of outcomes are missing, so I can multiply the planned sample size by 0.8 and conclude the design loses only precision.” That treats all missingness as MCAR and ignores the random variation in observed counts. It also overlooks that outcome-dependent missingness can bias an estimator, and that nonadherence changes the treatment contrast even when outcomes are complete.

Correction: Simulate the missingness and adherence mechanisms separately, connect them to the variables that plausibly predict them, and pass the resulting data through the intended analysis and adaptive rules. Report effects on both decision rates and estimation, as well as the probability of meeting feasibility targets.