Robustness to Operational Deviations

From Protocol to Operational Scenario

An adaptive design is evaluated under a planned protocol: when information is reviewed, which decision rule is applied, and how recruitment and follow-up proceed. In practice, the trial may depart from that plan. An interim review might occur later than scheduled, enrollment may continue while a decision is pending, or a resource limit may be applied differently than intended. These departures can change both the data available to the adaptation rule and the trial’s eventual operating characteristics.

The aim here is not to assume that every departure will occur, or to replace the planned design with an informal alternative. It is to make a departure precise enough to simulate, then measure its consequences for prespecified targets such as power, type I error, probability of reaching a resource cap, or expected final sample size. The earlier material on delayed or incomplete interim data, recruitment patterns, and missingness or nonadherence treats those particular sources of variation; the emphasis here is on evaluating departures from the protocol as implemented.

A scenario should describe an operational mechanism, not merely label a run “stress.” For example, “the interim is delayed” is incomplete unless the simulation specifies what happens during the delay: does enrollment pause, continue to a prespecified cap, or continue without a cap? These alternatives represent different protocols and can produce different results. Likewise, an enrollment overshoot should specify its size or distribution and whether the additional participants contribute outcome data to the final analysis.

Before simulating, define the implementation being compared and the target quantities. A useful target might be the probability of declaring efficacy under a specified treatment effect, or the probability that both an efficacy criterion and a resource constraint are satisfied. A difference is interpreted in the natural units of that target: a difference in power is a difference in probability, often reported in percentage points; a difference in mean enrollment is reported in participants.

Assumptions and Comparison Strategy

Simulation estimates are meaningful only for a sufficiently specified data-generating model and operational scenario. The model must generate data under the chosen null or alternative conditions, represent the adaptation and final analysis rules, and implement the deviation consistently. If the question concerns a review delay, for example, the code must encode whether recruitment or treatment continues during that delay. If it concerns enrollment overshoot, the rule for who is analyzed and when must also be explicit.

For a comparison, simulate each replicate under both the planned protocol and the deviation scenario. When both runs can use the same underlying simulated conditions, pair them by replicate. This common-random-number approach, covered in earlier material, can reduce simulation noise in the estimated difference. It does not make the two outcomes identical or remove model uncertainty; it makes the comparison more precise when the paired outcomes are positively correlated.

The paired comparison requires that each pair represent the same underlying simulated conditions in a meaningful way. Replicates should be generated independently across \(i\), with the planned and deviation runs within a pair linked as intended. For a probability target, \(Y\) is usually an indicator equal to 1 when the target event occurs and 0 otherwise. For a resource target, \(Y\) could be final sample size. The normal-based interval for \(\bar{D}\) is an approximation, most useful when the number of independent replicates is sufficiently large and the replicate differences are not dominated by a few extreme values.

Why Pairing Helps: A Short Justification

For a paired replicate, the difference is \(D_i=Y_{i,D}-Y_{i,P}\). Its variance is

This follows by expanding the variance of a difference: the variance terms add, while the covariance term enters twice with a negative sign. If the paired outcomes are positively correlated, the covariance term reduces the variance of \(D_i\). Across \(R\) independent pairs, the variance of their average is \(\operatorname{Var}(D_i)/R\), which motivates estimating its standard error with \(s_D/\sqrt{R}\). Pairing is not guaranteed to help: if outcomes are weakly or negatively correlated, the variance reduction may be small or absent.

Quantifying Resource Consequences

Operational deviations can affect burden and feasibility even when a statistical target changes little. For resource outcomes, use a consistent definition in both implementations: for example, final number enrolled, number randomized, or number with an observed primary outcome. State whether the quantity is measured before or after any cap, and do not treat those definitions as interchangeable.

Averages do not describe every operational risk. If the concern is exceeding a cap, define the event explicitly and estimate its probability. If the concern is a joint goal, such as declaring efficacy without exceeding a resource limit, define one indicator for that joint event rather than inferring its probability from two separate marginal summaries.

Diagnostics, Alternatives, and Limits

A credible robustness assessment checks both the simulation output and the implementation. First verify that the planned scenario reproduces its intended rules and that the deviation changes only the specified operational feature, unless other changes are part of the scenario definition. Unit tests and independent code checks from earlier material are useful safeguards. Then inspect whether the simulated traces show the expected operational sequence: for example, whether enrollment continues during the modeled delay and stops at the specified cap. A result that contradicts the encoded rule is a reason to investigate the code, not to reinterpret the result after the fact.

Compare the point estimate with its Monte Carlo uncertainty. A small estimated change with a wide interval may simply be imprecise; a precisely estimated change can still be operationally important. Conversely, statistical precision does not establish practical importance. The design team must specify in advance which changes matter in context—for example, an acceptable loss in power or an acceptable increase in participants. This is distinct from choosing a statistical significance threshold.

Paired simulation is often preferable for estimating a change when the same underlying conditions can be used in each implementation. Independent simulations are a reasonable alternative when pairing is not meaningful or practical, but the uncertainty for a difference must then account for the two separate estimates rather than treating runs as paired. A scenario-specific comparison is also different from summarizing performance across scenarios: it answers how the design behaves under one defined departure, not how likely that departure is or how to combine it with others.

Several limitations remain. Results depend on the data-generating assumptions and on how accurately the operational mechanism is represented. A stress scenario is not evidence that the real trial will experience that deviation. A limited set of scenarios cannot guarantee robustness to unexamined departures, and averaging can hide a harmful tail or a subgroup-specific consequence. Finally, Monte Carlo intervals quantify simulation error only; they do not quantify uncertainty in model inputs, implementation fidelity in the future trial, or the relevance of the chosen scenarios.

Common Mistakes

• Leaving the deviation vague. “The interim is late” does not specify what accrual or treatment does during the delay. Define those rules before simulating.
• Comparing unpaired runs as if they were paired. A paired standard error requires replicate-level differences constructed from meaningful pairs. Without pairing, use the uncertainty for independent estimates.
• Calling Monte Carlo precision design certainty. A narrow interval shows precise computation under the assumed model, not certainty that the model or scenario matches reality.
• Reporting only power. Enrollment, timing, resource caps, and joint objectives may be affected even when power changes little.
• Interpreting a mean as a guarantee. A mean increase of 35 participants does not imply each trial adds about 35; inspect the distribution when individual-trial burdens matter.