Stress-Testing Designs Under Operational Deviations

Why Stress-Test Operational Deviations?

A group sequential design is planned around analyses at specified information levels and a stopping rule applied at each analysis. In practice, data may be delayed, an interim review may be missed, or fewer outcomes than expected may be available. These are not merely scheduling inconveniences: they can change which statistics are observed, when a decision is made, and how much information supports it.

Stress-testing asks a practical question: if the trial is implemented under a plausible disruption, does the design still meet its objectives? The answer should come from simulating the disrupted process—not from assuming that the planned operating characteristics automatically carry over. The earlier material on simulation design and sensitivity to nuisance assumptions provides useful foundations; here the scenarios concern deviations in implementation and data availability.

Definition: An operational deviation scenario is a specified departure from the planned conduct of a trial, such as a delayed analysis, a missed analysis, a changed information level at an analysis, or incomplete outcomes. A stress test estimates the design’s operating characteristics after applying both that deviation and a clearly stated decision policy.

Let \(R\) denote the number of simulated trial paths. For path \(r\), let \(D_r=1\) if the design rejects \(H_0\), and \(D_r=0\) otherwise. Under a null scenario, the simulated type I error estimate is

\widehat{\alpha}=\frac{1}{R}\sum_{r=1}^{R}D_r.

Under a specified alternative, the corresponding estimate is power. Other useful summaries include the look at which a decision occurs, the information or sample size at stopping, and calendar time to the decision. A simulation is informative only if each path follows the operational policy that would actually be used.

Represent the Disruption as a Rule

For a planned design, write the planned information levels as \(I_1,\ldots,I_K\), with stopping boundaries \(b_1,\ldots,b_K\). At a look, the test statistic is compared with the applicable boundary; the trial stops if the rule is met and otherwise continues. In a two-sided design, the rule must also specify how both tails are handled. This tutorial uses a one-sided upper-boundary rule for illustration.

A delayed analysis can mean two different things. If the analysis is postponed but the data freeze remains unchanged, the statistic and information may be unchanged; the decision is simply later in calendar time. If enrollment or outcome collection continues during the delay, the analysis may instead occur with more information. These cases can have different statistical and operational consequences, so they should not be represented by one generic “delay” flag.

A missed analysis also needs a policy. For example, the protocol might skip the missed look and wait for the next planned look, or it might permit a replacement analysis when specified conditions are met. The stopping boundary and data used at that next analysis must follow the specified policy. Carrying a planned boundary to a different information level without evaluation is not automatically valid.

Incomplete outcomes require the same care. The simulation must specify which outcomes are observed, how missingness arises, which analysis method is applied, and how information is calculated. Randomly deleting outcomes is a reasonable model only when it represents the operational mechanism of interest. If missingness depends on observed or unobserved outcomes, that mechanism must be represented appropriately; a simulation under independent loss does not establish robustness to informative missingness.

Why a Missed Look Can Change Error and Power

With upper-boundary stopping, the planned rejection event is the union of crossing events across looks. If \(Z_j\) is the statistic at look \(j\), then

\{\text{reject}\}=\bigcup_{j=1}^{K}\{Z_j\geq b_j\}.

When a look is missed, the set of opportunities to stop changes. The trial may no longer reject early on a path that would have crossed at the missed look. But the consequences are not determined by that observation alone: continuing to a later look changes which paths reach that look, and an analysis at a shifted information level may use a different boundary. The overall type I error and power therefore depend on the full rule, not just on the number of looks.

This also explains why a stress test should simulate the decision process path by path. It is generally not enough to adjust the final sample size or to multiply a planned power estimate by an assumed retention fraction. Those shortcuts do not account for the timing and dependence of sequential statistics or for altered stopping opportunities.

Worked Example: Comparing a Planned Design With a Delayed Analysis

Suppose a hypothetical one-sided design has three planned looks. Its boundaries were calibrated for the planned information levels. A simulation uses \(R=10{,}000\) paths under the null and another \(10{,}000\) under a specified alternative. The operational policy says that, when the second analysis is delayed, it uses all outcomes available at the delayed analysis and applies the boundary specified by the prespecified rule for that information level.

Under the planned schedule, 248 of the 10,000 null paths reject, and 8,420 of the 10,000 alternative paths reject. Thus,

\widehat{\alpha}_{\text{planned}}=\frac{248}{10{,}000}=0.0248, \qquad \widehat{\text{Power}}_{\text{planned}}=\frac{8{,}420}{10{,}000}=0.842.

In a scenario where the second analysis is delayed and additional outcomes accrue, 275 null paths and 8,200 alternative paths reject:

\widehat{\alpha}_{\text{delayed}}=\frac{275}{10{,}000}=0.0275, \qquad \widehat{\text{Power}}_{\text{delayed}}=\frac{8{,}200}{10{,}000}=0.820.

These hypothetical results show a higher estimated type I error and lower estimated power under the specified delay policy. Whether that is acceptable depends on the design objectives and prespecified tolerances; the estimates alone do not determine acceptability. The comparison is interpretable because the scenario specifies what data enter the delayed analysis and how its boundary is selected. A simulation that merely postpones the calendar date while freezing the data at the original information level would answer a different question.

Model Incomplete Data and Information Explicitly

In a simple two-group comparison of means with independent outcomes, common known standard deviation \(\sigma\), and \(n_T\) and \(n_C\) observed participants, the standard error of the difference in sample means is

SE=\sigma\sqrt{\frac{1}{n_T}+\frac{1}{n_C}}.

This formula assumes independent observations, the stated common standard deviation, and that the observed participants represent the analysis being described. It illustrates how incomplete outcomes can reduce information. Actual group sequential designs may use other test statistics and information calculations; the simulation should use the method specified for the trial.

Worked Example: Translating Missing Outcomes Into Information

Consider a hypothetical balanced trial with a continuous outcome measured in points. The planned sample size is 100 participants per group, and the assumed standard deviation is 10 points in each group. Suppose 20% of outcomes are unavailable in each group, leaving 80 observed participants per group. Assume, for this calculation only, that outcomes are missing completely at random and the analysis is a complete-case comparison.

The planned standard error is

SE_{\text{planned}} =10\sqrt{\frac{1}{100}+\frac{1}{100}} =10\sqrt{0.02} =1.414\text{ points}.

With 80 observed participants per group, it becomes

SE_{\text{observed}} =10\sqrt{\frac{1}{80}+\frac{1}{80}} =10\sqrt{0.025} =1.581\text{ points}.

For equal groups under this model, information is proportional to the reciprocal of the variance of the difference, \(1/SE^2\). The information fraction is therefore

\frac{I_{\text{observed}}}{I_{\text{planned}}} =\frac{1/(1.581)^2}{1/(1.414)^2} =\frac{0.400}{0.500} =0.800.

So the observed information is 80% of the planned information under these assumptions. This calculation describes information loss; it does not establish that complete-case analysis is unbiased in a real missing-data setting. If missingness depends on outcome values or treatment, the simulation must represent that mechanism and the planned analysis method.

Build Scenarios and Inspect the Results

A useful scenario matrix separates the type of disruption from its severity. For example, compare the planned schedule with a modest delay, a larger delay, one missed look, and several plausible levels or patterns of incomplete outcomes. Include combinations only when they are operationally credible. For each scenario, record the assumptions and the response rule, so an operating characteristic can be traced to a well-defined implementation.

At minimum, examine type I error under the null and power under the alternatives that matter for the design. Also inspect stopping look, information at stopping, sample size, and calendar time. A design may retain power but require materially longer follow-up; it may stop at the same nominal look while using less information; or a disruption may make a planned interim review infeasible. These outcomes are not interchangeable, so report them separately.

The main alternative to stress-testing is to evaluate only the planned schedule. That analysis answers whether the design performs as intended under its planning assumptions, but not whether it is robust to operational departures. Another approach is a simple analytic adjustment, such as inflating enrollment for expected missing outcomes. That can be useful for planning a particular loss mechanism, but does not by itself account for changed analysis timing, stopping boundaries, or non-independent missingness. Simulation is especially useful when the operational rule and sequential stopping interact, provided the simulated data and decision process match the proposed trial.

Practical check: For each scenario, verify that (1) the disruption mechanism is explicit, (2) the analysis uses the intended available data, (3) the stopping rule identifies the correct boundary at each actual analysis, and (4) all operating characteristics are computed from the resulting trial paths.

Diagnostics, Limitations, and a Common Pitfall

Warning signs include analyses occurring at impossible information levels, a missed look being treated as if it had occurred, or a delayed look using the original data freeze despite the scenario allowing more outcomes to accrue. For incomplete data, check that the number observed by group and look is plausible, and that the analysis does not use unavailable outcomes. Compare planned and disrupted distributions of information and stopping time, not just their averages. Large changes in these distributions can reveal an operational weakness even when aggregate power appears similar.

Stress tests are conditional on the scenarios and assumptions selected. A favorable result under independent outcome loss does not guarantee robustness to informative loss, and a result under one delay pattern does not cover every possible delay. Simulation also estimates operating characteristics with finite replication; apparent small differences should be interpreted in light of simulation uncertainty rather than treated as exact.

Wrong reasoning: “The second look was missed, but the final boundary is unchanged, so type I error and power are unchanged.”
Correction: The missed look changes the paths eligible for early stopping, and the later analysis may occur at a different information level. Apply the specified policy to each path and estimate the resulting operating characteristics.

A design should not be called robust simply because it still runs. Robustness is relative to stated performance objectives: for example, acceptable error control, adequate power, and feasible duration under scenarios judged plausible. If a scenario fails an objective, the result is useful evidence to revise the schedule, operational policy, or assumptions before the design is implemented.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

A delayed analysis occurs two weeks later, but the data freeze is unchanged. Has the information necessarily increased?
In a null simulation, 180 of 8,000 disrupted paths reject. Calculate the estimated type I error.
Why is a missed interim look not equivalent to keeping the schedule and changing only the final sample size?
A simulation assumes outcomes are missing completely at random. What kind of conclusion does it support, and what does it not establish?

Solutions. (1) No. If the data freeze and analysis data are unchanged, the delay changes calendar time but need not change statistical information. (2) The estimate is \(180/8{,}000=0.0225\), or 2.25%. (3) A missed look removes an opportunity to stop, changes which paths continue, and may change the information and boundary at later analyses; the full stopping policy must be simulated. (4) It evaluates performance under the specified independent missingness assumption and analysis method. It does not establish performance when missingness depends on outcomes or treatment, or under other mechanisms not simulated.