Stress-Testing Re-Estimation Assumptions

Why Stress-Test a Re-Estimation Rule?

A sample-size re-estimation rule is a prespecified mapping from interim information to a final sample size. Its operating characteristics depend not only on the rule itself, but also on the assumptions used to generate the data. If the true outcome variance is larger than anticipated, events accrue more slowly than expected, or follow-up is shorter, a rule that performs well under its design assumptions may miss its power or feasibility targets.

Stress-testing uses targeted simulation to examine those vulnerabilities. The goal is not to search for one “correct” scenario. It is to learn which assumptions drive the rule’s performance, how severe plausible departures are, and what observable signs would indicate that the planned adaptation is inadequate. This complements the earlier material on simulating sample-size re-estimation rules and evaluating their operating characteristics: here the central task is choosing informative departures from the assumed data-generating conditions.

For each scenario, apply the complete planned rule to simulated trials, including the interim analysis, sample-size update, caps or floors, and final analysis. Summarize outcomes relevant to the design: probability of meeting the power objective under a stated treatment effect, probability of meeting an event target, achieved sample size, calendar duration, and other prespecified operating characteristics. Under a null treatment effect, rejection probability can also be recorded as a safety diagnostic; assessing or calibrating it is a separate task.

Designing Informative Simulation Scenarios

Begin with a reference scenario that represents the assumptions used to design the trial. Then create a small set of departures chosen because they are scientifically or operationally plausible. Change one factor at a time to isolate its influence, and include selected combined departures to assess whether risks compound. A stress test is not useful merely because it is extreme; the scenario should be interpretable and relevant to decisions about the rule.

For every scenario, specify the data-generating model and the rule being evaluated. For example, a continuous endpoint might be generated from independent normal outcomes in two randomized groups with common variance. A survival endpoint might be generated under proportional hazards, with an explicit event-time distribution, censoring mechanism, accrual pattern, and administrative end date. If these assumptions are not credible, the simulation may accurately evaluate the wrong model.

Use enough replications to make Monte Carlo uncertainty acceptably small. If \(R\) simulated trials produce an estimated probability \(\hat{q}\) of an outcome, its Monte Carlo standard error is \(SE_{\mathrm{MC}}=\sqrt{\hat{q}(1-\hat{q})/R}\). This quantifies simulation noise, not uncertainty about which scenario is realistic. At \(R=20{,}000\) and \(\hat{q}=0.80\), for example, the Monte Carlo standard error is about \(0.00283\), or 0.28 percentage points.

Events, Censoring, and Accrual Are Different Stressors

For an event-driven survival design, event count is often the information target. With equal allocation, proportional hazards, a log-rank test, a two-sided significance level \(\alpha\), and target power \(1-\beta\), a common approximation for required events is

where \(HR\) is the assumed hazard ratio. Translating \(D\) into enrollment requires assumptions about the probability of an observed event during the relevant follow-up. A lower observed event rate can arise from a lower underlying event incidence, more censoring, shorter follow-up, or some combination. Those mechanisms can have different implications for interpretation and duration, so they should not be collapsed into one unexplained “low event rate” scenario.

Accrual is another distinct dimension. If a design requires 618 participants, uniform accrual at 40 participants per month implies about \(618/40=15.45\) months of enrollment. At 25 per month, enrollment takes about \(618/25=24.72\) months. If every participant then receives 12 months of follow-up, the corresponding end-to-end durations are about 27.45 and 36.72 months. These calculations assume steady accrual and complete follow-up; a simulation should include pauses, varying recruitment rates, or an administrative study end when those are plausible. With complete follow-up and the same event process per participant, slower accrual may lengthen duration without changing eventual statistical power. A fixed calendar deadline can change both by preventing the planned follow-up or event target from being reached.

Treatment-Effect Scenarios and Interpretation

A sample-size rule that updates a nuisance parameter, such as variance, does not thereby learn the true treatment effect. Stress-testing should therefore distinguish uncertainty in nuisance parameters from uncertainty about the effect used to define the design objective. Under a smaller true effect, a rule that retains its original target effect may select the same sample size while the probability of success falls. This is a sensitivity analysis of the design’s power, not a reason to change the primary treatment question after seeing interim results.

Diagnostics, Comparisons, and Limitations

Useful stress-test summaries include the distribution of final sample size, the fraction of trials hitting the cap or floor, achieved power, event shortfall, and calendar duration. Inspect more than averages: a mean sample size can conceal a substantial fraction of trials at a cap, and an average duration can conceal a long tail. For each simulated performance probability, report Monte Carlo uncertainty; for duration or sample size, examine quantiles as well as the mean.

A warning sign is a performance change that cannot be explained by the scenario’s mechanism. For example, a variance-only change should not alter the simulated treatment effect, while an accrual-only change should not alter eventual power if every participant receives the same complete follow-up and the event process is unchanged. If it does, inspect the simulation code for unintended changes such as a calendar cutoff or altered censoring. Also check that each simulated trial applies the interim rule at the correct time and respects its sample-size bounds.

One-factor-at-a-time scenarios are easy to interpret but can miss interactions; combined scenarios reveal whether, for example, slow accrual and heavier censoring jointly make an event target infeasible. Analytic power and sample-size calculations are useful checks for simple models, but they cannot represent complex interim rules, caps, variable accrual, and censoring as flexibly as simulation. Conversely, simulation is only as credible as its data-generating assumptions. It does not establish that a scenario is plausible, and it cannot replace clinical judgment about meaningful effects or operational constraints.

A common mistake is to conclude that a rule is robust because it meets its target in the reference scenario. That reasoning tests only the assumptions under which it was designed. The correction is to prespecify plausible departures, evaluate each with the full rule, and report where performance degrades or feasibility fails. Another mistake is to read a difference between two simulated success rates as meaningful without considering Monte Carlo error; increase the number of replications or report the uncertainty before interpreting small differences.