Why Stress-Test Stagewise Assumptions?
A simulation can report a reassuring Type I error rate for a seamless design under one data-generating scenario and still miss a consequential failure. Stagewise test statistics may be dependent because they reuse controls; a variance model may fit poorly under unequal variances; outcomes may not be observed at the assumed analysis time; or the enrolled population may change at the transition. The purpose of stress-testing is to change these features deliberately and see whether the decision rule retains the properties it is supposed to have.
This tutorial builds on the earlier material on simulating Type I error. The emphasis here is not the mechanics of generating repeated trials, but choosing scenarios that challenge the assumptions behind stagewise tests and interpreting what a failure means. A simulated rejection rate is a diagnostic for the specified data-generating process, not a universal guarantee about a design.
What Must Hold for the Stagewise Test?
Consider two stagewise test statistics, \(Z_1\) and \(Z_2\), combined using fixed weights \(w_1\) and \(w_2\). A common inverse-normal combination statistic is \(Z_C=w_1Z_1+w_2Z_2\), with \(w_1^2+w_2^2=1\). For a one-sided test at level \(\alpha\), the independent-stage calibration rejects when \(Z_C>z_{1-\alpha}\). Under the global null, this rule has its nominal level if each \(Z_i\) is standard normal and the two statistics are independent. The fixed weights and the analysis rule must also be specified independently of the unblinded stagewise test results, unless the design's calibration explicitly accounts for adaptation.
Independence matters because, even when each stagewise statistic has variance 1, the combined statistic has variance \(1+2w_1w_2\operatorname{Cov}(Z_1,Z_2)\). The usual independent-stage cutoff therefore need not have its stated null rejection probability when the covariance is nonzero.
The following short derivation shows how a shared control creates dependence. Suppose each stage has its own treatment group, but both treatment contrasts use the same control sample mean. Let \(D_1=\bar{X}_{T1}-\bar{X}_C\) and \(D_2=\bar{X}_{T2}-\bar{X}_C\). Assume independent participant outcomes across groups, with variance \(\sigma^2\), and equal sample size \(m\) in each group. Then each contrast has variance \(2\sigma^2/m\). Their covariance is the variance of the shared control mean, because the two treatment means are independent of each other and of the control mean:
The same correlation applies to the standardized contrasts when they use the correct standard deviations. Thus, reusing a control is not equivalent to having independent stagewise evidence. The exact dependence in a real design depends on its allocation, participants, and analysis; it should be represented in the simulation rather than assumed away.
Worked Example: Shared Controls and a Miscalibrated Combination Test
Suppose two standardized stagewise statistics have a bivariate standard normal distribution under the global null, with correlation \(0.5\), as in the equal-size shared-control setup above. Use equal combination weights and a one-sided nominal level of \(0.025\).
The weights are \(w_1=w_2=1/\sqrt{2}\). Each statistic has variance 1, so the combination has variance
The independent-stage cutoff is \(z_{0.975}\approx1.960\). Under the stated correlated null model, \(Z_C\) is normal with mean 0 and standard deviation \(\sqrt{1.5}\approx1.225\). Its actual rejection probability at that cutoff is therefore
The nominal 2.5% test rejects about 5.5% of the time under this model. This calculation assumes known variances and the stated joint normal distribution; a trial simulation should instead reproduce the planned estimators, sample sizes, and control reuse. A covariance-adjusted combination rule or a design-level calibration that simulates the actual dependence is an alternative to applying the independent-stage cutoff unchanged.
Stress Scenarios and What They Diagnose
A useful stress test changes one feature at a time before exploring combinations. This makes it easier to identify the source of a failure. For each scenario, specify the population, treatment effects, outcome distributions, allocation, control reuse, outcome-availability process, adaptation rules, and final test. Under a global null, set the treatment effect to zero in every population relevant to the claim. Include alternative scenarios as well, because a procedure may preserve Type I error while losing power or answering a different question.
For example, evaluate whether the final one-sided test rejects at no more than 2.5% under the global null.
Represent randomization, stage sizes, control reuse, outcome timing, adaptations, and the analysis exactly as specified.
Test a shared-control dependence pattern, unequal variances, delayed outcomes, or changed population mix while holding other features fixed.
Report the rejection rate and its Monte Carlo standard error for each scenario; investigate which assumption explains a material difference.
For \(R\) simulated trials and \(K\) rejections, the estimated Type I error is \(\hat{p}=K/R\), with Monte Carlo standard error \(\sqrt{\hat{p}(1-\hat{p})/R}\). This standard error describes simulation noise, not uncertainty about whether the data-generating model is realistic. A rate modestly above the target may be consistent with Monte Carlo variation; a persistent excess across sufficiently large runs deserves investigation. Also inspect stage-specific rejection rates, estimated correlations, standard errors, and how often each adaptation is triggered. An overall rate can conceal a failure in a particular path through the design.
Worked Example: Unequal Variances and an Unequal Allocation
Consider a hypothetical stage with 40 participants assigned to treatment and 160 to control. Under the null, outcomes are independent and normally distributed with a common mean, but their standard deviations are 2 outcome units in treatment and 1 outcome unit in control. Compare a pooled-variance standard error with the variance that follows from the data-generating model.
The true variance of the treatment-minus-control mean difference is
A pooled variance estimate combines the group variances using their degrees of freedom. Its expectation under this model is
The pooled standard-error formula would consequently have expected squared standard error
This is substantially below the true variance, \(0.10625\). As a rough normal approximation, substituting these values gives a null statistic with standard deviation \(\sqrt{0.10625/0.04972}\approx1.462\). Its one-sided rejection probability at 1.960 is approximately \(1-\Phi(1.960/1.462)=0.090\), not 0.025. This is a diagnostic approximation: the random estimated variance and finite-sample distribution mean the precise rejection rate should be obtained by simulation. A Welch standard error, \(\sqrt{s_T^2/40+s_C^2/160}\), does not impose a common variance; the simulated design should use the actual planned test and degrees-of-freedom calculation.
Delayed Outcomes and Population Shifts
A delayed outcome is not automatically a source of Type I error. If availability is independent of the outcome and treatment, and the analysis correctly handles the resulting information and timing, delay may reduce precision without biasing a treatment contrast. The stress test should distinguish that case from outcome-dependent or treatment-dependent availability. If the probability that an outcome is observed by the analysis differs according to the unobserved outcome, analyzing only available cases can change the compared groups.
Worked Example: Outcome-Dependent Availability at an Interim
Suppose, under a treatment-effect null, each randomized participant's outcome is \(-1\) or \(+1\) units with equal probability in both arms. By the interim, a treatment participant's outcome is observed with probability 0.8 when it is \(+1\) and 0.4 when it is \(-1\). In the control arm, these probabilities are reversed. Assume for this illustration that availability is generated independently across participants conditional on arm and outcome.
Among observed treatment outcomes, the expected mean is
For observed control outcomes, the numerator is \((0.5)(0.4)(+1)+(0.5)(0.8)(-1)=-0.20\), and the denominator remains 0.60, giving an expected observed mean of \(-0.333\) units. The expected complete-case contrast is therefore \(0.333-(-0.333)=0.667\) units, although the randomized groups have identical outcome distributions. A simulation that includes only observed interim outcomes should reveal the resulting distortion. Possible responses depend on the clinical setting and prespecified estimand: model the observation process, use an appropriate missing-data method, or delay the analysis until adequate outcomes accrue. Simply treating missing interim outcomes as if they were independent omissions does not address this mechanism.
A population shift requires a separate interpretation. Changes in eligibility or recruitment can change the target population and the average treatment effect, even when each stage's randomized comparison is internally valid. For example, suppose the treatment effect is 4 outcome units in a biomarker-positive subgroup and 0 units in a biomarker-negative subgroup. If 80% of Stage 1 participants are positive, the Stage 1 average effect is \(0.8(4)+0.2(0)=3.2\) units. If only 20% of Stage 2 participants are positive, that stage's average effect is \(0.2(4)+0.8(0)=0.8\) units. These are different population averages, not evidence by themselves of a broken null test.
Under a true zero treatment effect in every subgroup, a population-mix change alone does not necessarily inflate Type I error: valid stagewise statistics can still be calibrated under their respective null distributions. But a common-effect interpretation may no longer fit, and a variance or weighting assumption may fail if the outcome distributions change. Simulate the population shift with subgroup-specific effects and variances, then state whether the claim concerns a fixed target population, each stage's enrolled population, or a specified mixture. Do not diagnose a changed estimand as Type I error inflation unless the null rejection rate actually exceeds its target.
Reading Simulation Results Carefully
For illustration, suppose a hypothetical study runs 100,000 trials per scenario and uses a one-sided nominal level of 0.025. The following rejection rates are hypothetical simulation outputs, not universal performance guarantees.
| Scenario | Rejection rate | Interpretation to investigate |
|---|---|---|
| Independent stages, correctly specified variance | 0.0251 | Close to nominal calibration |
| Shared control, independence cutoff retained | 0.0540 | Dependence may inflate rejection |
| Unequal variances and allocation, pooled test | 0.0880 | Variance estimator may be too small |
| Same unequal-variance scenario, Welch test | 0.0260 | More consistent with nominal calibration |
For the shared-control rate, the Monte Carlo standard error is \(\sqrt{0.054(0.946)/100{,}000}\approx0.000715\). A rough 95% Monte Carlo interval is \(0.054\pm1.96(0.000715)\), or about 0.0526 to 0.0554. This interval reflects simulation variation only. It does not account for uncertainty in the assumed correlation or whether the simulated data-generating process matches the trial.
Compare the approaches for the same scenario: retain an independence-based combination rule, adjust for the modeled covariance, or calibrate the complete decision procedure by simulation. For unequal variances, compare the planned pooled test with a variance-robust alternative. For delayed outcomes, compare the planned analysis with a method that addresses the specified availability mechanism. Each alternative must be evaluated under the same null and alternative scenarios; a method that fixes one failure may have other limitations, such as reduced power or greater dependence on modeling assumptions.
Common Pitfalls and Limits
- Assuming shared controls are harmless because treatment groups are disjoint. The reused control mean enters both contrasts, inducing covariance. Include that shared data in the simulation and calibration.
- Changing several assumptions at once. A failed rate then has no clear explanation. First vary one feature at a time, then assess plausible combinations.
- Calling every population change a Type I error problem. A changed mixture can alter the estimand without inflating rejection under a correctly tested global null. Report both calibration and target-population interpretation.
- Ignoring Monte Carlo error. A small difference from nominal may be simulation noise. Report \(R\), the rejection count, and the Monte Carlo standard error.
- Assuming a successful simulated scenario proves validity. Results apply only to the specified models, rules, and parameter values. Stress tests cannot cover every departure from assumptions.
A strong stress test is therefore not a search for one favorable overall number. It is a structured comparison: identify the assumption, construct a plausible departure, measure the consequence, and decide whether the design or its interpretation must change.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- With equal combination weights and stagewise correlation \(\rho=0.2\), what is the variance of the combined statistic under standard normal stagewise margins?
- Why can a population shift change the meaning of a pooled treatment effect without necessarily inflating Type I error under the global null?
- In 50,000 simulated null trials, 1,400 reject. Calculate the estimated Type I error and its Monte Carlo standard error.
- What feature of the delayed-outcome example makes the complete-case contrast biased under a treatment-effect null?
Solutions. 1. With \(w_1=w_2=1/\sqrt{2}\), the variance is \(1+2(1/2)(0.2)=1.2\). 2. A changed subgroup mixture can change the population-average effect, while a valid test may still have its nominal rejection rate when treatment effects are zero in every subgroup. The estimand and Type I error are distinct questions. 3. The estimate is \(1{,}400/50{,}000=0.0280\). Its Monte Carlo standard error is \(\sqrt{0.0280(0.9720)/50{,}000}\approx0.000738\). 4. Observation probabilities depend jointly on treatment arm and outcome value, so observed participants are not representative of their randomized arm's outcome distribution; the expected observed means differ despite identical underlying outcome distributions.