Why Stress-Testing Requires More Than One Scenario
A multi-endpoint sequential design is a joint decision rule: rejection of one hypothesis can affect what happens to another, and each endpoint may be assessed at several interim looks. Its operating characteristics therefore depend not only on nominal alpha levels, but also on the joint distribution of the endpoint statistics and on when each result is available. A graph that behaves acceptably under one correlation and one calendar schedule may behave differently under another plausible combination.
Stress-testing evaluates a fixed, prespecified testing rule under a range of plausible conditions. It does not change the rule after seeing trial results. The aim is to detect sensitivity: for example, a familywise error rate close to its limit only under one dependence pattern, or a marked loss of power when an endpoint matures later than planned.
A simulation must reproduce the entire decision rule, not just its nominal cutoffs. That includes endpoint-specific boundaries at each look, the graph's alpha updates after rejection, which tests can be performed when results are unavailable, and the rule for ending follow-up. The rule is held fixed across scenarios; the data-generating assumptions are varied.
Specify the Joint Data-Generating Model
Let \(Z_{jk}\) denote the standardized test statistic for endpoint \(j\) at its \(k\)th available analysis. Under a specified scenario, the vector of all such statistics has a joint distribution determined by the null or alternative means and a covariance structure. A common working model is multivariate normal:
Here \(\boldsymbol{\mu}\) depends on which hypotheses are true and on the assumed effects under false nulls; \(\boldsymbol{\Sigma}\) describes dependence both across endpoints and across repeated looks. The normal model is an assumption to be justified for the planned score statistics, not a consequence of the fact that a simulation is being run. If the design uses a different statistic or a non-normal model, simulate that model instead.
For a single endpoint with nested information and independent increments, the canonical standardized statistics have correlation \(\operatorname{Corr}(Z(I_1),Z(I_2))=\sqrt{I_1/I_2}\) when \(I_1\le I_2\). This follows from the shared information accrued by the earlier look. But that formula does not determine cross-endpoint correlations in every trial: different endpoints can have different information processes, missingness, or shared-data structures. Specify cross-endpoint covariance from the joint score process or a defensible model for it, and vary it in the stress tests.
Timing is part of the model. Define each analysis by its actual information level and availability, not merely by a nominal calendar label. A delayed endpoint is not tested at a look when its result is unavailable. Its first analysis occurs when the prespecified maturity condition is met; the simulation must then apply the correct boundary and graph rule at that information level.
How to Evaluate Error Control
Under the global null, all hypotheses are true, so every rejection is an error. The estimated FWER is the proportion of simulated trials in which the rule rejects at least one hypothesis. That is a useful initial check, but it is not sufficient to establish strong control: configurations with some false and some true nulls must also be examined. For each such configuration, count a trial as an FWER event if any true null is rejected, even when a false null is correctly rejected in the same trial.
A practical stress-test set includes the global null, relevant partial-null configurations, and plausible alternative effects on the false nulls. Vary endpoint correlations over a justified range, including values that represent weak, moderate, or strong dependence where scientifically plausible. If the correlation structure is uncertain, also examine alternative covariance models rather than treating one selected correlation as known.
Worked Example: Monte Carlo Precision Under the Global Null
Suppose a hypothetical two-endpoint graph uses a fixed three-look rule with target FWER \(\alpha=0.025\). Under one plausible joint-null covariance scenario, a simulation of \(100{,}000\) independent trial replicates records at least one rejection in 2,410 replicates. The simulated rule and analysis schedule are applied in every replicate.
The estimated FWER is
The approximate Monte Carlo standard error, which measures simulation noise rather than uncertainty in the statistical model, is
An approximate 95% Monte Carlo interval is \(0.02410\pm1.96(0.000485)\), or approximately \(0.02315\) to \(0.02505\). The point estimate is below 0.025, but the interval reaches slightly above it. This run is therefore consistent with FWER near the target; it does not demonstrate that the true simulated FWER is at most 0.025 with high precision. More replicates can reduce Monte Carlo error, but cannot repair an incorrect covariance model or an incorrectly coded rule.
For each simulated configuration, record both the FWER estimate and its Monte Carlo precision. The binomial standard error formula applies to the indicator of an FWER event across independent replicates. If simulations use a variance-reduction or dependent sampling scheme, use a precision estimate appropriate to that scheme. Do not interpret a small difference between scenarios as meaningful if it is comparable to their Monte Carlo error.
Derivation: Dependence Across Repeated Looks
The canonical correlation for repeated looks explains why simulating each look independently is generally wrong. Under a Brownian-motion score model, let \(S(I)\) be the cumulative score at information \(I\), with mean zero and variance \(I\) under the null. For \(I_1\le I_2\), the earlier score is contained in the later cumulative score, so \(\operatorname{Cov}\{S(I_1),S(I_2)\}=I_1\). Standardizing gives \(Z(I)=S(I)/\sqrt I\), and hence
Thus, the later statistic is correlated with the earlier one even under the null. A simulation that draws independent \(Z\)-statistics at each look removes this correlation and can give incorrect crossing probabilities and error rates. The derivation relies on the stated nested-information, independent-increment model; it should not be applied mechanically when the endpoint statistics do not follow that model.
Stress-Testing Unequal Information and Delayed Analyses
When endpoints accrue information at different rates, use the endpoint-specific information fractions at each actual analysis. For example, a three-calendar-look schedule might have endpoint A analyzed at information fractions 0.45, 0.75, and 1.00, while endpoint B has fractions 0.30, 0.62, and 1.00. If B is not mature at the first calendar look, B has no test at that time. The first B test occurs at its next available analysis, using the prespecified boundary appropriate to that information and the graph state then in effect.
A timing stress test should compare plausible schedules, not silently substitute a planned information fraction for the information actually reached. It should also preserve the correct joint dependence between the statistics that are available. Delayed availability can change which tests occur together, the sequence of graph updates, and the opportunities to reject. These effects cannot generally be summarized by shifting a single endpoint's nominal boundary.
Worked Example: A Delayed Endpoint Under a Partial Null
Consider a hypothetical two-endpoint design with endpoint A available at information fractions 0.45, 0.75, and 1.00. Endpoint B is unavailable at the first calendar analysis and is available at the next two, at fractions 0.62 and 1.00. Suppose B's null hypothesis is true and A's is false. In \(100{,}000\) simulated trials generated under the specified joint score model, the graph rejects true hypothesis B in 1,830 trials.
Because only rejections of true hypotheses count for FWER in this partial-null configuration,
The approximate Monte Carlo standard error is
This estimate is below 0.025 in this particular configuration. It is not, by itself, evidence of strong control across all configurations or schedules. A separate run must assess other true-null subsets and plausible endpoint dependence. In every run, the B statistic must first be generated and tested at its actual available information; assigning B a first-look result at 0.30 would simulate a different, unavailable analysis and would not evaluate this design.
To identify timing sensitivity, compare operating characteristics across schedules that could realistically occur: planned maturity, plausible delay, and any prespecified contingency schedule. Keep the analysis rule consistent with each schedule. A large change in FWER or power indicates that the design's performance depends materially on the timing assumption and merits design review.
Power Under Alternative Correlations
Power must be evaluated under stated alternatives, not inferred from null simulations. Specify which endpoints are false, their effect sizes, the information accrual, and the dependence structure. Report endpoint-specific rejection probabilities and, when clinically relevant, the probability of rejecting all or a specified subset of false hypotheses. Correlation can affect these quantities because joint paths determine which rejections occur and when the graph reallocates testing opportunities.
Worked Example: Comparing Joint Power Across Correlations
Suppose a hypothetical two-endpoint design is evaluated under alternatives for both endpoints. For each correlation scenario, \(100{,}000\) joint trial paths are simulated using the same effects, information schedule, and graph rule. The following counts are illustrative simulation output.
| Endpoint correlation | A rejected | B rejected | Both rejected |
|---|---|---|---|
| -0.30 | 75,000 | 66,000 | 46,200 |
| 0.00 | 76,000 | 67,500 | 51,300 |
| 0.80 | 78,000 | 70,000 | 61,000 |
For correlation 0.80, the estimated probability of rejecting both hypotheses is \(61{,}000/100{,}000=0.6100\). The estimated marginal powers are \(0.7800\) for A and \(0.7000\) for B. The joint probability cannot exceed either marginal probability, and \(0.6100\) is below both, as required. The estimates show that joint power is sensitive to the assumed correlation in this hypothetical design. They do not imply that positive correlation must always increase joint power; the direction and magnitude depend on the rule, effect sizes, and covariance structure.
Diagnostics, Alternatives, and Limitations
Before interpreting simulation output, validate the implementation and the assumptions. Useful checks include:
- Confirm that the covariance matrix is symmetric and positive semidefinite, and that simulated variances and correlations match the specified values.
- Check that information fractions are increasing within each endpoint and that unavailable results are not tested early.
- Verify that boundaries, graph updates, stopping rules, and the definition of an error event match the prespecified design.
- Run simple cases with known behavior, such as a single endpoint or a single analysis, to catch coding errors before relying on a complex simulation.
- Increase the number of replicates when Monte Carlo error is large relative to the difference being assessed; report the number of replicates and uncertainty with each estimate.
Joint simulation is flexible and handles complex decision paths, but it depends on the correctness of both the code and the assumed joint model. An analytic multivariate-normal calculation can be more efficient when the boundaries and covariance structure permit it, and can provide an independent check. Bonferroni-type bounds can offer conservative protection without specifying dependence, but may not describe the actual power of a graph. Treating endpoints as independent is simpler, but is appropriate only when independence is justified; otherwise it is a sensitivity scenario, not a default truth.
Simulation is also limited by the scenarios considered. A favorable result over a narrow correlation range does not guarantee performance under unexamined dependence, missingness, or timing patterns. Conversely, very extreme scenarios may be scientifically implausible and should not be presented as equally relevant. Distinguish assumptions supported by trial design or data from deliberately broad stress-test assumptions. If a plausible scenario produces FWER above its target, or if the result is too imprecise to assess the target, investigate the model and rule rather than dismissing the scenario.
- Pitfall: “The global-null estimate is below alpha, so the graph has strong FWER control.”
- Correction: Strong control concerns every true-null configuration. Assess relevant partial-null cases as well, and quantify Monte Carlo error for all estimates.
- Pitfall: “Endpoint correlation affects only the global-null calculation.”
- Correction: Dependence can affect both error rates and power, including the sequence of graph decisions under alternatives.
- Pitfall: “A delayed result can be treated as if it were available at its planned calendar look.”
- Correction: Simulate the actual availability and information schedule; an unavailable statistic cannot trigger a test or graph update.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- In a partial-null scenario, endpoint A is false and endpoint B is true. Which simulated event contributes to FWER?
- A design has \(40{,}000\) replicates and 1,000 FWER events. Calculate its estimated FWER.
- Why is it generally incorrect to simulate statistics at successive looks independently?
- Endpoint B is unavailable at a scheduled calendar analysis. What should the simulation do with B at that analysis?
- A simulation estimates FWER as 0.024 with a Monte Carlo standard error of 0.002 for target \(\alpha=0.025\). Is that result precise enough to establish that FWER is at most the target?
Solutions. (1) A trial contributes to FWER if B is rejected; rejecting false A alone is not an error. (2) The estimate is \(1000/40000=0.025\). (3) Under nested information, successive statistics share accumulated data; under the canonical model their correlation is \(\sqrt{I_1/I_2}\), not zero. (4) Do not test B or apply any B-related graph update until its result is available at the actual prespecified information level. (5) No. The estimate is close to the target, and its standard error is large relative to the 0.001 gap; simulation uncertainty alone makes the result inconclusive.