Diagnosing Poor Performance by Scenario

Why Diagnose Performance by Scenario?

An overall operating-characteristic summary can conceal where an adaptive trial design performs poorly. For example, a design might have acceptable average power across a set of assumptions, yet low power when recruitment is slow or a key nuisance parameter takes an unfavorable value. Diagnosing performance by scenario means examining the simulated results separately for each specified combination of assumptions, then asking which scenarios are responsible for estimates that warrant concern.

The purpose is to localize a warning, not to explain which design component caused it. A scenario is a set of assumed conditions under which the design is evaluated; it is not a randomly sampled group of patients. The scenario-specific probability describes the behavior of the design under those conditions. The simulation estimate also has Monte Carlo uncertainty because it is calculated from a finite number of simulated trials. Both facts matter when interpreting a concerning result.

The event must be stated precisely before results are inspected. It might be a successful trial, a false-positive conclusion under a null scenario, or meeting a prespecified safety rule. Each scenario is assessed against the relevant target for that event. For instance, a false-positive rate may have an upper limit, while power may have a lower limit. A result that is concerning for one operating characteristic is not automatically concerning for another.

Monte Carlo Uncertainty Within a Scenario

If simulated trials are independent and each has the same probability \(\theta_s\) of the event within scenario \(s\), then \(K_s\) has a binomial distribution with parameters \(R_s\) and \(\theta_s\). The estimated proportion has estimated Monte Carlo standard error \(SE_{\mathrm{MC}}=\sqrt{\hat{\theta}_s(1-\hat{\theta}_s)/R_s}\). This measures simulation error in estimating \(\theta_s\); it does not measure uncertainty about whether the scenario assumptions are correct, or uncertainty in the clinical effect or nuisance parameters within that scenario.

When the estimated event and non-event counts are not small, a useful approximate 95% Monte Carlo interval is \(\hat{\theta}_s\pm1.96SE_{\mathrm{MC}}\). It summarizes the precision of the simulation estimate under the assumed simulation model. It is not a probability statement about the fixed value \(\theta_s\). When counts are small, especially for rare events or estimates near zero or one, a normal interval can be inaccurate; a binomial interval with better boundary behavior is preferable.

Locating Warnings Without Overreacting to Noise

A practical diagnostic begins by organizing results by scenario and operating characteristic. For each scenario, record the event count, number of replicates, estimate, Monte Carlo standard error or interval, and the relevant target. This makes it possible to see both the size of a potential problem and how precisely it has been estimated. The earlier material on worst-case operating characteristics and scenario weighting distinguishes worst-scenario performance from average performance; scenario-level diagnostics supply the detail needed to see which assumptions are associated with a warning.

A short derivation explains why the simulation standard error takes its stated form. Each \(Y_{sr}\) is Bernoulli with variance \(\theta_s(1-\theta_s)\). Independence gives the variance of their average as

Replacing the unknown \(\theta_s\) by its estimate gives the estimated variance \(\hat{\theta}_s(1-\hat{\theta}_s)/R_s\), and taking its square root gives \(SE_{\mathrm{MC}}\). This derivation depends on independent replicates with a common event probability within the scenario. If runs are dependent, or the simulation procedure changes across replicates, this binomial formula need not describe the uncertainty.

Comparing Scenarios

Contrasting scenario estimates can help prioritize follow-up work. When simulation batches for two scenarios are independent, the estimated difference \(\hat{\theta}_A-\hat{\theta}_B\) has estimated standard error \(\sqrt{SE_{\mathrm{MC},A}^2+SE_{\mathrm{MC},B}^2}\). This quantifies Monte Carlo uncertainty in the contrast. A large difference relative to this standard error supports a clear distinction between the simulated probabilities, but it does not establish that the changed scenario condition caused the difference in a real trial.

This calculation assumes independent batches. If the simulation uses shared random numbers to pair runs across scenarios, the estimates may be correlated; then the variance of their difference includes a covariance term. Using the independent-batch formula in that setting can misstate uncertainty. Record how simulation runs were generated before making scenario contrasts.

Diagnostics, Alternatives, and Limitations

A common mistake is to sort scenarios by their point estimates and call the lowest-power or highest-error scenario a definite failure. That reasoning ignores that the most extreme estimate among many scenarios can be extreme partly by chance. The correction is to inspect uncertainty alongside estimates, consider whether the apparent concern is large enough to matter in context, and avoid treating a noisy ranking as definitive. If several scenarios are screened, a focused follow-up simulation can increase the number of runs for the concerning scenarios. This improves Monte Carlo precision, though it does not address uncertainty in the scenario inputs themselves.

Other failure modes include mixing different event definitions in one comparison, using an overall average to hide a poor subgroup of scenarios, and applying the binomial standard error when replicates are dependent. Check that each row refers to a clearly defined scenario, target, and event; verify counts against the number of runs; and confirm the simulation design supports the uncertainty calculation. For rare events, very small counts can make normal-based intervals unreliable. Do not infer that the event is impossible merely because no event occurred in a finite simulation.

A pooled summary is useful for an average-performance question but can obscure a vulnerable scenario. A worst-case summary emphasizes the most unfavorable specified scenario but does not reveal how stable that ranking is or how close other scenarios are. Scenario-stratified estimates with uncertainty complement both summaries: they show where concern arises and how much Monte Carlo evidence supports it. They remain limited to the scenarios actually simulated, the specified model, and the fidelity of the simulation code.