Map Performance Across Scenario Grids

Why Map Performance Across Scenarios?

A design comparison is rarely settled by one simulated trial condition. A response-adaptive randomization procedure might perform well when the treatment effect is large and outcomes are stable, yet lose power or allocate patients less evenly when responses are noisy or treatment effects vary across patients. A single scenario can therefore make a design appear generally superior when its advantage is actually conditional.

A scenario grid is a planned set of data-generating conditions formed by varying selected factors across specified values. This tutorial focuses on four factors: average treatment effect, response variability, treatment-effect heterogeneity, and total sample size. The goal is not to simulate every imaginable condition. It is to select a structured set that helps reveal where performance changes and where the ranking of competing designs may reverse.

The earlier material on framing a simulation comparison, specifying data-generating mechanisms, matching comparator conditions, and selecting performance metrics supplies the basic comparison framework. Here, the emphasis is on organizing that framework across a set of scenarios. Every design being compared should face the same scenario, and its performance should be evaluated using the same metric definition.

Define the Scenario Factors

The symbols in the index are compact labels, not a substitute for defining the model. For example, in a continuous-outcome model one might write the potential outcome for participant \(i\) under treatment \(a\in\{0,1\}\) as

Here \(Z_i\) is a measured, centered participant characteristic, and \(\varepsilon_i\) is independent of \(Z_i\) with the stated normal distribution. The treatment effect for a participant with characteristic value \(Z_i\) is \(\tau+\gamma Z_i\). Thus, with \(E(Z_i)=0\), the average treatment effect is \(\tau\), while \(\gamma\) controls how much individual treatment effects vary with \(Z_i\). In this example, \(\sigma\) is residual response variability, not the standard deviation of every outcome pooled across treatment groups. The model is illustrative; other outcome types require their own clearly defined parameters and assumptions.

Choose levels with interpretation in mind. Values of \(\tau\) might represent no effect, a small effect, and a clinically important effect. Values of \(\sigma\) should represent plausible lower, central, and higher residual variability. Values of \(\gamma\) might include no heterogeneity and one or more nonzero levels. Sample sizes should span the range relevant to the design decision. Specify units for \(\tau\) and \(\sigma\), and explain how the heterogeneity parameter is scaled. If the treatment effect is measured in outcome units, then \(\gamma\) in the model above has the same units per one standard-deviation increase in \(Z_i\).

When both \(\tau\) and \(\sigma\) vary, report or inspect their ratio as well as their separate values. For example, \(\tau/\sigma\) is a standardized effect in the continuous-outcome setting just described. A grid that changes \(\tau\) and \(\sigma\) together may otherwise make it hard to tell whether a performance shift is associated with effect size, noise, or both.

Plan the Grid Before Running It

A full factorial grid includes every combination of the specified levels. If the four factors have \(L_\tau,L_\sigma,L_\gamma,L_n\) levels, respectively, the number of scenarios is

A full factorial grid is transparent and supports comparisons that vary one factor while holding the others fixed. It can also become expensive quickly. A fractional or targeted grid includes only selected combinations, for example to concentrate on a plausible region or a suspected ranking boundary. Such a grid can be useful, but its omissions must be explicit: conclusions apply to the conditions simulated, not automatically to every point between or outside them.

A useful plan names each factor, lists its levels and units, and records the rationale for including them. Include a reference condition that reflects the central assumptions of interest. Add boundary or stress conditions when they answer a specific question, rather than adding extreme values merely because they are possible. For each factor, check that changing its level has the intended meaning under the data-generating mechanism.

Compare Designs Within Each Scenario

For a performance metric such as power, coverage, allocation imbalance, or patient benefit, calculate the metric separately for each design in every scenario. A ranking is therefore scenario-specific. It should not be obtained by averaging across scenarios with arbitrary equal weights unless that average is itself the explicitly defined target summary.

For a binary replicate-level outcome, such as whether a test rejects \(H_0\), the estimated operating characteristic in one scenario is \(\hat{p}=K/R\), where \(K\) of the \(R\) replicates meet the specified event. If the replicate outcomes are independent and have a common event probability \(p\), the Monte Carlo standard error is approximately \(\sqrt{\hat{p}(1-\hat{p})/R}\). This describes simulation uncertainty in the estimated operating characteristic; it is not the sampling standard error of an estimate from one clinical trial.

When the goal is to compare two designs, paired replicates can make the comparison more precise: both designs are evaluated using corresponding simulated conditions, and the replicate-level difference is retained. For a metric \(M\), define \(D_r=M_{A,r}-M_{B,r}\) in replicate pair \(r\). The estimated difference is \(\bar{D}=R^{-1}\sum_{r=1}^R D_r\), and its Monte Carlo standard error is \(s_D/\sqrt{R}\), where \(s_D\) is the sample standard deviation of the paired differences. The pairing must be constructed consistently with the simulation design; the two results within a pair need not be identical, but they must correspond to a deliberate shared set of simulated conditions.

Locate and Check Ranking Changes

A ranking change occurs when the estimated difference between two designs changes sign across scenarios. It is more informative to track the difference than to report separate rankings alone. For power, for example, define \(\Delta=\text{power}_A-\text{power}_B\). A positive value favors design A on power; a negative value favors design B. This statement concerns that metric only. It does not imply an overall design preference.

One short justification explains why the sign of the difference is the relevant quantity. If \(\widehat{M}_A\) and \(\widehat{M}_B\) estimate the same performance metric in the same scenario, then

If \(\widehat{\Delta}>0\), then \(\widehat{M}_A>\widehat{M}_B\); if \(\widehat{\Delta}<0\), then \(\widehat{M}_A<\widehat{M}_B\). A ranking reversal across two scenarios requires opposite signs for their estimated differences. But each estimate has Monte Carlo error, so a reversal close to zero may reflect simulation noise. Reporting the difference with its Monte Carlo uncertainty, and increasing replication or refining the grid when needed, distinguishes a stable pattern from a fragile one.

Diagnostics, Alternatives, and Limitations

Inspect the grid as a structured object rather than as a long list of isolated results. Tables can show exact values; heatmaps or line plots can reveal patterns when two factors vary and the others are held fixed. For each display, label the held-fixed factors. A plot of performance against \(n\), for example, should identify the effect, variability, and heterogeneity levels represented by each line or panel.

• Check for confounded factor changes. If \(\tau\) and \(\sigma\) always increase together, their separate effects on performance cannot be distinguished from that grid alone.
• Check monotonic patterns rather than assuming them. Power often rises with sample size under standard conditions, but adaptive allocation, analysis choices, or model violations can produce non-simple patterns for other metrics.
• Check uncertainty near apparent boundaries. A ranking that changes by a fraction of a percentage point may not be distinguishable from Monte Carlo error at the chosen \(R\).
• Check scenario definitions and units. An incorrectly scaled \(\gamma\), an unclear meaning of \(\sigma\), or inconsistent total sample size can make a well-computed grid answer the wrong question.
• Check grid coverage. Large gaps between levels can conceal a narrow region of poor performance or a ranking reversal.

A full factorial design is the clearest approach when the number of factor levels is manageable and interactions matter. A one-factor-at-a-time analysis is simpler to display, but can miss interactions: the effect of changing \(\sigma\), for instance, may depend on \(\tau\) or \(\gamma\). A targeted or fractional grid reduces computational burden, but it may not support conclusions about omitted combinations. A dense grid near a suspected crossover is a useful follow-up, but should not replace the broad initial map if the broader conditions matter.

The grid is only as credible as its data-generating assumptions and implementation. It does not establish which design is best in settings not represented, and no finite set of scenarios proves universal superiority. If rankings depend on a narrow region, interpolation between simulated points needs support rather than assumption. The later material on interpreting trade-offs across design metrics addresses how to combine metric-specific findings; here, the essential output is a transparent map of those findings by scenario.

Common Pitfall: Treating One Ranking as Universal

Another frequent error is to select factor levels after seeing which design they favor, then present the resulting grid as if it had been planned to represent the scientific problem. Define the relevant ranges and reference conditions first, document any follow-up grid as exploratory, and avoid claiming that untested combinations have been covered.