Why Organize Scenarios Into a Grid?
A simulated operating characteristic describes how a particular design behaves under a particular data-generating scenario. A single scenario can be informative, but it cannot show whether conclusions depend on the assumed treatment effects, control outcome rate, outcome variability, or allocation. A scenario grid makes those assumptions explicit and organizes a set of coherent cases for systematic evaluation.
The aim is not simply to vary every input mechanically. Each grid cell should represent a plausible, clearly specified trial scenario, and the design and analysis rules should remain identifiable across cells. A well-constructed grid can reveal, for example, whether a design meets a familywise error target under null configurations while retaining useful power across several treatment-effect profiles.
For a MAMS design, a scenario must be sufficiently detailed to generate outcomes for all treatment arms and stages. It should state the control outcome distribution, the treatment effect for each arm, the allocation to each arm, and any other data-generating quantities required by the design. The stopping and testing rules are usually held fixed when comparing design performance across scenarios. If those rules change too, the comparison concerns both the scenario and the design.
For continuous outcomes, one possible scenario specification is a control mean \(\mu_0\), an effect vector \((\delta_1,\ldots,\delta_K)\), a common variance \(\sigma^2\), and treatment-to-control allocation ratios \((q_1,\ldots,q_K)\), where \(\mu_j=\mu_0+\delta_j\) and \(q_j=n_j/n_0\). For binary outcomes, a scenario might specify a control event probability \(p_0\) and treatment probabilities \(p_1,\ldots,p_K\). The chosen effect scale must be stated: for example, risk differences \(p_j-p_0\). A grid should not silently treat a risk difference and a relative effect as interchangeable.
Specify the Grid Before Simulating
A useful construction begins by naming the dimensions to vary and listing their levels. The full Cartesian product contains every combination of the listed levels. If there are \(d\) dimensions with \(m_1,\ldots,m_d\) levels, its size is
This count is a planning aid, not a requirement to simulate the full product. Some combinations may be scientifically implausible, mathematically invalid, or irrelevant to the design question. In those cases, define a smaller, explicit set of scenarios and explain the rule for including or excluding combinations. Do not report a truncated grid as if it were the complete Cartesian product.
Treatment effects require particular care because there is one effect for each experimental arm. A compact grid can use profiles such as “all arms null,” “one arm effective,” or “all arms have the same effect.” These profiles are scenarios in their own right. Alternatively, arms can be assigned effect levels independently, but that can quickly create many combinations. The earlier material on partial-null scenarios and alternative treatment-effect configurations provides the relevant vocabulary for describing these profiles.
Worked Example: Counting a Continuous-Outcome Grid
Suppose a hypothetical three-arm design has two treatment arms and a shared control. The investigators want to examine two treatment-effect profiles, two common variances, and two allocation ratios for each treatment arm. They specify the effects as \((0,0)\) or \((-0.30,-0.30)\) outcome units, variances as \(1.00\) or \(1.44\) outcome-units squared, and equal treatment-to-control ratios \(q_1=q_2\) of \(1\) or \(2\).
Check the model and units. Assume independent normally distributed participant outcomes with a common variance within each scenario, control mean \(\mu_0=0\), and treatment means \(\mu_j=\mu_0+\delta_j\). The common-variance assumption is part of this hypothetical data-generating model; the grid compares two values of that variance rather than assuming the values are known in real trials. The allocation ratios specify the intended sample-size ratio for each treatment arm relative to the shared control.
There are \(2\) effect profiles, \(2\) variance levels, and \(2\) allocation settings. Thus
For example, one cell is the null profile \((0,0)\), variance \(1.00\) outcome-units squared, and allocation ratios \(q_1=q_2=1\). Another is the profile \((-0.30,-0.30)\) outcome units, variance \(1.44\) outcome-units squared, and \(q_1=q_2=2\). These are distinct scenarios; results for one do not substitute for results in the other seven.
The effect-profile definition also makes the scope clear: this grid compares a global-null case with a case in which both arms have the same nonzero effect. It does not cover a case where only one arm is effective. If that configuration matters to the design question, it must be added explicitly.
Keep Every Cell Coherent
A Cartesian product is useful only when its combinations make sense. For a binary outcome, probabilities must lie between zero and one. If a grid uses control probabilities \(p_0\) and risk differences \(\delta_j=p_j-p_0\), then each cell must satisfy \(0\leq p_0+\delta_j\leq 1\) for every arm. If the intended scientific assumptions are incompatible with a constant risk difference across control rates, the grid should specify a different effect scale or use a set of carefully selected scenarios.
Worked Example: Checking Binary-Outcome Combinations
Consider a hypothetical binary outcome with control event rates \(p_0=0.20\) or \(0.40\), and treatment risk differences \(\delta=-0.10\) or \(-0.30\). There is one treatment arm, and the effect is defined by \(p_1=p_0+\delta\). The four candidate combinations give:
- \(p_0=0.20,\ \delta=-0.10:\quad p_1=0.20-0.10=0.10\).
- \(p_0=0.20,\ \delta=-0.30:\quad p_1=0.20-0.30=-0.10\).
- \(p_0=0.40,\ \delta=-0.10:\quad p_1=0.40-0.10=0.30\).
- \(p_0=0.40,\ \delta=-0.30:\quad p_1=0.40-0.30=0.10\).
The second combination is invalid because it implies a negative event probability. The other three are valid probability pairs. The original four-cell Cartesian product therefore cannot be used as a set of four binary-outcome scenarios without modification. The investigators could exclude the invalid combination and report a three-cell grid, or choose different effect levels that preserve valid probabilities. They should not silently replace the negative probability with zero: that changes the stated treatment effect and creates a different scenario.
Allocation ratios also need an unambiguous definition. In a shared-control design, say whether \(q_j\) is the ratio of participants in treatment arm \(j\) to participants in the control group, and state whether the ratios apply at each stage or to planned totals. A grid cell is not reproducible if “allocation ratio 2” could mean either twice as many participants in each treatment arm as in control or twice as many participants in total across all treatment arms.
Evaluate and Report Cell by Cell
For each scenario, simulate the same specified MAMS design and record the operating characteristics relevant to the design question. These may include familywise error, power for individual arms, disjunctive or conjunctive power, expected sample size, and trial duration. The definition of each measure and the simulation procedure should be consistent across cells. For example, familywise error is the probability of at least one false rejection under a scenario in which one or more treatment null hypotheses are true; its value can depend on which null hypotheses are true and which effects are present in other arms.
The grid should preserve those distinctions. A null scenario and a partial-null scenario should not be combined into a single “error rate” without saying how the scenarios were weighted and why. Similarly, a mean across scenarios is not a substitute for showing whether a design performs acceptably in a particular important case. A weighted average can be useful when weights represent specified probabilities for future settings, but it answers a different question from performance in each scenario.
Worked Example: Reading a Small Design-Comparison Grid
Suppose two hypothetical candidate designs, A and B, are evaluated in three prespecified scenarios. The table reports simulated disjunctive power, the probability of rejecting at least one treatment null when both arms have the stated beneficial effect, and expected total sample size in participants. For this illustration, assume the effect and control-rate specifications are valid, the same endpoint model and decision rules are used for both designs, and the simulations reproduce the shared-control structure and interim stages.
| Scenario | Control event rate | Effects in arms 1 and 2 | Design A: power | Design A: expected sample size | Design B: power | Design B: expected sample size |
|---|---|---|---|---|---|---|
| 1 | 0.20 | −0.10, −0.10 | 0.78 | 240 | 0.81 | 270 |
| 2 | 0.40 | −0.10, −0.10 | 0.70 | 260 | 0.74 | 290 |
| 3 | 0.20 | −0.10, 0 | 0.76 | 230 | 0.79 | 255 |
In scenario 2, design B has \(0.74-0.70=0.04\), or 4 percentage points, greater disjunctive power than design A. Its expected sample size is \(290-260=30\) participants larger. The table therefore shows a power–sample-size trade-off in that scenario, not an unconditional ranking of the designs.
Scenario 3 has one nonzero treatment effect and one null effect. Its power interpretation is not interchangeable with scenario 1, where both effects are nonzero. In particular, the result for scenario 3 does not by itself give the probability of rejecting the null arm incorrectly. That error probability would need to be estimated and reported as an error-rate operating characteristic for this partial-null configuration.
Choosing the Grid and Checking Its Limits
A full factorial grid is transparent and can expose interactions: for instance, an allocation choice that performs well at one variance may not do so at another. Its cost is that the number of cells grows multiplicatively. A one-factor-at-a-time set, in which one input changes while others remain fixed, is smaller and easier to explain, but can miss interactions between inputs. A short list of representative scenarios is still more compact, but its conclusions apply only to the cases selected. These are complementary ways to organize a comparison, not interchangeable guarantees of coverage.
A practical grid often combines a small set of scientifically motivated profiles with carefully chosen levels of nuisance quantities. Keep the labels descriptive, retain the complete parameter values in the results file, and report any omitted combinations. If the purpose is to average across scenarios, give the weights and their interpretation. If the purpose is to assess performance across cases, show the cell-specific results and identify which cases are most consequential for the design decision.
Common diagnostics for a poorly specified grid include duplicate cells, invalid probabilities, missing treatment arms in effect profiles, and results that cannot be traced back to the parameters used. Check that every scenario has a unique identifier and a complete parameter record. Confirm that the intended number of cells agrees with the listed scenarios, after accounting for any exclusions. Also check that comparisons use the same design rules and outcome definitions. If a result changes unexpectedly between neighboring cells, first verify that the intended parameter changed and that all other settings were held fixed; then investigate the simulation output and implementation.
A scenario grid is a structured evaluation tool, not a guarantee that all future trial conditions have been covered. It cannot correct a misspecified data-generating model, make unrealistic parameter values relevant, or establish performance between grid points without further justification. Its value is that it makes assumptions visible and comparisons repeatable. The introductory material on benchmarking against analytic special cases complements this step: analytic checks can help verify selected simulation results, while a grid organizes the broader range of scenarios to evaluate.
Specify which operating characteristics and design comparisons the evaluation must address.
Write down effect profiles, control rates, variances, and allocation ratios with units and definitions.
Remove or revise invalid and incoherent cells, and document exclusions rather than silently changing inputs.
Use consistent design rules and outcome definitions; interpret results in the scenario where they were generated.
Check Your Understanding
Try each question, then compare your reasoning with the solutions below.
- A grid has three effect profiles, two variance levels, and four allocation settings. How many cells are in the full Cartesian product?
- For a binary endpoint, \(p_0=0.25\) and the treatment risk difference is \(-0.35\). Is this a valid scenario? Show the calculation.
- Why might a one-factor-at-a-time set miss a design feature that a factorial grid could reveal?
- A design has estimated disjunctive power of 0.80 in a scenario where both arms are effective. Does this give its familywise error rate when only one arm is effective? Explain.
Solutions. (1) The full product has \(3\times2\times4=24\) cells. (2) The implied treatment probability is \(p_1=p_0+\delta=0.25-0.35=-0.10\), which is outside \([0,1]\); the scenario is invalid and must be revised or excluded explicitly. (3) A one-factor-at-a-time set changes each input separately, so it may not include combinations where two inputs jointly alter performance; a factorial grid includes such combinations when they are in the product. (4) No. Disjunctive power under two effective arms is not the false-rejection probability for a null arm in a partial-null scenario. That error rate must be evaluated under the relevant partial-null configuration and reported separately.