Constructing Null and Alternative Scenario Grids

Why Scenario Grids Matter

A simulated design performs under the data-generating conditions supplied to the simulation. A result from one set of conditions is therefore not a general statement about how the design behaves. To evaluate an adaptive trial responsibly, investigators specify a set of scenarios, including conditions under which the null hypothesis is true and alternatives representing effects that would matter clinically. They also vary important nuisance parameters, such as a control-group response probability or the dropout rate.

A scenario grid makes these choices explicit. It is not a probability distribution over possible trials, and it does not by itself say which scenario is most plausible. Instead, it is a structured list of conditions under which the fixed simulation and decision rule will be evaluated. The quality of the resulting assessment depends on the scientific relevance, clarity, and internal validity of those conditions.

State the Estimand and Hypotheses First

The grid must follow the scientific question, rather than define it by convenience. Consider a hypothetical two-arm trial with a binary efficacy outcome, where a larger response probability is better. Let \(p_C\) be the response probability in the control arm, \(p_T\) the response probability in the treatment arm, and \(\Delta=p_T-p_C\) the treatment effect on the risk-difference scale. The hypotheses for a one-sided efficacy assessment are \(H_0:\Delta\leq 0\) and \(H_a:\Delta>0\). The clinically relevant effect size is chosen with clinical input; it is not automatically the smallest effect that a simulation can detect.

For this example, assume outcomes are Bernoulli within each arm, with probabilities \(p_C\) and \(p_T\). Participants are randomized as specified by the design, and, when dropout is included, each randomized participant is independently observed with probability \(1-r\), where \(r\) is a common dropout probability in both arms. This simplified model assumes dropout is independent of outcome and treatment assignment. A real simulation should state the model it uses; a grid cannot compensate for assumptions that do not represent the intended trial.

A scenario cell gives values for all varied parameters. Under this model, its treatment response probability is determined by \(p_T=p_C+\Delta\). Thus, the cell is valid only if both probabilities are between 0 and 1, inclusive, and \(r\) is also between 0 and 1. These constraints should be checked for every combination, not just for the parameter levels considered separately.

Building a Grid Systematically

Begin by listing the dimensions that need to vary and the values selected for each. Some dimensions define the treatment effect and distinguish null from alternative scenarios; others describe nuisance features that affect the data but are not the treatment effect of interest. A full factorial grid combines every selected value of each dimension with every value of the others. If there are \(k\) dimensions with \(m_1,\ldots,m_k\) levels, the number of cells is the product of those level counts.

For a grid intended to assess a one-sided efficacy test, label each cell by its actual \(\Delta\): values satisfying \(H_0\) are null configurations, and values satisfying \(H_a\) are alternatives. Include the boundary \(\Delta=0\) when assessing behavior at the null boundary, unless a clear design rationale dictates otherwise. Include negative effects if harm or lack of efficacy in that direction is relevant to the question. State the clinically relevant alternative values in the effect scale being used.

Why Complete Combinations Need Checking

A list of plausible levels is not enough: the combinations created from those levels must also make scientific and mathematical sense. This is particularly important when an effect is defined relative to a baseline probability. A fixed risk difference may imply very different treatment probabilities at different control risks. The constraint \(0\leq p_C+\Delta\leq1\) is a direct check for the binary-outcome example; for other estimands and data models, the corresponding constraints differ.

A full factorial grid is transparent and easy to count, but it may include combinations that are implausible or not meaningful for the scientific setting. If combinations are excluded, record the rule and the reason, and recount the remaining cells. Do not quietly omit an inconvenient combination or label a reduced set a full factorial grid. If the scientific question calls for linked parameter values, define those linked scenarios directly and explain the relationship.

From Grid Cells to Operating Characteristics

Once the cells are specified, the simulation applies the same adaptive design and decision rule under each cell. An operating characteristic is a probability defined under a particular scenario. For example, under a null cell it may be the probability that the trial declares efficacy, while under an alternative cell it may be the probability of declaring efficacy. The interpretation depends on the prespecified decision rule and the scenario’s data-generating conditions.

For a simulated rate, if \(R\) independent simulated trials are run under one cell and \(S\) meet a specified criterion, the estimated rate is \(\hat{q}=S/R\). This estimate describes that criterion under that cell’s conditions; it is not an estimate averaged over the entire grid. Monte Carlo uncertainty in a simulated rate is distinct from the difference between scenario conditions. The earlier material on Monte Carlo error addresses the former; the grid defines the conditions for the latter.

Diagnostics, Alternatives, and Limitations

Before running a large simulation, inspect the grid as data. A scenario table should identify each cell, list all parameter values with units or scales where relevant, and state whether it is null or alternative. Automated checks can confirm the expected cell count, valid parameter bounds, and consistency between effect values and hypothesis labels. These checks are especially useful when grids are assembled by code or edited by several people.

Common warning signs include a cell count that does not equal the product of the specified level counts, a missing null-boundary value, duplicate cells, an invalid implied outcome probability, and labels that conflict with the hypotheses. Also check the effect scale: a risk difference of 0.10 means a ten-percentage-point difference, whereas a relative effect of 0.10 has a different meaning. Confusing these scales can produce a grid that is arithmetically valid but scientifically mislabelled.

A full factorial grid offers broad, even coverage of chosen levels but can become large. A smaller set of selected scenarios can reduce simulation burden, but only if the selection is explicit and still addresses the required null and clinically relevant alternative conditions. Varying one parameter at a time is another possible compact comparison, but it may omit combinations in which two parameter values occur together. These approaches answer different questions; none makes omitted scenarios irrelevant by itself.

The grid is limited by its selected values and by the data-generating assumptions. It cannot establish performance for every possible parameter value, and it does not assign likelihood or importance to scenarios unless a separate framework does so. The immediate goal here is a clear, valid set of conditions for the simulation. Whether the chosen nuisance values adequately represent uncertainty around those inputs requires additional assessment.

Common Mistake

Wrong reasoning: “The control response probabilities, effect sizes, and dropout rates are each plausible, so every combination of them must be a plausible and valid scenario.” This overlooks constraints between parameters and the possibility that a combination has no scientific interpretation.

Correction: Evaluate every complete cell. Calculate any implied parameters, verify all model constraints, and document excluded combinations. Then check that the null/alternative labels follow the stated hypotheses—not an informal label such as “small effect” or “large effect.”