Specifying Data-Generating Scenarios

Why Scenario Specification Matters

Design assurance asks how a proposed trial design would behave under conditions that may occur in practice. The protocol rules determine what the trial does as data accumulate; a data-generating scenario describes the conditions that produce those data. A sound assessment needs both. If the scenario leaves out an important source of variation, or combines incompatible assumptions, the analysis can give a precise answer to the wrong design question.

A scenario is not a prediction that one particular future will occur. It is a structured, hypothetical account of the treatment effects, patient outcomes, enrollment, and other conditions under which the design is to be examined. Scenarios should therefore be clinically interpretable, statistically complete enough to generate trial data, and distinct enough to reveal which conditions matter.

One compact representation is a scenario vector \(\psi\). Its components might include a control event probability, treatment effects, an outcome variance, subgroup prevalence, an outcome correlation, and accrual rates. The trial algorithm is held fixed while the scenario changes. This separation makes it possible to ask whether design performance depends on assumptions about the setting rather than on a change to the design itself.

What a Complete Scenario Specifies

Begin with the clinical question and the trial population. Then specify the distributions that produce patient-level outcomes, including dependence among outcomes or repeated measurements when relevant. State which quantities describe treatment effects and which are nuisance parameters. Finally, describe the time structure: enrollment, outcome delays, follow-up, and any censoring or missingness that can affect the information available to the design.

Treatment effects should be defined on an interpretable scale and tied to a particular population and outcome. For a binary outcome, a risk difference \(\delta\) is the treatment-group probability minus the control-group probability. For a continuous outcome, a mean difference \(\Delta\) is the treatment-group mean minus the control-group mean. These parameters do not fully specify the data distribution: baseline risk, variance, and possibly other distributional features are also needed.

A nuisance parameter is required to define the data-generating process but is not itself the primary treatment effect being assessed. Examples include the control-group event probability, the standard deviation of a continuous outcome, subgroup prevalence, within-patient correlation, and dropout probability. Calling a parameter a nuisance parameter does not make it unimportant: a design can be sensitive to it, especially when it affects the amount or timing of information.

Scenario specifications should also state assumptions about dependence and heterogeneity. For example, are patient outcomes independent conditional on treatment and subgroup, or do outcomes from the same site share additional variation? Is the treatment effect common across subgroups, or does it vary? Such choices change the data that the trial can encounter and must not be left implicit.

Constructing a Scenario Set

A single scenario can answer a narrow question, but design assurance usually requires several scenarios because the true conditions are uncertain. A useful set includes conditions directly tied to the design objective and conditions that test vulnerability to plausible departures. The set might include a reference case, clinically important alternatives, and stress cases for uncertain nuisance parameters. A stress case should still be defensible; an extreme combination with no plausible clinical interpretation can be less informative than a smaller, coherent set.

For a binary endpoint, one may vary baseline probability and treatment effect together. The probabilities must remain valid:

This constraint matters when treatment effects are specified on an absolute scale. For example, \(p_0=0.90\) and \(\delta=0.20\) cannot define a Bernoulli treatment outcome, since the implied \(p_1=1.10\) is not a probability. If the clinical effect is more naturally expressed as a relative effect, derive the treatment probability from that effect and the baseline risk, then check the resulting value is between zero and one.

For a continuous outcome, a scenario needs a distributional assumption and its parameters, not merely a mean difference. Under a normal model, for example, one could specify \(Y\mid A=0\sim N(\mu_0,\sigma_0^2)\) and \(Y\mid A=1\sim N(\mu_1,\sigma_1^2)\). This assumes normal outcomes within each group, with the stated means and variances. Equal variances are an additional assumption, not a consequence of normality. If the outcome is skewed, bounded, or heavy-tailed, the assumed distribution should reflect that feature rather than being chosen only for algebraic convenience.

Accrual, Timing, and Joint Features

Calendar time can change what information is available when the design makes an adaptation. A scenario should describe how patients arrive and, where relevant, how long their outcomes take to become available. A fixed total sample size alone does not capture this feature. In particular, a design that uses interim outcome data may behave differently when accrual is rapid and many outcomes are pending than when enrollment is slow.

One simple accrual assumption is a piecewise-constant arrival rate. If arrivals follow a Poisson process with rate \(\lambda_j\) patients per month during interval \(j\), the expected number enrolled in that interval is its duration multiplied by \(\lambda_j\). This assumption implies independent increments and a constant average arrival rate within each interval. It is a modeling choice; real enrollment may vary by site, season, or operational changes.

Joint outcomes and subgroups require coherent joint specifications. If a scenario includes efficacy and toxicity, specifying each marginal probability is not enough when their association affects decisions. A joint distribution, or a model that determines one, is needed. Likewise, subgroup-specific treatment effects should be paired with subgroup prevalence and outcome distributions; otherwise the overall population mixture is not defined. Earlier material on joint outcomes and heterogeneous effects provides the relevant modeling concepts, but each design-assurance scenario still needs to state the particular values and assumptions being examined.

Justifying and Checking a Scenario

The scenario should be checked as a probability model before it is used. One basic derivation shows why the binary outcome parameters determine both expected event counts and their variability. If \(Y_i\) are independent Bernoulli outcomes with common probability \(p\), then \(E(Y_i)=p\) and \(\operatorname{Var}(Y_i)=p(1-p)\). For \(n\) such patients, \(S=\sum_{i=1}^nY_i\), so linearity of expectation gives \(E(S)=np\). Independence gives

Consequently, a scenario that specifies only an absolute treatment effect but omits baseline risk does not determine expected event counts or their variability. The same risk difference can imply different outcome distributions at different baseline risks.

Before adopting a scenario, check that its assumptions and values are mutually compatible. Useful checks include:

• Probabilities lie between zero and one, variances are positive, and any correlation matrix is mathematically valid.
• Subgroup prevalence values are nonnegative and sum to one; subgroup-specific distributions combine to the stated population distribution.
• Effect measures have a stated direction, time point, outcome definition, and target population.
• Accrual, follow-up, and outcome-delay assumptions align with the trial's calendar-time rules.
• Dropout or missingness assumptions specify whether missingness depends on observed or unobserved outcomes, rather than silently treating all missing data as random.
• Dependence assumptions match the setting; independence is not credible if meaningful clustering or repeated measurements are present and ignored.

A useful diagnostic is to simulate or otherwise inspect data from the proposed distributions before assessing the design. Check whether generated values have the intended ranges, event frequencies, means, variability, subgroup mix, and timing. This is a model check, not evidence that the scenario is true. Implausible generated data, impossible combinations, or a mismatch between the scenario and the protocol's time structure indicate that the specification needs revision.

Alternatives, Limitations, and Common Mistakes

A single “best guess” scenario is easy to communicate but hides uncertainty in the assumptions. A scenario grid makes that uncertainty explicit and permits comparisons across clinically meaningful conditions. The grid should vary a manageable number of factors in a planned way; indiscriminately crossing every possible value can create a large collection of scenarios that are difficult to interpret. Another option is to use empirical data to inform distributions or parameter ranges. Such data can improve realism, but historical or external data may not represent the future trial population, endpoint definitions, or clinical practice. The source and limitations of those inputs should remain clear.

Scenario analysis does not remove uncertainty about which conditions will occur. It only evaluates the design under the specified conditions. Conclusions are limited by the breadth and credibility of the scenario set, by the correctness of the assumed distributions, and by any features left out of the model. A favorable result across a narrow set of scenarios is not evidence of robustness to unexamined conditions.

The practical test is whether another analyst could use the written scenario to generate data without guessing at an important parameter or dependence assumption. If not, the specification is incomplete. Keep scenario assumptions separate from the design's adaptation rules, and keep the rationale for each scenario visible so that results can later be interpreted in clinical rather than purely computational terms.