Building a Scenario Matrix for Design Evaluation

Why a Scenario Matrix Is a Design Tool

A simulation can generate many trial paths, but those paths answer a design question only after the conditions that generate them have been specified. A scenario matrix makes those conditions explicit. It is not a table of simulation results: it is a structured plan for which combinations of assumptions the design will face.

For a group sequential design, relevant conditions can include the treatment effect, nuisance parameters such as outcome variability, the pattern of accrual and loss to follow-up, and assumptions used by the analysis. The primary scenario represents the planning conditions on which the design is principally based. Other scenarios examine plausible alternatives or stress the design under adverse conditions. Keeping these roles distinct prevents a favorable planning case from being mistaken for evidence that the design performs similarly under every condition.

The scenario matrix should be consistent with the design specification and software inputs. Each row needs enough detail that a different analyst could implement the same scenario without guessing. In particular, distinguish the assumptions that generate trial data from the assumptions supplied to the analysis. They may match in the primary case and differ in a deliberate stress test.

Specify the Axes Before Filling the Matrix

Begin with the design held fixed: for example, its endpoint, planned information times, maximum information, stopping rule, and target error-rate convention. These are not automatically scenario factors. Vary a design feature only when the evaluation question asks how performance changes if that feature changes. Otherwise, keep it constant so that differences across rows can be attributed to the assumptions that were intentionally varied.

For each scenario factor, define its meaning, units, and selected levels. A useful matrix commonly considers the following:

• Treatment effect: include the null effect when evaluating behavior under no treatment difference, the design alternative used for planning, and possibly a smaller or otherwise relevant alternative. State the effect on its analysis scale, such as a mean difference in outcome units or a log hazard ratio.
• Nuisance parameters: specify quantities that affect the distribution or precision of the endpoint but are not the treatment effect, such as a standard deviation, event rate, or control-group outcome rate. Use levels with a substantive basis, not merely convenient round numbers.
• Accrual and follow-up: specify the recruitment pattern, its duration, and operational features such as delayed enrollment or loss to follow-up. For time-to-event outcomes, distinguish calendar-time accrual from the event process and follow-up.
• Analysis assumptions: specify analysis inputs and methods relevant to the design, including any assumed nuisance value or planned handling of incomplete data. If the analysis assumption differs from the data-generating condition, record both explicitly.

The matrix is more informative when every factor has a clear contrast. If the question is whether slower accrual matters, for example, use a reference accrual pattern and a defined slower pattern while keeping other factors fixed in that comparison. Avoid changing several factors in one row unless the combination is intentional and its purpose is stated.

Use a Full Grid as a Starting Point, Not a Requirement

If all levels of all factors are crossed, the result is a full factorial grid. When factor \(j\) has \(k_j\) levels, the number of rows is the product of the level counts:

This count follows because each choice for the first factor can be paired with every choice for the remaining factors; repeating that multiplication across the factors gives the product. A full grid can reveal interactions, but it can also include implausible combinations and use substantial computational effort on combinations that do not address the planning question. A reduced matrix is acceptable when its omissions and selection logic are explicit.

Distinguish Primary, Alternative, and Stress-Test Scenarios

Labels should convey the role of a scenario, not an unsupported judgment about its probability. “Primary” identifies the reference planning assumptions. “Alternative” can identify a credible different assumption, such as a smaller treatment effect. “Stress test” signals a purposeful challenge, such as higher variability combined with slower accrual. A stress test need not represent the most likely future trial; it should represent a condition worth checking and be plausible enough to interpret.

For an auditable matrix, each row should record at least:

• a unique scenario identifier and category;
• the value of every varied factor, including units and parameter scale;
• the data-generating assumptions and the analysis assumptions, separately;
• the fixed design features inherited by the row; and
• a short rationale stating the question the row addresses.

A rationale might say, “Tests the reference design under a 20% larger standard deviation,” or “Checks the planned analysis model when the generated outcome variability is higher.” Avoid labels such as “worst case” unless the direction and scope of “worst” are defined. A scenario can be adverse for one operating characteristic and not another.

A Practical Construction and Review Sequence

A disciplined sequence reduces ambiguity before simulation code is run. It also makes it easier to check whether each row answers a distinct question rather than adding complexity without purpose.

This sequence does not prescribe a universal number of rows. The right matrix is the smallest documented set that covers the primary planning case and the important uncertainties or operational challenges relevant to the design decision. If interactions matter, include joint scenarios to examine them; if they do not, avoid implying that a sparse set of one-factor contrasts has assessed them.

Diagnostics, Comparisons, and Limitations

Review the matrix for structural problems before interpreting any simulation output. A matrix is likely to be misleading if the primary scenario is not identifiable, a factor changes without being recorded, or two rows intended as a single-factor contrast differ in several ways. Check that units and parameter scales are consistent—for example, a standard deviation must not be entered in one row as points and another as a variance without an explicit conversion.

A common alternative to a scenario matrix is to evaluate only the primary assumptions. That approach is compact, but it cannot show how conclusions depend on assumptions that are uncertain or operationally variable. At the other extreme, a full factorial grid covers all specified combinations but may be computationally expensive and may include combinations that are implausible or irrelevant. A purposeful reduced matrix offers a middle ground, provided its limited coverage is stated honestly.

A further alternative is to vary one factor at a time around the primary scenario. This makes individual contrasts easy to interpret, as in P1 versus V1. However, it does not assess interactions: higher variability and slower accrual together may have consequences not indicated by either change alone. Add joint stress scenarios when such combinations are plausible and important. Do not infer that the design is robust across unexamined combinations.

A scenario matrix also cannot eliminate uncertainty about whether its assumptions are realistic. Chosen levels may omit an important condition, and labels such as “plausible” or “adverse” require substantive justification. The matrix organizes evaluation; it does not by itself establish design adequacy, quantify Monte Carlo uncertainty, or guarantee that results will transfer to a future trial.

• Mistake: treating every combination in a stress-test row as the primary planning assumption. Correction: label the primary row and each stress-test purpose separately.
• Mistake: changing the effect, variability, and accrual together, then attributing a difference to variability. Correction: use a one-factor contrast for that attribution, or describe the row as a joint challenge.
• Mistake: calling a sparse subset “the full scenario range.” Correction: list its covered contrasts and state which combinations or interactions were not examined.