Why Construct a Scenario Matrix?
A design comparison can look decisive under one data-generating model and become less useful when influential assumptions change. For example, a sampling schedule that performs well when concentrations are measured precisely may perform less well under greater measurement error. A scenario matrix makes the conditions of a simulation comparison explicit: each row describes one coherent set of assumptions under which the candidate designs will be evaluated.
The purpose is not to list every imaginable combination. It is to select a manageable set that is relevant to the design decision, spans influential assumptions, and remains interpretable when results are reported. The earlier material on defining simulation questions and performance estimands establishes what the comparison is intended to answer; the matrix specifies the conditions under which that answer will be assessed.
A useful matrix has a defined reference scenario, explicit alternative levels for influential assumptions, and a rationale for including each row. Every row must be sufficiently specified to run the simulation consistently. Assumptions that do not vary across rows still belong in the accompanying model or simulation specification; omitting them from the matrix should not make them ambiguous.
Choose Factors and Levels Before Expanding the Grid
A scenario factor is an assumption or condition intentionally varied across rows. Its levels are the specific values or categories assigned to it. In a pharmacometric design evaluation, possible factors might include a covariate-effect form, a population mix, an assay-error condition, or a dosing condition. These are examples, not a checklist: include a factor when changing it could alter the design comparison or its interpretation.
For each candidate factor, ask three questions. First, is there a substantive reason it could affect the result? Second, are the proposed levels plausible and adequately specified? Third, will a reader be able to explain what a scenario represents from its row and label? Factors that are merely easy to vary, but have no bearing on the design decision, add complexity without adding useful evidence.
A matrix often starts with a reference row: the central or currently preferred set of assumptions. Alternatives can then represent meaningful departures from it. Levels should be stated concretely—for example, a specified covariate relationship rather than simply “different covariate effect.” If a factor has a continuous range, the matrix must state the particular values or categories used; a range by itself does not define a scenario.
The matrix should also respect the data-generating model established in the earlier material on building a data-generating model. A row must combine assumptions in a way that can actually define a coherent simulation. If a particular covariate relationship depends on the population context, or an alternative model requires other settings to change with it, those dependencies should be explicit rather than treated as independent switches.
Worked Example: Counting a Full Factorial Grid
Suppose a hypothetical evaluation considers three factors: covariate-effect form (linear or nonlinear), residual-error condition (lower or higher), and target-population composition (mostly adults or a specified mix of adults and older participants). How many distinct scenarios would a full factorial matrix contain?
The factor-level counts are \(2\), \(2\), and \(2\). A full factorial grid contains every possible combination, so its row count is the product of the level counts:
Thus, eight rows are needed to represent all combinations. This count is not the number of simulation replicates; it is the number of distinct assumption sets. The number of replicates is separately chosen for each scenario and candidate design.
The count grows quickly. If the same evaluation also varied a dosing condition with three levels, the full grid would have \(2 \times 2 \times 2 \times 3 = 24\) rows. That growth is a reason to decide whether every combination is relevant, not a reason to omit a combination automatically. The key question is whether the design decision requires information about that combination.
Balance Coverage and Interpretability
A full factorial matrix is attractive when the combined effects of factors are important and all combinations are coherent. It gives direct coverage of the defined levels, including joint conditions. But it can be unnecessarily large, and some combinations may be implausible or irrelevant. A reduced matrix can be more efficient to interpret, provided its omissions are deliberate and documented.
Three useful building blocks are a reference scenario, one-factor contrasts against that reference, and targeted joint scenarios. The reference anchors the comparison. One-factor contrasts show what changes when a single assumption changes while the others remain at reference levels. Targeted joint scenarios examine combinations that are substantively important—for example, when two adverse conditions may occur together or when an assumption's impact may depend on another assumption.
One-factor-at-a-time scenarios are not a substitute for all combinations when interactions matter. If the effect of changing one factor depends on the level of another, a matrix containing only separate changes may miss that behavior. In that case, include the relevant joint combinations, or use a fuller factorial structure if broad interaction assessment is needed. Do not interpret the absence of a joint row as evidence that joint conditions have no effect.
Worked Example: Building a Focused Matrix
A hypothetical design comparison has a reference condition with a linear covariate effect and lower residual error. The team judges that a nonlinear effect, higher residual error, and their joint occurrence are all relevant. It wants a compact matrix that makes each contrast easy to explain.
| Scenario | Covariate-effect form | Residual-error condition | Reason for inclusion |
|---|---|---|---|
| S0 | Linear | Lower | Reference |
| S1 | Nonlinear | Lower | Isolates the covariate-form change |
| S2 | Linear | Higher | Isolates the error-condition change |
| S3 | Nonlinear | Higher | Assesses the targeted joint condition |
There are \(2 \times 2 = 4\) possible combinations, and the matrix includes all four. The reference is S0. Comparing S1 with S0 changes only the covariate-effect form; comparing S2 with S0 changes only the residual-error condition. S3 is not redundant: it represents the joint condition, which cannot be inferred merely by looking at the two separate contrasts.
If the joint condition had little relevance, the team might instead use only S0, S1, and S2. That three-row matrix would still be interpretable, but it would answer a narrower question: performance under the reference and two separate changes. It would not cover the higher-error, nonlinear combination.
Keep Rows Coherent and Their Meaning Visible
A scenario matrix is useful only if each row represents a scientifically and operationally meaningful condition. A mechanical cross-product can generate combinations that cannot occur, conflict with the model specification, or do not represent the intended population. In those cases, first define the dependency or constraint clearly; then include only admissible combinations and explain why other combinations are excluded.
For example, if a population category is defined by a clinical characteristic that also determines which covariate model applies, treating population and covariate form as fully independent factors may produce incoherent rows. A better matrix could define the relevant combinations as scenario categories, or explicitly specify the applicable model within each population condition. The goal is not to force every column to vary independently, but to make each row complete and defensible.
Use concise scenario identifiers, but do not rely on identifiers alone. Include columns or an associated specification that identify factor levels, fixed settings, and the rationale for each row. When a row differs from the reference on several factors, say why that combination matters. If a factor does not vary, record its fixed setting elsewhere in the simulation specification so that “not varied” cannot be mistaken for “not specified.”
Worked Example: Auditing an Incoherent Combination
A hypothetical evaluation considers two population settings: “adult-only” and “adult plus older participants.” It also considers two covariate-model settings: an adult-only covariate relationship and a relationship explicitly developed for a mixed-age population. A draft full factorial grid has four rows. One row pairs the adult-only population with the mixed-age covariate relationship.
The draft row count is \(2 \times 2 = 4\), but the calculation only counts combinations; it does not establish that all four combinations make sense. In this example, the mixed-age relationship is specified for use in the mixed-age setting, so applying it to the adult-only setting is not an intended or interpretable condition.
The team therefore defines three admissible scenarios: adult-only with the adult-only relationship; mixed-age with the adult-only relationship as a specified extrapolation condition; and mixed-age with the mixed-age relationship. The revised count is \(1 + 2 = 3\) scenarios: one admissible adult-only combination and two mixed-age combinations. The excluded row is documented as outside the intended model conditions, rather than silently dropped.
This revision preserves a meaningful comparison: within the mixed-age population, the two model assumptions can be contrasted, while the adult-only reference remains available. If the adult-only relationship is not scientifically meaningful even as an extrapolation condition, the second mixed-age row should also be removed; the matrix must reflect the question the evaluation can responsibly answer.
Justifying a Reduced Matrix
A reduced matrix is defensible when its rows are selected to answer the stated design question, not simply to minimize computation. A practical construction sequence is to define the reference, identify factors likely to affect the comparison, choose justified levels, enumerate admissible combinations, and then select the combinations needed for direct contrasts or important joint conditions.
The reduction logic can be checked by asking whether each retained row has a purpose and whether each important comparison can be made from the rows included. Suppose a matrix is intended to compare two levels of one factor within a particular level of another. Both rows needed for that comparison must be present. If one row is absent, the intended contrast is not represented.
A compact matrix should not imply a ranking or probability for the scenarios. Unless weights are separately defined for a specific purpose, scenario rows are conditions to examine, not categories with implied frequencies. Scenario selection and the representation of parameter uncertainty are distinct tasks; the latter is treated in the following material. Here, the concern is to specify an interpretable set of fixed conditions for evaluation.
A simple justification is to report the reference, factors and levels, included combinations, excluded combinations, and the reason for those exclusions. This makes the scope of the evaluation visible. It also prevents an attractive result in one row from being mistaken for evidence that all unexamined conditions would behave similarly.
Common Pitfalls, Diagnostics, and Alternatives
| Pitfall | Why it causes trouble | Correction |
|---|---|---|
| Varying many factors without stating why | The matrix becomes large, while its rows may not address the design question. | Retain factors with a plausible connection to the comparison and explain their levels. |
| Using only one-factor contrasts when joint conditions matter | Separate changes do not show performance under a combined condition. | Add targeted joint scenarios or use a fuller factorial matrix when needed. |
| Counting combinations as though all are valid | Some combinations may be incoherent or outside the intended model conditions. | Check admissibility and document constraints before finalizing rows. |
| Using vague labels such as “high risk” | Readers cannot tell what input conditions the label represents. | State the actual level or point to an unambiguous specification. |
| Interpreting untested combinations | The matrix does not provide direct evidence for rows it omits. | Limit conclusions to the conditions evaluated, or add the missing conditions. |
The main alternatives are a full factorial matrix, a reduced reference-plus-contrasts matrix, and a targeted set of scenarios. A full factorial is clearest when all combinations are meaningful and interactions need broad coverage. A reference-plus-contrasts matrix is compact and makes individual changes easy to interpret, but may miss joint behavior. A targeted set can focus on decision-relevant combinations, but requires especially clear justification so that selection does not become arbitrary.
Useful diagnostics are practical checks on the matrix itself: does every row have a unique and interpretable specification? Are any rows duplicates? Are the reference contrasts genuinely one-factor comparisons? Are important joint conditions represented? Can every omitted combination be explained? These checks do not establish that the underlying model is correct; they establish whether the scenario set faithfully represents the intended evaluation.
The matrix also has limits. It cannot compensate for a misspecified data-generating model, make implausible assumptions credible, or support conclusions about conditions it does not include. A larger matrix is not automatically better: additional rows can obscure the main comparisons if they are not tied to a decision. Conversely, a very small matrix may give a false sense of robustness if it covers only minor variations.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A matrix varies three factors with 2, 3, and 2 levels. How many rows are in the full factorial grid?
- Why does a reference scenario plus two separate one-factor alternatives not establish performance under the joint change?
- A draft matrix includes a row that combines assumptions that cannot occur together in the target setting. What should be done before running the simulation?
- When is a full factorial matrix preferable to a reduced matrix?
- A report gives conclusions for a factor combination that was not included in any scenario. What is the problem?
Solutions. (1) The full factorial count is \(2 \times 3 \times 2 = 12\) rows. (2) Each one-factor alternative changes one assumption while holding the other at reference; neither row represents both changes at once, so joint behavior is untested. (3) Check the scientific and model constraints, remove or redefine the inadmissible row, and document the restriction before simulation. (4) A full factorial is preferable when all combinations are meaningful and broad comparison of combinations or interactions is important. (5) The conclusion is unsupported by the matrix; it must be limited to evaluated conditions or the missing combination must be added and evaluated.