Why Calibration Starts With Scenarios
An operating characteristic describes how a trial procedure behaves under specified conditions. The conditions matter: a decision rule may behave differently when the control response is low rather than high, when the true treatment effect is near a clinical threshold rather than large, or when the planned sample size changes. Calibration therefore needs an explicit set of scenarios that says which conditions the assessment is intended to cover.
A scenario matrix is a structured list of combinations of clinical truths and design conditions. Each row represents one combination to assess; columns record the values that define it. The matrix is a planning specification, not a set of trial observations and not a prescription for how to generate trial data. The latter requires a separate data-generating mechanism.
The earlier material on operating characteristics and performance metrics distinguishes the procedure being evaluated from its performance summaries. Here, the focus is on setting out the conditions over which those summaries will be assessed. The trial procedure—including its estimand, analysis, and decision rule—should be fixed or explicitly represented as a design factor; otherwise, differences between rows may be hard to interpret.
Choose the Factors Before Filling the Rows
A useful matrix usually separates three kinds of factors. First are clinical truth factors, such as the treatment effect and the control outcome. Second are design-condition factors, such as total sample size, treatment allocation, or the timing of a planned interim analysis. Third are identifiers and documentation fields, such as a scenario label, the reason for including it, or whether it is a primary or stress scenario.
For a binary outcome where response is beneficial, one convenient parameterization is the control response probability \(p_C\) and the risk difference \(\Delta=p_T-p_C\), where \(p_T\) is the treatment response probability. This parameterization makes treatment-effect values interpretable in absolute percentage points while allowing the control outcome to vary. For continuous outcomes, an analogous choice might specify the control mean and a treatment-minus-control mean difference. The chosen scale should match the clinical question and estimand.
The values should cover a clinically meaningful range, not merely convenient round numbers. Include values that distinguish no treatment benefit, a benefit that would not be clinically sufficient, a benefit near the clinical threshold, and a larger benefit when those distinctions matter to the decision. Include plausible control outcomes because the same absolute treatment effect can imply different treatment response probabilities at different control levels. The range and spacing should be justified by clinical knowledge, prior evidence, or the intended use of the trial—not presented as universal defaults.
Design conditions belong in the matrix when they are part of the intended calibration question. If the procedure will be used at two candidate sample sizes, for example, the matrix can include both. A design factor should not be varied casually: changing sample size and treatment effect together makes it difficult to identify which change accounts for a performance difference.
Worked Example: Crossing Control Response and Treatment Effect
Suppose a hypothetical superiority trial has a beneficial binary response endpoint. The team chooses control response probabilities \(p_C=0.20\) and \(p_C=0.40\), and treatment effects on the risk-difference scale of \(\Delta=-0.05\), \(0\), and \(0.10\). Construct the truth portion of the matrix and check that each combination is possible.
For each row, calculate \(p_T=p_C+\Delta\). For \(p_C=0.20\), the treatment probabilities are \(0.20-0.05=0.15\), \(0.20+0=0.20\), and \(0.20+0.10=0.30\). For \(p_C=0.40\), they are \(0.40-0.05=0.35\), \(0.40+0=0.40\), and \(0.40+0.10=0.50\).
| Scenario | \(p_C\) | \(\Delta\) | \(p_T=p_C+\Delta\) | Interpretation |
|---|---|---|---|---|
| A | 0.20 | -0.05 | 0.15 | 5 percentage-point harm |
| B | 0.20 | 0 | 0.20 | No difference |
| C | 0.20 | 0.10 | 0.30 | 10 percentage-point benefit |
| D | 0.40 | -0.05 | 0.35 | 5 percentage-point harm |
| E | 0.40 | 0 | 0.40 | No difference |
| F | 0.40 | 0.10 | 0.50 | 10 percentage-point benefit |
All six combinations are probability-coherent because each resulting \(p_T\) lies between 0 and 1. The risk difference is measured in absolute probability units: \(\Delta=0.10\) means a 10 percentage-point higher response probability, not a 10% relative increase. These rows define truth conditions; they do not, by themselves, specify the trial's sample size or how outcomes will be generated.
Crossing Design Conditions and Counting Scenarios
A full crossing includes every combination of the selected values for every factor. If there are \(a\) values for one factor, \(b\) for a second, and \(c\) for a third, the full grid contains \(abc\) scenarios. This count is a direct consequence of the multiplication rule: for each of the \(a\) first-factor choices there are \(b\) second-factor choices, and for each pair there are \(c\) third-factor choices.
Here \(k\) is the number of crossed factors and \(m_j\) is the number of selected values for factor \(j\). The formula counts combinations, not the number of simulated trials or participants. Those are separate planning quantities.
Worked Example: Adding Sample Size and Allocation
Extend the six truth combinations above with two total sample sizes, \(n=120\) and \(n=240\), and two treatment-to-control allocation ratios, 1:1 and 2:1. Assume \(n\) is the total number randomized and that allocation is exact.
For \(n=120\), a 1:1 ratio gives 60 participants per group, while a 2:1 ratio gives 80 on treatment and 40 on control. For \(n=240\), the allocations are 120 and 120, or 160 and 80, respectively. Each allocation sums to the stated total.
There are six truth combinations, two sample-size values, and two allocation values, so a complete crossing contains \(6\times2\times2=24\) rows. For instance, one row combines \(p_C=0.20\), \(\Delta=0.10\), \(p_T=0.30\), \(n=120\), and an 80:40 treatment-control allocation. Another combines the same truth values with \(n=240\) and a 160:80 allocation. Holding the truth fixed while comparing these rows isolates the change in design condition; holding design fixed while comparing truth rows isolates the change in clinical truth.
This example assumes that the stated allocation ratios can be implemented exactly at the chosen sample sizes. If they cannot, the planned rounding rule must be specified, and the actual group sizes should be recorded. Likewise, if an interim timing is a design factor, define precisely what the listed value means—for example, a planned interim after a stated number of participants have completed follow-up. Naming a factor without defining its operational meaning leaves the scenario ambiguous.
Use a Full Grid or a Targeted Set?
A full factorial matrix is systematic and makes interactions visible: it shows how performance may change across combinations rather than only across one factor at a time. But it can become large quickly, and the full crossing may include combinations that are clinically implausible or mathematically impossible. A targeted matrix instead selects combinations chosen to answer specific calibration questions, such as behavior under a null effect, around a clinically meaningful threshold, or under a plausible adverse control outcome.
| Approach | Useful when | Main caution |
|---|---|---|
| Full factorial grid | Interactions across selected factors matter and all combinations are meaningful | Can create an unwieldy matrix or implausible combinations |
| One-factor-at-a-time set | A small initial assessment is needed to show isolated changes | May miss interactions between factors |
| Targeted set | Specific boundary, reference, or stress conditions need priority | Coverage can be incomplete unless selection is justified and documented |
These approaches are not interchangeable. A one-factor-at-a-time set can show how a procedure changes with sample size at one chosen truth, but cannot establish that the same change occurs at every treatment effect and control outcome. A targeted set may be more efficient for a focused question, but should identify what it deliberately leaves out. A practical matrix can combine a small core set of clinically central conditions with separately labelled stress scenarios.
Worked Example: Replacing an Incoherent Full Crossing
Suppose an initial proposal crosses \(p_C=0.95\) with \(\Delta=0.10\). The implied treatment response probability is \(p_T=0.95+0.10=1.05\), which is impossible because a probability cannot exceed 1. The full crossing should not retain this row merely because it follows mechanically from the factor lists.
One correction is to choose a clinically justified feasible control value, such as \(p_C=0.80\), for the \(0.10\) risk-difference scenario. Then \(p_T=0.80+0.10=0.90\), which is coherent. This correction changes the scenario, so it should be documented rather than hidden as a computational adjustment.
Alternatively, if \(p_C=0.95\) is essential, the largest possible positive risk difference is \(1-0.95=0.05\). A proposed \(\Delta=0.05\) gives \(p_T=1.00\), a boundary value that is mathematically possible but may be clinically extreme. Whether to include it depends on the calibration question. A targeted matrix can preserve the high-control stress condition while pairing it with feasible effects and explicitly labeling the boundary case.
Construction Checks and Common Failure Modes
Before using the matrix, verify that its rows answer the intended calibration question and that every column is defined. These checks are especially important when a matrix is assembled by combining lists prepared by different clinical and statistical team members.
State the estimand and the trial procedure being assessed, including the decision rule. Record any deliberate procedure variations as factors rather than mixing them into the scenario definitions.
Choose effect and control-outcome values on stated scales. Explain why the values represent relevant reference, boundary, or stress conditions.
Use a complete crossing when all selected combinations are meaningful. Otherwise, list a justified subset and identify the coverage it is intended to provide.
Verify probability bounds, units, allocation totals, and factor definitions. Give each row a stable label and preserve its rationale.
A frequent mistake is to treat every possible Cartesian combination as automatically appropriate. For example, independently selected values for a baseline response and an absolute risk difference can imply an impossible treatment probability. The correction is to calculate the implied outcome values row by row and remove or redesign incoherent combinations with a documented clinical rationale.
Other warning signs include duplicate rows with different labels, missing values that leave a factor's meaning unclear, and a matrix in which the clinically important threshold is absent. A matrix may also be too narrow if it includes only the most likely scenario: calibration is intended to examine performance over a defined range, not to imply that one assumed truth is certain. Conversely, adding many weakly motivated values can make the assessment difficult to interpret without improving its relevance.
A scenario matrix does not establish that the selected range is correct, nor does it guarantee that the procedure performs acceptably between listed values. It is a transparent statement of the scope being assessed. The assessment's conclusions should therefore be tied to the scenarios actually included, and important omissions or uncertain clinical inputs should be acknowledged.
Check Your Understanding
Answer each question, then compare with the solutions.
- A binary-outcome scenario has \(p_C=0.35\) and \(\Delta=-0.08\). What is \(p_T\), and is the combination feasible?
- A full grid crosses four treatment-effect values, three control-outcome values, and two sample sizes. How many scenarios are included?
- Why can a one-factor-at-a-time set miss an important feature of procedure performance?
- At \(p_C=0.92\), what is the largest feasible nonnegative risk difference? What would you check before including that boundary scenario?
Solutions. 1. \(p_T=p_C+\Delta=0.35-0.08=0.27\). It is feasible because 0.27 lies between 0 and 1. 2. The full grid has \(4\times3\times2=24\) scenarios. 3. It varies one factor while holding the others fixed, so it may not reveal interactions; for example, a sample-size change could behave differently at different true effects. 4. The largest feasible value is \(1-0.92=0.08\), giving \(p_T=1.00\). Check whether this boundary value is clinically relevant and whether its inclusion is justified for the intended calibration question.