Why Hold Scenarios Back?
A Bayesian trial decision rule may look well calibrated in the scenarios used to choose it and still perform poorly under a different, clinically plausible set of conditions. For example, a cutoff selected to control false-positive decisions across several null scenarios may fail when the baseline event rate or another data-generating feature differs. A useful check is to reserve scenarios that do not influence the cutoff, apply the already-fixed rule to them, and examine the resulting operating characteristics.
The key distinction is between new simulated trials and new scenarios. Running more replicates under a scenario used during calibration improves the Monte Carlo estimate for that scenario; it does not show that the rule works in a different scenario. Held-out-scenario validation addresses the latter question. It is a check on generalization across the specified scenario set, not proof that the rule will work under every possible data-generating process.
Let \(s\) index scenarios, and let \(c\) denote the cutoff in a decision rule. For instance, the rule might declare a treatment promising when the posterior probability of benefit exceeds \(c\). Write \(D=1\) when the rule declares success and \(D=0\) otherwise. Under scenario \(s\), the operating characteristic of interest is
Under a null scenario, \(\theta_s(c)\) is the probability of a false-positive decision. Under a specified alternative scenario, it is the probability of a successful decision. These are repeated-trial probabilities under the scenario—not posterior probabilities about whether a particular trial’s null hypothesis is true.
Set Up a Genuine Held-Out Check
Before simulation, identify which scenarios will be used for calibration and which will be reserved for validation. The split should be made at the scenario level: a held-out scenario should not influence the candidate cutoffs, their ranking, or a decision about which rule to report. It is reasonable to use clinical and design knowledge to define both sets in advance. The validation set should represent conditions relevant to the intended use of the rule, rather than being chosen after seeing which scenarios make the rule look good.
A typical calibration task might select the least conservative cutoff in a prespecified grid that meets a false-positive-rate target across the calibration null scenarios. “Meets” must be defined in advance: it could mean meeting a requirement in every calibration scenario, or meeting a specified average across scenarios. Those criteria answer different questions. After choosing \(c\), freeze the complete decision procedure—not just the numerical cutoff—and apply it unchanged to the validation scenarios.
A held-out evaluation also needs enough simulated trials to make its Monte Carlo uncertainty interpretable. If \(R_s\) independent simulated trials are run under scenario \(s\), and \(K_s\) lead to \(D=1\), then
The standard error measures simulation noise in estimating the scenario-specific probability. It does not measure uncertainty about whether the chosen scenarios adequately represent future trials or whether the data-generating assumptions are correct. As a rough guide, a normal-approximation interval is \(\hat{\theta}_s(c)\pm1.96SE\{\hat{\theta}_s(c)\}\); near zero or one, or with few decisions, a binomial interval designed for those boundaries is preferable.
Why Separation Helps
Suppose a cutoff is selected after comparing estimated operating characteristics across many calibration simulations. The reported calibration estimate can be optimistic: the selection process may favor a cutoff that happened to look good partly because of simulation noise. A separate validation set gives a direct check that is not used to make that choice.
The basic justification is conditional unbiasedness. Once the calibration data have selected a cutoff \(c\), treat that cutoff as fixed. If validation repetitions are generated independently under a fixed validation scenario \(s\), then each validation decision is a Bernoulli outcome with success probability \(\theta_s(c)\). Therefore,
The conditioning matters: the cutoff was chosen using the calibration data, but the validation repetitions are independent of those data. This argument supports an unbiased estimate of performance under the specified validation scenario. It does not make the validation estimate exact; finite \(R_s\) leaves Monte Carlo error. Nor does it extend to scenarios selected after examining validation results, or to scenarios that indirectly affected the cutoff.
Worked Example: Checking a Null-Rate Target on a New Scenario
Hypothetical trial simulation. Consider a Bayesian binary-outcome trial using the rule “declare benefit if the posterior probability of benefit exceeds \(c\).” A prespecified calibration grid is \(c=0.90,\ 0.95,\ 0.97\). The target is a false-positive rate no greater than 0.05 in each calibration null scenario. The table gives simulated false-positive proportions from two calibration scenarios.
| Cutoff \(c\) | Calibration null A | Calibration null B |
|---|---|---|
| 0.90 | 0.080 | 0.060 |
| 0.95 | 0.045 | 0.035 |
| 0.97 | 0.030 | 0.025 |
Use the prespecified selection rule: choose the smallest cutoff whose estimated false-positive rate is at most 0.05 in both calibration scenarios. At \(c=0.90\), the estimates are 0.080 and 0.060, so the rule fails. At \(c=0.95\), they are 0.045 and 0.035, each no greater than 0.05. Thus the selected cutoff is \(c=0.95\), and it is now fixed.
A third null scenario, excluded from cutoff selection, is simulated 10,000 times using that fixed rule. There are 710 false-positive decisions, giving
A rough 95% Monte Carlo interval is \(0.0710\pm1.96(0.002568)\), or approximately \(0.0660\) to \(0.0760\). The estimated false-positive rate in this held-out null scenario is above the 0.05 target, and the interval is also above 0.05. This is evidence that the calibration result does not generalize to this scenario under the chosen rule. It does not show that the posterior probability in an individual trial was miscalculated; it shows that the repeated-trial decision frequency under this specified condition exceeded the target.
What to Evaluate, and at What Level
Validation should use operating characteristics that match the purpose of the calibration. If the cutoff was chosen to control false-positive decisions, evaluate false-positive rates in held-out null scenarios. If the rule is also intended to achieve a minimum success probability under specified alternatives, evaluate that probability in held-out alternative scenarios too. A favorable result on one measure does not compensate automatically for failure on another; report each prespecified criterion in context.
When the requirement applies scenario by scenario, inspect each held-out scenario separately. A pooled estimate answers a different question: it estimates performance under a particular weighting of scenarios. That distinction is consequential when a few difficult scenarios are hidden by many easier ones.
Worked Example: A Held-Out Alternative Scenario
Hypothetical trial simulation. The cutoff \(c=0.95\) has already been selected using only the calibration set. In a held-out alternative scenario with a specified beneficial treatment effect, 876 of 1,000 simulated trials meet the success rule. The estimated probability of success is
The rough 95% Monte Carlo interval is \(0.876\pm1.96(0.01042)\), or approximately \(0.856\) to \(0.896\). If a prespecified validation goal for this alternative scenario is a success probability of at least 0.80, the estimate is above the goal, and the interval is also above it. This supports the conclusion that the rule meets that goal for this scenario in the simulation. The conclusion is scenario-specific; it is not a guarantee of success in every trial or under every alternative.
Worked Example: Why a Pooled Rate Can Hide a Failure
Hypothetical trial simulation. Suppose two held-out null scenarios are each simulated 5,000 times with the fixed cutoff. Scenario A produces 100 false-positive decisions and scenario B produces 400. The separate estimates are
If the simulations are pooled with equal numbers of repetitions, the pooled rate is
That pooled number meets a 0.05 target exactly, but scenario B has an estimated false-positive rate of 0.080. If the requirement is to control the rate in every scenario, the rule fails the validation criterion. The pooled estimate is not mathematically wrong; it describes an equally weighted mixture of these two scenarios. It simply cannot stand in for the scenario-specific results when the goal concerns each scenario separately.
Diagnostics, Alternatives, and Limits
A useful validation review starts by checking whether the separation was real. Confirm that no held-out result was used to revise \(c\), change the analysis model, adjust the prior, or choose which scenarios to report. Also check that the simulation code implemented the same frozen decision rule in calibration and validation. A discrepancy in code or a post hoc rule change makes the stated validation result difficult to interpret.
Then distinguish a potentially important discrepancy from simulation noise. Compare each estimate with its target and consider its Monte Carlo interval or standard error. An estimate just above a target with substantial uncertainty is not the same evidential situation as an estimate clearly above it. Conversely, increasing the number of repetitions can narrow simulation uncertainty but cannot repair a scenario that was omitted, misspecified, or unrepresentative.
Three approaches answer related but different questions:
| Approach | What it evaluates | Main limitation |
|---|---|---|
| More replicates in calibration scenarios | Monte Carlo precision under scenarios already used to tune the rule | Does not test performance in new scenarios |
| Held-out-scenario validation | Performance of a fixed rule in specified scenarios excluded from tuning | Conclusion is limited to the scenarios and simulation assumptions considered |
| Average across a scenario distribution | Performance under a chosen weighting or distribution of scenarios | A favorable average can conceal poor performance in individual scenarios |
A tempting alternative is to split simulated trial replicates randomly while leaving every scenario represented in both parts. That can provide an independent estimate of performance in those same scenarios, but it is not a test of generalization to a new scenario. Another option is to use a broad scenario grid for calibration and require the rule to meet a worst-case criterion on that grid. This can provide a demanding calibration requirement, but it still does not replace checking the fixed rule under separately reserved conditions.
Held-out validation has important limits. No finite set of scenarios establishes performance over all possible patient populations, trial designs, or departures from the simulation model. The validation scenarios may share structural assumptions with the calibration scenarios, so a result can look reassuring even if both sets omit the same important feature. A failure is informative because it identifies a condition under which the rule did not meet its goal in the simulation; it does not, by itself, identify the cause or prove that the rule will fail in every related setting.
Common Mistakes
- Tuning after looking at validation results. Calling a scenario “held out” does not preserve the check if its result is then used to select a new cutoff. Report that scenario as part of tuning, or reserve a fresh validation set for the revised rule.
- Treating more replicates as more scenario coverage. Ten thousand repetitions under one baseline event rate can estimate that scenario precisely, but they do not represent a different baseline event rate.
- Reporting only an average when the requirement is scenario-specific. State the weighting if the target is an average, and report individual scenario results when the target applies to every scenario.
- Confusing Monte Carlo uncertainty with generalizability. A narrow interval concerns simulation precision for a specified scenario. It does not establish that the scenario set covers the settings where the trial rule might be used.
- Calling a false-positive rate a posterior probability. Under a null scenario, the false-positive rate is the repeated-trial probability that the decision rule declares success. It is not the posterior probability that the null hypothesis is true.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A cutoff is selected using three null scenarios. A fourth scenario is simulated with the fixed cutoff. What makes the fourth scenario a genuine held-out validation scenario?
- In a held-out null scenario, 240 of 4,000 simulated trials declare success. Calculate the estimated false-positive rate and its approximate standard error.
- Two held-out null scenarios have estimated false-positive rates of 0.01 and 0.09. Their equally weighted average is 0.05. If the goal is a rate no greater than 0.05 in every scenario, did the rule meet the goal?
- After observing a held-out false-positive rate above its target, an analyst raises the cutoff and reports the revised rule’s performance on that same scenario. What is the problem, and what should the analyst do?
Solutions.
1. The fourth scenario must have been excluded from decisions about the cutoff and the rest of the procedure. The rule is applied unchanged, using independent simulation repetitions. If its results influenced the cutoff, it has become part of calibration rather than a held-out check.
2. The estimate is \(240/4{,}000=0.0600\), or 6.00%. Its approximate standard error is
Thus the Monte Carlo standard error is about 0.00376, or 0.376 percentage points. It describes simulation uncertainty for this scenario.
3. No. Although the equal-weight average is \((0.01+0.09)/2=0.05\), the second scenario’s estimated rate is 0.09, greater than 0.05. The rule fails a requirement that applies separately in every scenario.
4. The scenario has influenced a revised cutoff, so it can no longer provide an independent validation of that revised rule. The analyst should treat it as calibration evidence and evaluate the revised, frozen rule on a fresh set of scenarios or independent validation simulations not used to make the revision.