Worst-Case Operating Characteristics in Bayesian Trials

Why Look for the Worst Case?

A trial design can perform well in a typical setting but poorly when an important nuisance parameter takes an unfavorable value. A single average operating characteristic may hide that weakness. Worst-case analysis instead asks: among the scenarios we have specified, what is the most adverse performance? This is useful when a design must meet a minimum success rate or stay below an upper bound on an undesirable event throughout a defined range of conditions.

In a Bayesian trial, the decision rule may use a posterior probability threshold. Its operating characteristic, however, describes how often that rule reaches a particular decision when trial data are repeatedly generated under a specified scenario. The posterior probability used to make a decision in one trial is not itself that repeated-sampling operating characteristic.

Definition: Let \(\theta\) denote a scenario, including the treatment effect and any relevant nuisance parameters, and let \(\Theta\) be the prespecified set of scenarios under consideration. For a fixed trial design, analysis, and decision rule, an operating characteristic \(g(\theta)\) is a probability or other performance measure calculated under data generation at \(\theta\). For a favorable probability, the worst-case value over \(\Theta\) is \(\inf_{\theta\in\Theta} g(\theta)\). For an adverse probability or loss, it is commonly \(\sup_{\theta\in\Theta} g(\theta)\). If the extremum is attained, these are the minimum and maximum, respectively.

The direction matters. A high probability of meeting a prespecified success criterion is favorable, so its worst case is the minimum. A high false-positive probability is adverse, so its worst case is the maximum. For a continuous measure such as expected loss, the same direction applies: use the largest loss when larger values are worse. Always state the metric and its adverse direction before reporting an extremum.

The scenario set is part of the claim. It might be a finite grid of event rates, treatment effects, accrual patterns, or other specified conditions. A worst-case value on that grid is a guarantee only for the grid scenarios, not automatically for every possible value between them or outside their range. If \(\Theta\) is a continuous set, the infimum or supremum describes its limiting extreme; a minimum or maximum is assured, for example, when \(\Theta\) is compact and \(g\) is continuous.

Estimating an Extreme From Simulation

Suppose the operating characteristic is a probability \(g(\theta)=P_\theta(D=1)\), where \(D=1\) denotes the decision event of interest and \(P_\theta\) is probability under the data-generating scenario \(\theta\). Simulate \(M\) independent trials at each scenario, apply the same prespecified design and decision rule to every simulated trial, and record \(D_m(\theta)\), equal to 1 when the event occurs and 0 otherwise. The estimated probability is the proportion of simulated trials with the event.

\hat{g}(\theta)=\frac{1}{M}\sum_{m=1}^{M}D_m(\theta)

At a fixed scenario, the indicators are independent Bernoulli variables with probability \(g(\theta)\), provided the simulations are independently generated under that scenario and the event is consistently defined. The Monte Carlo standard error (the simulation error in estimating the probability) is estimated by

\widehat{SE}\{\hat{g}(\theta)\} = \sqrt{\frac{\hat{g}(\theta)\{1-\hat{g}(\theta)\}}{M}}.

For a finite scenario set \(\Theta=\{\theta_1,\ldots,\theta_K\}\), estimate a favorable probability's worst case by \(\min_{1\leq j\leq K}\hat{g}(\theta_j)\), and estimate an adverse probability's worst case by \(\max_{1\leq j\leq K}\hat{g}(\theta_j)\). This calculation identifies the most adverse estimated value among the simulated scenarios. It does not, by itself, establish that the true extremum over a larger continuous range has been found.

Worked Example: Lowest Probability of Trial Success

A hypothetical Bayesian trial uses a fixed decision rule: declare success if the posterior probability that the treatment effect is beneficial exceeds 0.95. The design team defines success, for calibration purposes, as declaring success when the true effect is beneficial. It simulates \(M=10{,}000\) trials at each of four prespecified beneficial-effect and nuisance-parameter scenarios. The success counts are 7,400, 8,100, 6,800, and 7,600.

The estimated success probabilities are \(0.740\), \(0.810\), \(0.680\), and \(0.760\), since each count is divided by 10,000. For example, at the third scenario, \(\hat{g}(\theta_3)=6{,}800/10{,}000=0.680\). The lowest estimated success probability on this four-scenario set is therefore \(0.680\), at \(\theta_3\).

At that scenario, the estimated Monte Carlo standard error is

\widehat{SE} = \sqrt{\frac{0.680(1-0.680)}{10{,}000}} = \sqrt{0.00002176} \approx 0.004665.

Thus the simulation estimates a 68.0% chance of declaring success at the least favorable of these four scenarios. The standard error describes simulation variability at that fixed scenario, not uncertainty about whether the four scenarios cover all relevant conditions. A pointwise normal approximation gives \(0.680\pm1.96(0.004665)\), or approximately \(0.671\) to \(0.689\). Because the scenario was selected for having the smallest estimate, this pointwise interval should not be treated as a simultaneous uncertainty interval for the selected extreme.

What a Worst-Case Guarantee Means

The worst-case calculation has a simple logical interpretation. If \(\gamma\) is a required minimum favorable probability, then a design meets that requirement on \(\Theta\) exactly when its operating characteristic is at least \(\gamma\) at every scenario in \(\Theta\). Equivalently, its infimum over \(\Theta\) is at least \(\gamma\). Likewise, if \(r\) is a maximum tolerable adverse probability, the requirement is that the probability be no greater than \(r\) at every scenario, equivalently that its supremum be no greater than \(r\).

\inf_{\theta\in\Theta} g(\theta)\geq\gamma \quad\Longleftrightarrow\quad g(\theta)\geq\gamma\ \text{for every }\theta\in\Theta

For a finite set, the justification follows directly from the definition of a minimum. If the minimum is at least \(\gamma\), no scenario's value can be below \(\gamma\). Conversely, if every scenario's value is at least \(\gamma\), their minimum must be at least \(\gamma\). The corresponding argument for a maximum and an adverse-probability upper bound is the same, with the inequalities reversed as appropriate.

This guarantee is conditional on the scenario set, design, analysis, and decision rule being the ones actually assessed. In simulation, it is also an estimate unless Monte Carlo uncertainty has been accounted for in the decision about whether the requirement is met. A point estimate just above a target is not strong evidence of a guaranteed threshold if simulation error could plausibly place the true value below it.

Worked Example: Worst-Case False-Positive Probability

Consider a hypothetical Bayesian trial whose rule declares treatment benefit when the posterior probability of a beneficial effect exceeds 0.95. To assess false positives, the design team simulates under a null treatment effect at three prespecified combinations of baseline event risk and accrual conditions. In \(10{,}000\) simulated trials per scenario, the numbers of declarations of benefit are 400, 700, and 300.

The estimated false-positive probabilities are \(400/10{,}000=0.040\), \(700/10{,}000=0.070\), and \(300/10{,}000=0.030\). Because false positives are adverse, the worst-case estimate is the maximum, \(0.070\), at the second scenario. Its estimated Monte Carlo standard error is

\widehat{SE} = \sqrt{\frac{0.070(1-0.070)}{10{,}000}} = \sqrt{0.00000651} \approx 0.002551.

The estimated worst-case false-positive probability is 7.0% among the three scenarios assessed. If the design requirement were a false-positive probability no greater than 5% at every assessed scenario, the point estimates would fail it. The conclusion concerns this particular null scenario set; it does not establish the maximum over unexamined baseline risks or accrual conditions.

Joint Requirements and Practical Checks

A design may need to meet more than one operating-characteristic requirement. For example, it may need a minimum success probability under beneficial effects and a maximum false-positive probability under null effects. These are separate extrema over their respective scenario sets. A favorable result on one criterion does not compensate for failure on the other unless the protocol has explicitly defined a combined criterion.

Worked Example: Checking Two Design Requirements

A hypothetical design is assessed using 20,000 simulated trials per scenario. For three beneficial-effect scenarios, the success counts are 15,600, 14,600, and 13,800. For three null scenarios, the false-positive counts are 640, 940, and 1,020. The prespecified requirements are a success probability of at least 0.70 at every beneficial-effect scenario and a false-positive probability of at most 0.05 at every null scenario.

The success estimates are \(15{,}600/20{,}000=0.780\), \(14{,}600/20{,}000=0.730\), and \(13{,}800/20{,}000=0.690\). The worst-case estimate is the minimum, \(0.690\); it is below 0.70 by 0.010. Thus the simulated point estimates do not meet the minimum-success requirement across the set.

The false-positive estimates are \(640/20{,}000=0.0320\), \(940/20{,}000=0.0470\), and \(1{,}020/20{,}000=0.0510\). Their maximum is \(0.0510\), exceeding the 0.05 limit by 0.0010. The point estimates therefore also fail the false-positive requirement. Both results should be reported: each identifies the metric, its adverse direction, the extreme value, and the scenario at which that value was estimated.

Before interpreting a worst-case result, check the conditions that make it meaningful:

Scenario set: The set should be prespecified and justified by the scientific and operational range of conditions the design is meant to address. An omitted adverse scenario cannot be protected by the reported minimum or maximum.
Consistent simulation: Each scenario must generate data according to its specified assumptions, and the same analysis and decision rule must be applied across scenarios. A coding or implementation difference can look like a performance difference.
Monte Carlo precision: Increase the number of replicates where an estimate is close to a design threshold or where the extreme is unstable. Selecting the smallest or largest among noisy estimates can make the selected estimate appear more extreme than its underlying value.
Grid coverage: A coarse grid can miss an interior extremum or sharp change. Examine plausible boundaries and refine the grid near regions where estimates change quickly or where the current extreme occurs.

Worst Case Versus Performance Averaged Across Scenarios

A worst-case value and an average answer different questions. The worst case asks whether performance is acceptable even at the most adverse scenario in a defined set. An average combines performance across scenarios and therefore depends on how scenarios are weighted. Without weights, there is no unique scenario-average performance to report; conversely, a worst-case calculation does not say how frequently the adverse scenario should be expected in practice.

This distinction is useful when communicating the purpose of calibration. A minimum success probability supports a statement about performance throughout the assessed set, subject to simulation and scenario coverage. An average supports a statement about performance under a specified distribution or weighting of scenarios. Neither summary replaces the other, and an average above a target cannot establish that every scenario meets that target.

A common mistake is to calculate the mean of scenario-specific success probabilities and describe it as the “worst-case” success rate. The correction is to identify the decision question first: use the minimum for a guaranteed lower bound on a favorable probability, and use the maximum for a guaranteed upper bound on an adverse probability. Also name the scenario set and report where the extreme occurred. Do not claim protection outside that set.

Key takeaway: Worst-case operating characteristics are extrema over an explicitly defined scenario set. A minimum favorable probability or maximum adverse probability supports a robust requirement across that set; it is not an average, and it is not a guarantee for scenarios that were never assessed.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

A favorable decision occurs in 720 of 1,000 simulations at one scenario. What is the estimated operating characteristic?
For an adverse event, three scenario-specific estimates are 0.018, 0.031, and 0.026. Which value is the worst case, and why?
A favorable-probability estimate is 0.64 at every point of a five-point grid. What can and cannot be concluded about a continuous range containing that grid?
Explain why a scenario-average success probability above 0.70 does not show that the success probability is at least 0.70 at every scenario.

Solutions. (1) The estimate is \(720/1{,}000=0.720\), or 72.0%. (2) The worst case is \(0.031\), because a higher adverse-event probability is worse; it occurs at the second scenario. (3) The estimated minimum on the five-point grid is 0.64. This does not establish the minimum over the continuous range, because an unexamined value between grid points could be lower. (4) An average can exceed 0.70 even when one or more scenario-specific probabilities are below 0.70; only checking the minimum across the relevant set addresses whether every scenario meets that threshold.