Why Operating Characteristics Matter
A Bayesian decision rule is applied to the data observed in a particular trial. For regulatory review, it is also important to ask how that rule would behave across repeated trials under different assumptions about the true treatment effect, event rate, enrollment, and other features of the setting. Operating characteristics answer that question. They help reviewers see the consequences of applying the proposed rule when treatment is ineffective, when it has a clinically relevant effect, and when conditions differ from the central design assumptions.
An operating characteristic is not the posterior probability calculated from one trial. It is a probability over hypothetical repeated trials, conditional on a specified data-generating scenario and the trial procedure. For example, a posterior probability of benefit of 0.97 describes uncertainty about the treatment effect conditional on one dataset and the model. A simulated false-positive rate of 0.06 describes how often the procedure would declare success across trials simulated under a specified no-benefit scenario.
To make these quantities interpretable, the scenario, decision, and target population must be explicit. If \(\theta\) denotes the treatment-effect parameter, a no-benefit scenario might be \(\theta=0\), while a clinically important benefit scenario might be \(\theta=\theta_1\). For a procedure that declares success when a prespecified posterior-probability criterion is met, define \(D=1\) when it declares success and \(D=0\) otherwise. Then the false-positive rate under a no-benefit scenario is \(P(D=1\mid\theta=0)\), and power at \(\theta_1\) is \(P(D=1\mid\theta=\theta_1)\). These are repeated-sampling probabilities; they are not posterior probabilities that a claim is true.
Choose the Scenario Before Summarizing Performance
An operating characteristic is always conditional on the assumptions used to generate the hypothetical trials. Those assumptions should be chosen to represent the regulatory question and the intended use of the design. Relevant scenarios may include no treatment effect, a clinically important effect, and intermediate effects; they may also vary the control event rate, variability, enrollment, or other influential features. The simulation plan should specify how each scenario is generated and how the fixed analysis and decision procedure are applied. The earlier material on simulation plans that support design review covers that planning process; here the emphasis is on selecting and interpreting the summaries.
A false-positive rate requires a definition of “no benefit.” For a parameter with a point null, this may be the rate when the true effect is zero. In some clinical settings, a range of effects may be considered too small to support a claim. If so, reviewers should be told which values belong to that no-benefit region and whether the reported rate is evaluated at a boundary or averaged over a distribution of values. Similarly, power is not one universal property of a design: it is the probability of a positive decision at a specified effect and other specified scenario values.
At minimum, a useful report states the scenario, the number of simulated trials, the decision event counted, the estimate, and its Monte Carlo uncertainty. A single value at one assumed effect can hide performance at nearby values. A small table across a grid of effects is often more informative than a single headline number.
Estimating a Decision Probability
Suppose \(M\) independent trials are simulated under one fixed scenario, and let \(D_m=1\) if trial \(m\) meets the prespecified success rule and \(D_m=0\) otherwise. The estimated probability of success is the fraction of simulated trials meeting that rule. Because this is an estimate from a finite simulation, it has Monte Carlo error even if the simulation code and model are correct.
The standard error follows because each \(D_m\) is a Bernoulli variable with success probability \(q\), so its variance is \(q(1-q)\). The mean of \(M\) independent indicators has variance \(q(1-q)/M\); replacing the unknown \(q\) by \(\hat{q}\) gives the estimated Monte Carlo standard error. A rough 95% Monte Carlo interval is \(\hat{q}\pm1.96SE_{\mathrm{MC}}\). This interval describes simulation precision, not uncertainty about the treatment effect in a future trial. For estimates near zero or one, or with few simulated successes or failures, the simple normal approximation can be poor; an appropriate binomial interval or more simulation replicates may be preferable.
Worked Example: Reporting a False-Positive Rate and Power
Consider a hypothetical binary-outcome trial with a fixed sample size of 100. The analysis uses a \(\mathrm{Beta}(1,1)\) prior for the treatment event probability \(p\), and declares success if the posterior probability \(P(p>0.20\mid X)\) is at least 0.95. Suppose 1,000 independent trials are simulated under each of two scenarios: \(p=0.20\), representing no benefit relative to the prespecified boundary, and \(p=0.30\), representing a beneficial scenario. Assume the simulation produces 47 successes under \(p=0.20\) and 812 under \(p=0.30\).
Under the no-benefit scenario, the estimated false-positive rate is \(47/1000=0.0470\), or 4.70%. Its estimated Monte Carlo standard error is \( \sqrt{0.0470(1-0.0470)/1000}=\sqrt{0.00004479}=0.006692 \), or 0.669 percentage points. The rough 95% Monte Carlo interval is \(0.0470\pm1.96(0.006692)\), giving approximately 0.0339 to 0.0601. The result is therefore an estimated false-positive rate of 4.70%, with simulation precision corresponding to a rough interval of 3.39% to 6.01%.
Under \(p=0.30\), estimated power is \(812/1000=0.812\), or 81.2%. Its Monte Carlo standard error is \( \sqrt{0.812(1-0.812)/1000}=\sqrt{0.0001527}=0.01236 \), or 1.236 percentage points. The rough 95% Monte Carlo interval is \(0.812\pm1.96(0.01236)\), approximately 0.7878 to 0.8362. Thus the estimated probability of meeting the success rule at \(p=0.30\) is 81.2%; this is not the probability that the treatment is effective after observing a trial.
These summaries apply only to the stated prior, analysis, rule, sample size, and data-generating values. They do not establish performance at other values of \(p\), or under different event rates or data-generating assumptions.
Expected Sample Size in Adaptive Designs
When sample size can vary across trials, reporting only the maximum planned enrollment does not describe typical resource use. Let \(N_m\) be the final sample size in simulated trial \(m\). Under a specified scenario, the estimated expected sample size is the average across simulated trials. Its interpretation is the long-run average enrollment if the same design were run repeatedly under that scenario; it is not a promise about the size of any particular trial.
where \(s_N\) is the sample standard deviation of the simulated final sample sizes. The standard error quantifies Monte Carlo precision in the estimated mean. Because a mean alone can conceal the distribution of enrollment, also consider reporting the median, selected quantiles, and the probability of reaching the maximum, especially when enrollment is highly variable. State whether sample size counts randomized participants, participants with observed outcomes, or another operational quantity.
Worked Example: Summarizing Enrollment Under an Adaptive Rule
In a hypothetical adaptive design, 1,000 simulated trials under one specified scenario have final sample sizes of 60 for 180 trials, 90 for 520 trials, and 120 for 300 trials. Assume these numbers count randomized participants and that 120 is the design maximum.
The estimated expected sample size is \( \{180(60)+520(90)+300(120)\}/1000 \). The numerator is \(10{,}800+46{,}800+36{,}000=93{,}600\), so the estimated mean is 93.6 participants per trial. The median is 90: 180 trials are at 60, and the cumulative count reaches 700 at 90, which contains the 500th observation. The probability of reaching the maximum is \(300/1000=0.300\), or 30.0%.
These three summaries answer different questions. The mean describes average enrollment, the median describes the center of the simulated distribution, and the maximum-sample probability describes how often the design uses its full enrollment. Reporting only the maximum of 120 would overstate typical enrollment; reporting only the mean would hide the 30.0% chance of reaching that maximum. For a full report, calculate the Monte Carlo standard error of the mean from the trial-level sample sizes, using \(s_N/\sqrt{1000}\); the grouped counts alone can be used to compute \(s_N\) by expanding or weighting the values.
Compare Performance Across Scenarios
Operating-characteristic summaries are most useful when read together. False-positive rate describes behavior in a no-benefit scenario; power describes behavior at a beneficial effect; expected sample size describes resource use. A design may have acceptable performance for one summary and less favorable performance for another. A scenario table can make these trade-offs visible and show whether performance changes sharply as the true effect moves from no benefit toward a clinically important benefit.
Worked Example: Reading a Scenario Grid
Suppose a hypothetical simulation reports the following results for one fixed Bayesian procedure. Each value is estimated from 2,000 trials; the “positive decision” is defined by the same prespecified rule in every row.
| True treatment effect scenario | Positive decisions | Estimated probability | Mean sample size |
|---|---|---|---|
| No benefit | 100 of 2,000 | 0.050 | 250 |
| Small benefit | 700 of 2,000 | 0.350 | 310 |
| Clinically important benefit | 1,600 of 2,000 | 0.800 | 360 |
For the no-benefit row, the estimated false-positive rate is \(100/2000=0.050\), or 5.00%. For the clinically important benefit row, estimated power is \(1600/2000=0.800\), or 80.0%. The small-benefit row is neither the false-positive rate nor the power at the clinically important effect; it shows the probability of a positive decision at that intermediate value. The mean sample size rises from 250 to 360 across the two stated rows, so the report should describe that scenario dependence rather than presenting one expected sample size as universal.
The grid does not by itself determine whether the procedure is acceptable. It gives a structured account of performance for reviewers to assess in light of the clinical question and design context. The estimates also have Monte Carlo uncertainty: for example, for the no-benefit rate, \(SE_{\mathrm{MC}}=\sqrt{0.05(0.95)/2000}=0.004873\), or 0.487 percentage points. A plotted curve across a grid of true effects can reveal patterns that a few selected rows might miss.
Diagnostics, Alternatives, and Limitations
Before interpreting results, check that each simulated trial applies the intended design, analysis, and decision rule, and that the scenario values match the report labels. Confirm that the same definition of a positive decision is used in every scenario. Inspect the distribution of final sample size rather than only its mean. Review Monte Carlo uncertainty, particularly for rare outcomes: observing zero false positives in a finite simulation does not show that the true false-positive rate is zero. If no events are observed in \(M\) independent replicates, the rough “rule of three” gives an upper 95% bound of about \(3/M\), illustrating why more replicates may be needed to characterize very small probabilities.
A common mistake is to say that a Bayesian posterior probability threshold of 0.95 means the false-positive rate is 5%. That confuses a conditional posterior probability from one dataset with a repeated-trial probability. The false-positive rate must be estimated by applying the full procedure to repeated trials generated under a specified no-benefit scenario. Another mistake is to report “power” without naming the treatment effect and other scenario assumptions. Power is a function of those assumptions, not a single property of a Bayesian method.
A conventional frequentist operating-characteristic analysis also reports repeated-sampling properties such as type I error and power. These are useful comparison quantities, but a Bayesian design’s probability of a positive decision is determined by its own prior, likelihood, design, and decision rule. A posterior probability threshold does not automatically guarantee a particular frequentist false-positive rate. Report the actual simulated performance rather than assuming the two quantities coincide.
Operating characteristics remain conditional on the simulation model and the scenarios explored. They cannot guarantee performance under unexamined departures, and results may be sensitive to assumptions about prior distributions, external information, or the outcome process. If a reported estimate changes substantially under plausible alternative scenarios or analysis assumptions, that instability is itself important information for review. The report should make clear what the simulations establish, what they do not establish, and which scenarios are most relevant to the intended regulatory decision.
Check Your Understanding
Answer each question, then compare with the solutions.
- In 800 simulated trials under a no-benefit scenario, 32 meet the success rule. What is the estimated false-positive rate?
- Why is a posterior probability of benefit of 0.95 not, by itself, evidence that the false-positive rate is 5%?
- In 1,000 simulated adaptive trials, final sample sizes are 50 for 250 trials and 100 for 750 trials. Calculate the estimated expected sample size and the probability of reaching 100.
- What additional information is needed to interpret a reported power of 80%?
- What does observing zero false positives in a finite set of simulations imply about the true false-positive rate?
Solutions. (1) The estimate is \(32/800=0.040\), or 4.00%. (2) The posterior probability is conditional on one dataset and the model; a false-positive rate is the repeated-trial probability of a positive decision under a specified no-benefit scenario, estimated by simulating and applying the full rule. (3) The mean is \(\{250(50)+750(100)\}/1000=(12{,}500+75{,}000)/1000=87.5\) participants, and the probability of reaching 100 is \(750/1000=0.750\), or 75.0%. (4) Name the true effect and other data-generating assumptions, and specify the design, analysis, and decision event used. (5) It does not show the true rate is zero; finite simulation may miss rare events. With \(M\) replicates and zero events, the rough rule-of-three upper 95% bound is \(3/M\), so more replicates may be needed to estimate a small rate precisely.