What Is Being Calibrated?
A posterior-probability decision rule declares a trial positive when the posterior probability of a specified treatment benefit exceeds a cutoff. The cutoff is not itself a false-positive rate. It is a decision threshold whose repeated-use false-positive rate depends on the trial design, the analysis model, the prior, and the true outcome-generating scenario. Calibration uses simulated trials to assess that operating characteristic and select an acceptable cutoff.
For a concrete setting, let \(p_T\) and \(p_C\) denote the treatment and control response probabilities, and let \(\delta=p_T-p_C\). Suppose the decision is to declare a benefit if the posterior probability that \(\delta>0\) exceeds \(c\). The same calibration logic applies to other endpoints and benefit definitions, but the event defining a positive result must be specified before simulation.
The design is held fixed during this exercise: for example, sample sizes, allocation, follow-up, analysis, prior, and any interim rules are not changed while candidate cutoffs are compared. The null scenarios are also specified in advance. A cutoff that meets the target for a few simulated scenarios has not thereby been shown to meet it for every possible null scenario; the conclusion is limited to the scenarios examined and the quality of the simulation evidence.
From Posterior Probabilities to Replicate-Level Decisions
In each simulated replicate, generate trial data under a chosen null scenario, apply the planned Bayesian analysis, calculate the posterior probability of the target benefit, and record whether that probability exceeds \(c\). The proportion of positive decisions across replicates estimates the false-positive rate for that cutoff and scenario.
For instance, in a two-arm binary-outcome trial with \(n_T\) and \(n_C\) participants, a simple model might assume independent outcomes within and between arms, with \(X_T\sim\operatorname{Binomial}(n_T,p_T)\) and \(X_C\sim\operatorname{Binomial}(n_C,p_C)\). If the analysis uses independent \(\operatorname{Beta}(1,1)\) priors, then after observing \(x_T\) and \(x_C\), the posterior distributions are \(p_T\mid D\sim\operatorname{Beta}(x_T+1,n_T-x_T+1)\) and \(p_C\mid D\sim\operatorname{Beta}(x_C+1,n_C-x_C+1)\). These assumptions describe this illustrative model; a real calibration must simulate and analyze using the actual planned design and model, including relevant features such as clustering or interim monitoring.
Worked Example: Applying a Cutoff to One Replicate
Consider a deliberately tiny illustration with one participant per arm, independent binary outcomes, and independent \(\operatorname{Beta}(1,1)\) priors. Suppose the treatment participant responds and the control participant does not: \(x_T=1\), \(x_C=0\). The posterior distributions are \(p_T\mid D\sim\operatorname{Beta}(2,1)\) and \(p_C\mid D\sim\operatorname{Beta}(1,2)\). Find the posterior probability of benefit and apply cutoffs of 0.80 and 0.90.
The treatment posterior density is \(f_T(x)=2x\), and the control posterior cumulative distribution function is \(F_C(x)=1-(1-x)^2\), for \(0\leq x\leq1\). Therefore,
At \(c=0.80\), this replicate is positive because \(0.8333>0.80\). At \(c=0.90\), it is not positive because \(0.8333\) is not greater than 0.90. This example checks how a single replicate becomes a decision; it does not estimate a false-positive rate. Calibration requires many such replicates generated under null scenarios.
A Simulation-Based Calibration Procedure
Choose the null scenarios, target rate, candidate cutoffs, and simulation rule before inspecting results. For each scenario and candidate cutoff, simulate \(R\) independent replicates. If \(K(c;\theta)\) replicates are positive, the estimated false-positive rate is \(\hat{\phi}(c;\theta)=K(c;\theta)/R\). Compare the rates across scenarios, not just the most convenient scenario. A conservative scenario-wise summary is the largest estimated rate among the specified null scenarios.
The following justification explains why increasing the cutoff cannot increase the true false-positive rate, provided the posterior probability and all other parts of the decision rule remain fixed. If \(c_1<c_2\), then every data set with posterior probability above \(c_2\) also has posterior probability above \(c_1\). Thus \(\{A_{c_2}=1\}\subseteq\{A_{c_1}=1\}\), and taking probabilities under any null scenario gives \(\phi(c_2;\theta)\leq\phi(c_1;\theta)\). Consequently, on the same simulated data, the estimated rates must also be non-increasing as the cutoff rises. A violation of this pattern is a useful signal to check the simulation or bookkeeping.
Specify the planned trial, analysis, prior, benefit event, and how a posterior probability is converted into a decision.
State the scenarios to be assessed and the maximum acceptable false-positive rate \(\alpha\). Include clinically and statistically relevant parts of the null, rather than choosing scenarios after seeing their results.
For each scenario, generate replicates, run the planned analysis, and record a positive-decision indicator for each candidate cutoff.
Compare each candidate's estimated rates with the target across all scenarios. State whether selection is based on point estimates or on a more conservative uncertainty-aware criterion.
Worked Example: Screening Cutoffs Across Null Scenarios
Suppose a fixed two-arm trial design is simulated \(R=5{,}000\) times under each of two null scenarios, with equal response probabilities \(p_T=p_C=0.20\) and \(p_T=p_C=0.50\). The analysis and benefit event are held fixed, and the decision is based on whether the posterior probability of benefit exceeds \(c\). The target maximum false-positive rate is \(\alpha=0.025\). The table gives hypothetical positive-decision counts from the simulations.
| Cutoff \(c\) | Positives when \(p_T=p_C=0.20\) | Estimated rate | Positives when \(p_T=p_C=0.50\) | Estimated rate |
|---|---|---|---|---|
| 0.90 | 310/5,000 | 0.0620 | 481/5,000 | 0.0962 |
| 0.95 | 171/5,000 | 0.0342 | 265/5,000 | 0.0530 |
| 0.975 | 92/5,000 | 0.0184 | 139/5,000 | 0.0278 |
| 0.99 | 40/5,000 | 0.0080 | 74/5,000 | 0.0148 |
For \(c=0.975\), the largest estimated rate is \(139/5{,}000=0.0278\), which exceeds the target \(0.025\). For \(c=0.99\), the largest is \(74/5{,}000=0.0148\), below the target. Thus, among these candidate cutoffs, 0.99 passes a point-estimate screen across both scenarios and 0.975 does not. This is a conclusion about the specified simulated scenarios, not a proof that the true maximum over every possible null value is below the target.
For scale, the estimated Monte Carlo standard error of the largest rate at \(c=0.975\) is
An approximate upper endpoint using 1.96 standard errors is \(0.0278+1.96(0.002325)\approx0.03236\), further indicating that the result is not convincingly below 0.025. At \(c=0.99\), for the scenario with the largest estimate, \(SE=\sqrt{0.0148(1-0.0148)/5{,}000}\approx0.001708\); the corresponding approximate upper endpoint is \(0.0148+1.96(0.001708)\approx0.01815\). Both the point estimate and this rough uncertainty check are below 0.025 for that scenario. In practice, use a suitable interval method and apply the prespecified uncertainty criterion consistently to every required scenario.
What the Cutoff Costs
A cutoff that controls false positives may also reduce the chance of a positive decision when treatment truly helps. Therefore, after finding candidates that meet the false-positive target, examine their behavior under clinically meaningful alternatives. This does not change the null calibration criterion: it shows the practical consequence of choosing a more stringent cutoff and helps avoid treating lower false-positive rates as the only design objective.
Worked Example: Checking the Power Consequence
Continue with the fixed design and suppose an alternative scenario has \(p_T=0.35\) and \(p_C=0.25\). In a separate set of 5,000 hypothetical replicates, assume 1,210 are positive at \(c=0.975\) and 840 are positive at \(c=0.99\). The estimated probabilities of a positive decision under this alternative are
The higher cutoff reduces the estimated positive-decision probability by \(0.242-0.168=0.074\), or 7.4 percentage points, in this alternative scenario. Since \(c=0.975\) failed the stated null target in the preceding example, the comparison does not make it an acceptable choice; it quantifies a consequence to consider among cutoffs that do meet the target.
Diagnostics, Alternatives, and Limitations
Calibration is only as credible as the simulated trial and the scenarios used. Check that simulated sample sizes, allocation, outcome distributions, analysis, and decision rule match the plan. Verify that replicate-level posterior probabilities are valid probabilities and that the recorded indicator implements the intended strict or non-strict comparison consistently. Compare estimated rates across scenarios and cutoffs; unexplained non-monotonicity across cutoffs using the same replicates can reveal a coding or data-alignment error. Repeated simulation with an independently generated batch can also help identify unstable estimates.
A conventional posterior cutoff such as 0.95 is a possible starting candidate, not a guarantee of any particular false-positive rate. The rate is a repeated-sampling property under a specified data-generating scenario, whereas the posterior probability is conditional on the observed data and the analysis model. An analytic calculation may sometimes provide a benchmark or exact answer, but simulation is useful when the decision rule and design make the sampling distribution difficult to derive. Changing the sample size or analysis to meet the target is a different design exercise; it would no longer be calibration of a cutoff for the same fixed design.
Important limitations remain. A finite set of null scenarios may miss a scenario with a higher rate; a finite number of replicates leaves Monte Carlo uncertainty; and a mismatch between the simulated and actual trial process can make even precise estimates misleading. If the target is close to an estimated rate, the result can depend on simulation noise. In that situation, report the estimate and its uncertainty, increase replication if warranted, and avoid describing a borderline point estimate as established control. Calibration supports a design decision under stated assumptions; it does not establish that the prior or model is scientifically correct.
Common Mistake
Check Your Understanding
Answer each question, then compare your reasoning with the brief solutions.
- In a simulation with 2,000 replicates under a null scenario, 46 trials meet the posterior decision rule. What is the estimated false-positive rate?
- Why is a cutoff of 0.95 not, by itself, evidence that the false-positive rate is 0.05?
- If \(c_1<c_2\), what relationship should hold between their false-positive rates when every other part of the rule is unchanged?
- A cutoff's estimated rate is below the target in one null scenario but above it in another required scenario. Does it meet a rule requiring the target in every specified scenario?
- Why should a cutoff that meets the null target also be assessed under treatment-benefit scenarios?
Solutions. (1) The estimate is \(46/2{,}000=0.023\), or 2.3%. (2) The cutoff applies to a posterior probability conditional on observed data; the false-positive rate is the repeated-sampling probability of a positive decision under a null scenario, and must be estimated or derived separately. (3) The higher cutoff cannot increase the rate: \(\phi(c_2;\theta)\leq\phi(c_1;\theta)\), because its positive-decision event is a subset. (4) No. The target must hold in every required scenario under that rule, so the failing scenario prevents the candidate from passing. (5) A stricter cutoff can lower the chance of a false positive while also lowering the chance of a positive decision when there is a real benefit; the latter is important for judging the design's clinical usefulness.