Why Calibrate a Posterior Threshold?
A rule such as “declare success when the posterior probability of a clinically meaningful benefit exceeds 0.95” is easy to state, but the number 0.95 does not by itself determine the trial’s repeated-sampling properties. The probability of declaring success depends on the sample size, outcome model, prior, clinically meaningful effect, and any interim stopping rules. Calibration examines those consequences before the rule is used.
The central task is to simulate many trials under specified true treatment effects, apply the proposed Bayesian analysis and decision rule to every simulated trial, and summarize the decisions. This estimates operating characteristics such as type I error and power. If the design can stop early, the same simulations estimate expected sample size. These are properties of the entire design-and-decision procedure, not properties of the posterior probability from one particular trial.
Model and Decision Rule
Consider a hypothetical, equal-allocation, two-arm trial with a binary response. Let \(p_C\) and \(p_T\) be the true response probabilities in the control and treatment arms. At an analysis with \(n_C\) and \(n_T\) participants, let \(x_C\) and \(x_T\) be the observed numbers of responses. Assume independent binomial sampling conditional on the arm-specific response probabilities:
For illustration, assign independent \(\operatorname{Beta}(1,1)\) priors to \(p_C\) and \(p_T\). Conditional on the observed data, the posterior distributions are independent, with \(p_C\mid x_C\sim\operatorname{Beta}(1+x_C,1+n_C-x_C)\) and \(p_T\mid x_T\sim\operatorname{Beta}(1+x_T,1+n_T-x_T)\). The prior and likelihood assumptions must be the same in each simulated trial as in the proposed analysis.
Let \(\Delta=0.05\) be the minimum clinically meaningful increase in response probability, a difference of five percentage points. Define the posterior evidence quantity
A candidate decision threshold \(c\) declares success at an analysis if \(q(x_C,x_T)\ge c\). The threshold is a design choice to be calibrated; it is not automatically a frequentist significance level. For independent beta posteriors, \(q\) can be evaluated by integration. If \(f_T\) is the treatment posterior density and \(F_C\) is the control posterior cumulative distribution function, then
The lower integration limit is \(\Delta\), because values \(p_T\le\Delta\) cannot satisfy \(p_C<p_T-\Delta\). In practice, sufficiently accurate numerical integration or carefully controlled posterior simulation can evaluate this probability.
For this example, define the null as \(p_T-p_C\le\Delta\), and the alternative of interest as an effect greater than \(\Delta\). The null boundary is \(p_T-p_C=\Delta\). Evaluating the boundary is often informative, but it is not automatically sufficient: nuisance parameters or a non-monotone operating characteristic can make another null scenario worse. A rigorous calibration considers a prespecified range of plausible control rates and treatment effects.
How the Simulation Is Built
Suppose each arm begins with 30 participants and may accrue to 60 per arm. At the interim analysis, stop for efficacy if \(q\ge c\), stop for futility if \(q\le f\), and otherwise continue to the maximum sample size. At the final analysis, declare success if \(q\ge c\). Here \(f=0.10\) is held fixed while candidate values of \(c\) are compared. This is one explicit decision algorithm; changing its interim rules changes the operating characteristics and requires recalibration.
Choose true values of \(p_C\) and \(p_T\) under the null and under clinically relevant alternatives. Include important nuisance-rate values where appropriate.
For each replicate, generate independent binomial response counts for the two arms at the interim. If the trial continues, generate the additional outcomes using the same true rates.
Calculate the posterior probability \(q\), apply the futility and efficacy rules, and record the final decision and total sample size.
Across replicates, estimate the success rate under each scenario and the mean sample size. Quantify Monte Carlo error and repeat or enlarge the simulation if precision is inadequate.
Using additional outcomes conditional on the interim data is equivalent to simulating the full set of outcomes in advance and revealing the later outcomes only for trials that continue. Each replicate must represent a new trial, with outcomes generated under the specified true rates. The randomization ratio, missing-data assumptions, analysis timing, prior, and stopping rules should reflect the design being evaluated.
Worked Example: Posterior Evidence at an Interim
Suppose a hypothetical trial has 30 participants per arm at its interim analysis, with 6 responses in control and 12 in treatment. Under the independent \(\operatorname{Beta}(1,1)\) priors, the posterior distributions are
For \(\Delta=0.05\), estimate \(q=P(p_T-p_C>0.05\mid data)\) using 20,000 independent draws from each posterior, pairing one treatment draw with one control draw on every iteration. Suppose 17,438 paired draws satisfy the inequality. The simulation estimate is
Its approximate Monte Carlo standard error is \(\sqrt{0.8719(1-0.8719)/20{,}000}=0.002363\). This uncertainty is only from estimating the posterior probability by simulation; it is not uncertainty about the clinical treatment effect. With candidate efficacy threshold \(c=0.90\), this estimate is below the threshold, so this interim does not meet the efficacy stopping rule. It is also above the futility threshold \(f=0.10\), so the hypothetical trial continues. The outcome is determined by the stated rules, not by treating 0.8719 as a probability that the rule itself is correct.
Estimating Operating Characteristics
For \(M\) simulated trials under one fixed true scenario, let \(I_m=1\) if replicate \(m\) declares success and \(I_m=0\) otherwise. The estimated probability of success under that scenario is the proportion of simulated trials declaring success:
Under a null scenario, \(\widehat r\) estimates the type I error for that scenario; under an alternative, it estimates power at that alternative. The standard error is Monte Carlo error: it describes the variability of the estimated rate across repeated simulation runs of size \(M\). It does not describe uncertainty in the true clinical response rates or in the model assumptions.
If \(N_m\) is the total number of participants accrued in replicate \(m\), the estimated expected sample size is \(\bar N=M^{-1}\sum_m N_m\). Its Monte Carlo standard error is \(s_N/\sqrt{M}\), where \(s_N\) is the sample standard deviation of the simulated sample sizes. Report whether sample size means total participants across both arms or participants per arm. Under the design above, total sample size is 60 at the interim and 120 at the maximum.
Worked Example: Monte Carlo Error for a Type I Error Estimate
Suppose 10,000 simulated trials are generated at a specified null scenario, and 600 declare success. The estimated type I error at this scenario is \(600/10{,}000=0.060\), or 6.0%. The Monte Carlo standard error on the probability scale is
On a percentage-point scale, this is \(0.002375\times100=0.2375\) percentage points. Thus the estimated 6.0% error rate has Monte Carlo standard error about 0.2375 percentage points. This does not mean that the true type I error is known to within that amount: it measures simulation precision at the chosen scenario, not design-model uncertainty. More replicates reduce this Monte Carlo error approximately in proportion to \(1/\sqrt{M}\).
Worked Example: Comparing Candidate Thresholds
Consider a hypothetical simulation with 10,000 replicates per scenario, the interim and maximum sample sizes described above, \(f=0.10\), and three candidate values of \(c\). The null scenario is \(p_C=0.20,\ p_T=0.25\), which lies on the margin \(\Delta=0.05\); the alternative is \(p_C=0.20,\ p_T=0.40\). The following are illustrative simulation results, not results from a real trial.
| Threshold \(c\) | Successes under null | Estimated type I error | Successes under alternative | Estimated power | Mean total N under null | Mean total N under alternative |
|---|---|---|---|---|---|---|
| 0.90 | 1,080 / 10,000 | 0.108 | 8,240 / 10,000 | 0.824 | 90 | 102 |
| 0.95 | 600 / 10,000 | 0.060 | 7,600 / 10,000 | 0.760 | 96 | 108 |
| 0.975 | 350 / 10,000 | 0.035 | 7,040 / 10,000 | 0.704 | 102 | 111 |
For \(c=0.95\), the estimated type I error is \(600/10{,}000=0.060\), while power at the stated alternative is \(7{,}600/10{,}000=0.760\). If 40% of null replicates stopped after the interim, the mean total sample size would be \(0.40(60)+0.60(120)=96\), matching the table. Under the alternative, if 20% stopped at interim, the mean would be \(0.20(60)+0.80(120)=108\).
The table shows the trade-off for these particular scenarios and this particular rule: raising \(c\) reduces the estimated false-positive rate, but also reduces power and tends to increase sample size because fewer trials stop early for efficacy. A sponsor or regulator would compare these estimates with prespecified requirements, alongside the results for other plausible scenarios. A threshold is not selected just because its posterior probability sounds numerically stringent.
Assumptions, Diagnostics, and Limitations
Simulation does not repair a misspecified model. The binomial setup assumes independent participant outcomes within each arm conditional on the true rates, complete ascertainment of the binary endpoint, and the stated allocation and follow-up. Clustered outcomes, nonadherence, missing outcomes, changing response rates over time, or unequal follow-up may require a different data-generating model and matching analysis. The prior used in the trial analysis must also be specified, because it can affect posterior probabilities and thus the probability of crossing a decision threshold.
Useful checks include the following:
- Verify the null and alternative scenarios. Check that each generated data set uses the intended rates and that the null includes the clinically defined margin. Examine more than one null control rate when nuisance-rate variation is important.
- Test the decision code. Check boundary cases, such as \(q=c\), \(q=f\), and a trial that continues to the maximum. Confirm that ties and inequalities follow the protocol exactly.
- Check Monte Carlo precision. Report the number of replicates and Monte Carlo standard errors. Increase \(M\) when the estimated rate is too imprecise for the decision being made.
- Check posterior-probability computation. Compare numerical integration with a high-precision computation on selected data configurations. If posterior draws are used inside every replicate, control the extra simulation noise; otherwise decisions near \(c\) can be misclassified by noisy estimates of \(q\).
- Inspect scenario sensitivity. A result at one null boundary and one alternative does not establish performance over all plausible rates. Display operating characteristics across the range relevant to the planned population.
Simulation-based calibration is particularly useful when the decision algorithm includes interim looks, nonstandard priors, or other features that make analytic calculation difficult. Analytic calculations can be faster and exact for simple fixed-sample models, and they are valuable as a check on simulation. The posterior probability \(q\), by contrast, summarizes evidence conditional on the observed data and model. It does not replace an operating-characteristic assessment: the posterior cutoff can be selected to achieve desired repeated-sampling behavior, but that behavior must be evaluated under the scenarios that matter.
A common mistake is to set \(c=0.95\) and claim that the trial therefore has a 5% type I error. The posterior probability is conditional on the observed data and prior, whereas type I error averages the trial’s decisions over repeated data sets generated under a null scenario. The correction is to simulate the complete trial and decision rule under null scenarios and estimate the fraction declaring success. A second limitation is that a low Monte Carlo standard error cannot compensate for unrealistic assumptions about the outcome process; precision of computation and credibility of the model are separate questions.
Check Your Understanding
Answer each question using the definitions and decision rule in this tutorial. Solutions follow the questions.
- If 450 of 10,000 simulated trials declare success under a null scenario, what is the estimated type I error for that scenario?
- Why does a posterior threshold of 0.95 not guarantee a type I error of 0.05?
- In a design with 30 participants per arm at interim and 60 per arm maximum, 25% of simulated trials stop at interim. What is the mean total sample size?
- What additional concern arises if posterior probabilities in each replicate are themselves estimated using a small number of posterior draws?
- Why should calibration examine more than one true response-rate scenario?
Solutions. (1) The estimate is \(450/10{,}000=0.045\), or 4.5%, at that specified null scenario. (2) The posterior threshold is applied to evidence conditional on each observed data set; type I error is the repeated-trial probability of declaring success under the null. It must be estimated by simulating the full procedure. (3) Total sample size is 60 at interim and 120 at maximum, so the mean is \(0.25(60)+0.75(120)=15+90=105\) participants. (4) Posterior Monte Carlo noise can move an estimated probability across the decision threshold, adding computational error to the operating-characteristic estimate; use accurate integration or enough draws and assess the effect. (5) Control response rates and treatment effects can change the probability of success and stopping, so a result at one scenario may not represent the design’s performance across plausible conditions.