From a Posterior Threshold to a Repeated-Trial Error Rate
A Bayesian trial decision may be expressed as a rule such as: declare treatment benefit if the posterior probability that the treatment effect is beneficial is at least 0.95. That rule is applied to the data from one trial. Its posterior probability describes uncertainty about the effect under the specified Bayesian model; it does not, by itself, tell us how often the rule will declare benefit when the treatment has no true benefit.
To quantify that repeated-trial behavior, simulate trials under a null scenario, analyze each simulated data set using the planned Bayesian model, and record whether the fixed decision rule is met. The fraction of simulated trials meeting the rule estimates the false-positive rate under that scenario. The rule, prior, analysis, trial design, and data-generating assumptions all matter: changing any of them can change the rate.
This definition separates two probabilities. The posterior probability \(P(\delta>0\mid D)\) is calculated within a trial, conditional on its data and Bayesian model. The false-positive rate is calculated across repeated trials generated under a specified true null scenario. A threshold of 0.95 is not a guarantee that the false-positive rate is 0.05.
Set Up the Simulation Before Counting
A valid estimate requires a precise specification of both how trial data are generated and how each simulated trial is analyzed. The data-generating model represents the assumed truth for operating-characteristic evaluation. The analysis model is the planned model used to obtain the posterior probability and make the decision. They may be the same, but this should be an explicit design choice rather than an accidental assumption.
For a particular null scenario, define \(I_r=1\) if replicate \(r\) meets the fixed decision rule, and \(I_r=0\) otherwise. With \(R\) independent simulated trials, the estimated false-positive rate is the average of these indicators. If the true repeated-trial rate under the scenario is \(\pi_0\), then each indicator has a Bernoulli distribution with success probability \(\pi_0\). This gives the Monte Carlo standard error of the estimate.
The standard error here measures simulation error from using a finite number of replicates. It is not the standard error of a treatment-effect estimate within a trial, nor does it represent uncertainty about whether the data-generating assumptions are realistic. Increasing \(R\) can reduce Monte Carlo error, but it cannot correct a misspecified trial model or an incorrectly implemented decision rule.
Fix the treatment-benefit direction, posterior threshold \(q\), null value or nuisance-parameter settings, trial design, and data-generating model.
Generate a replicate under the null scenario, apply the planned Bayesian analysis, and compute its posterior probability of benefit.
Set \(I_r=1\) exactly when the posterior probability is at least \(q\). Average the indicators and report Monte Carlo uncertainty.
Worked Examples
Worked Example: Estimating the Rate at a Null Boundary
Suppose a hypothetical trial uses the fixed rule \(P(\delta>0\mid D)\ge 0.95\) to declare benefit. Under the boundary-null scenario \(\delta=0\), 10,000 independent trial replicates are generated and analyzed using the planned prior and model. In 338 replicates the posterior probability meets or exceeds 0.95.
The replicate-level outcome is \(I_r=1\) for each of those 338 trials and \(I_r=0\) for the other 9,662. Therefore,
The estimated false-positive rate at this boundary scenario is 0.0338, or 3.38%. Its estimated Monte Carlo standard error is
Thus simulation error in this estimate is about 0.181 percentage points. A rough normal-approximation interval for the repeated-trial rate is \(0.0338\pm1.96(0.001807)\), or approximately 0.0303 to 0.0373. This interval reflects Monte Carlo uncertainty under the specified scenario; it does not account for uncertainty about the scenario assumptions. The result says the rule declared benefit in about 3.38% of simulated trials at \(\delta=0\), not that the posterior probability threshold itself was 3.38%.
Worked Example: Checking More Than One Null Value
Suppose benefit is represented by \(\delta>0\), so the composite null is \(\delta\le 0\). A hypothetical simulation examines three null values, using 5,000 replicates at each value and the same prespecified posterior threshold. The counts of positive decisions are 60 at \(\delta=-0.20\), 170 at \(\delta=0\), and 105 at \(\delta=-0.10\), with effects measured in the trial's specified outcome units.
The estimated rates are calculated separately:
Among these three simulated scenarios, the largest estimated rate is 0.034, at the boundary \(\delta=0\). For example, the Monte Carlo standard error at that value is
The maximum observed on a finite grid is not automatically the exact worst-case rate over every \(\delta\le0\). The result supports the statement that 0.034 was the largest estimate among the three tested values. Additional values, a justified monotonicity result, or a more comprehensive search may be needed to support a claim about the entire composite null. Each point also has Monte Carlo uncertainty, so small differences between rates may not be meaningful.
Worked Example: Classifying a Trial Under a Normal Analysis Model
Consider a hypothetical analysis in which the trial estimate \(Y\), measured in outcome units, has likelihood \(Y\mid\delta\sim N(\delta,0.20^2)\). The analysis uses the prior \(\delta\sim N(0,0.50^2)\), and declares benefit when \(P(\delta>0\mid Y)\ge0.95\). Assume for this illustration that the sampling standard deviation is known and the stated normal likelihood and prior are used for every simulated replicate.
For this normal likelihood and normal prior, the posterior variance is the reciprocal of the sum of the prior and likelihood precisions. The posterior mean is the precision-weighted mean of the prior mean and observed estimate:
Suppose one simulated trial generated under the null \(\delta=0\) produces \(Y=0.40\). Its posterior mean is \(m=25(0.40)/29\approx0.3448\), and its posterior standard deviation is \(\sqrt{1/29}\approx0.1857\). Since the posterior is normal,
Because 0.968 exceeds 0.95, this replicate is classified as positive and contributes \(I_r=1\) to the false-positive-rate estimate. The classification is based on the prespecified rule, not on whether this single estimate seems surprising. Repeating the generation and analysis many times under \(\delta=0\) estimates how frequently such positive classifications occur.
Assumptions, Diagnostics, and Interpretation
The simulation estimate is meaningful only relative to its assumptions. At minimum, replicates should represent independent trials generated under the stated scenario, and the same planned analysis and decision rule should be applied consistently to every replicate. If the actual trial includes randomization, allocation, missing outcomes, interim decisions, or other design features that can affect the posterior probability, the simulation should reproduce those features when they are part of the target design.
Several checks help detect implementation problems:
- Check the decision boundary. Test that a posterior probability equal to \(q\) is counted as positive when the rule is defined using “at least” \(q\), and confirm the benefit direction is correct.
- Inspect replicate-level outputs. Review posterior probabilities near the threshold and confirm that each replicate contributes exactly one binary outcome.
- Verify the data generator and analysis separately. Simulated outcomes should have the intended null behavior, while posterior calculations should use the specified analysis model, prior, and estimand.
- Check simulation precision. Report \(R\), the estimated rate, and its Monte Carlo standard error or interval. A small number of replicates can make the estimated rate unstable, especially for rare positive decisions.
- Probe null scenarios. For a composite null, examine plausible values of nuisance parameters and treatment effects rather than assuming that a single null point represents all cases.
A common mistake is to interpret a 0.95 posterior probability threshold as an automatic 5% false-positive rate. The reasoning is incorrect because 0.95 is a within-trial posterior probability, whereas the false-positive rate is a repeated-sampling probability under a specified null truth. The correction is to simulate null trials, apply the fixed threshold to each trial, and estimate the fraction classified as positive.
Simulation is especially useful when the full trial design or posterior decision rule makes an analytic calculation difficult. An analytic benchmark can be more efficient when available, and it is useful for checking simulation code under simplified conditions. A frequentist type I error analysis also describes repeated-trial false positives, but its test and rejection rule need not match a Bayesian posterior-threshold rule. These approaches answer related questions, yet they are not interchangeable unless they evaluate the same data-generating scenarios and decision rule.
Finally, an estimated rate is conditional on the data-generating assumptions, analysis model, prior, trial design, and fixed threshold used. A low estimated rate in a limited set of scenarios does not establish that the rule performs well under every plausible null configuration. Conversely, a high rate is not a posterior probability that the null is true. It is evidence about the operating behavior of the specified rule under the simulated assumptions.
Check Your Understanding
Answer each question, then compare with the brief solutions.
- A threshold rule is applied to 4,000 simulated trials at a null value, and 120 trials are positive. What is the estimated false-positive rate?
- In one replicate, the posterior probability of benefit is 0.949 when the threshold is 0.95. How should the replicate be classified?
- Why does a threshold of 0.95 not imply a 5% false-positive rate?
- A composite-null simulation finds its largest estimated rate at one of four grid points. What can be concluded, and what cannot be concluded without further justification?
- What does the Monte Carlo standard error describe, and what important source of uncertainty does it not describe?
Solutions. (1) \(120/4{,}000=0.030\), so the estimated rate is 3.0% at that scenario. (2) It is not positive, because 0.949 is below 0.95. (3) The threshold is a posterior probability conditional on one trial's data; the false-positive rate is the fraction of repeated trials under a null truth that cross the threshold. (4) The largest estimated rate among the tested grid points is identified, but this does not establish the maximum over all null values unless the grid is adequate or a result justifies that conclusion. (5) It describes finite-replicate simulation error in the estimated rate, not uncertainty about whether the data-generating and analysis assumptions represent the intended trial.