Estimating Type I Error For Bayesian Decision Rules

What Type I Error Means for a Bayesian Decision Rule

A Bayesian trial may declare success when the posterior probability of a clinically favorable treatment effect exceeds a chosen threshold. That rule is based on the posterior distribution in one trial. Its Type I error, by contrast, describes how often the rule declares success across repeated trials generated under a null treatment effect. It is a property of the complete trial procedure under that specified scenario, not a posterior probability that the null hypothesis is true.

For example, fix the success rule “declare success if the posterior probability that the treatment effect exceeds zero is at least 0.95.” To estimate its Type I error at a true treatment effect of zero, repeatedly generate trial data under that effect, apply the same analysis and rule to every simulated trial, and count the declarations of success. The decision threshold must be held fixed during this estimation.

This definition concerns a specified null scenario. If other features of the data-generating mechanism matter—such as the control-group outcome rate, outcome variability, or allocation ratio—they must also be specified. A result is conditional on those scenario choices. The definition here does not by itself summarize behavior over a composite null hypothesis.

Specify the Procedure Before Simulating

A simulation is meaningful only if each replicate represents the same intended procedure. State the data-generating mechanism under the null, the analysis model and prior, and the rule that maps trial data to a decision. Keep sample size, allocation, follow-up, handling of missing outcomes, and any interim or stopping rules fixed unless those features are explicitly randomized by the scenario.

Here is a simple model used in the first worked example. Suppose outcomes in each arm are independent and normally distributed with a known common standard deviation \(\sigma\), and each arm has \(n\) participants. Write \(\theta=\mu_T-\mu_C\) for the treatment-minus-control mean difference. For the observed difference \(D=\bar{x}_T-\bar{x}_C\), the analysis likelihood is \(D\mid\theta\sim N(\theta,2\sigma^2/n)\). Suppose the prior is \(\theta\sim N(0,\tau^2)\), and the fixed rule declares success when \(P(\theta>0\mid D)\ge c\). These assumptions make the posterior calculation explicit; they are not necessary for the general simulation definition.

Estimate the Error Rate From Replicates

For replicate \(r\), define \(I_r=1\) if the procedure declares success and \(I_r=0\) otherwise. Generate \(R\) independent trial replicates under the specified null scenario, and apply the same procedure to each. If \(\alpha(\theta_0)\) denotes the true Type I error for that scenario, then each indicator has a Bernoulli distribution with success probability \(\alpha(\theta_0)\). The estimated Type I error is the sample mean of these indicators.

where \(K\) is the number of simulated trials that declared success. Under independent replicates generated from the same scenario, \(E(I_r)=\alpha(\theta_0)\), so \(E\{\widehat{\alpha}(\theta_0)\}=\alpha(\theta_0)\). Also, \(\operatorname{Var}(I_r)=\alpha(\theta_0)\{1-\alpha(\theta_0)\}\). Independence gives \(\operatorname{Var}\{\widehat{\alpha}(\theta_0)\}=\alpha(\theta_0)\{1-\alpha(\theta_0)\}/R\). Replacing the unknown rate by its estimate yields the estimated Monte Carlo standard error (MCSE).

The MCSE measures the simulation’s numerical uncertainty from using a finite number of replicates. It does not measure uncertainty about whether the data-generating model, prior, or chosen null scenario is scientifically appropriate. Increasing \(R\) reduces this simulation uncertainty, but does not correct a misspecified scenario or an incorrectly implemented decision rule.

Checking the Simulation and Interpreting Its Result

Before interpreting a rate, verify that the replicate-level indicator answers the intended question. It should equal one exactly when the complete, pre-specified procedure declares success. If a trial can stop early, fail to produce an analyzable outcome, or trigger a fallback analysis, those possibilities must be handled according to a rule fixed in advance. Quietly dropping non-evaluable replicates changes the denominator and may change the operating characteristic being estimated.

Useful checks include confirming that the treatment effect is set to the specified null value in the generator, checking representative simulated data against the intended distributions, and testing that the posterior calculation and threshold comparison behave as expected. Under a single fixed data set and otherwise identical rule, raising the required posterior probability cannot create additional declarations of success: a trial passing the stricter threshold must also pass the lower one. A violation of this property can reveal a coding or reporting error.

A small number of replicates may make the estimate noisy, especially when the rate is near zero. Report \(R\), \(K\), the estimated rate, and its MCSE so readers can assess that precision. If no simulated trial declares success, the estimate is zero, but that does not establish that the true error rate is zero. Likewise, a high or unexpected estimate should prompt checks of the generator, analysis, and rule before any substantive conclusion.

How This Estimate Relates to Other Approaches

Simulation is useful when the complete trial procedure is too complicated for a convenient probability calculation. It can incorporate realistic features such as unequal allocation, complicated outcome distributions, or pre-specified interim decisions, provided those features are implemented in the generator and analysis. The estimate still answers a conditional question: how often this coded procedure declares success under this coded scenario.

When a simple model permits an exact or reliable analytic calculation, that calculation can serve as a benchmark for checking the simulation. In the normal example above, the posterior probability is a monotone function of \(D\), so the Bayesian rule can be expressed as a threshold on \(D\); under the assumed null, the distribution of \(D\) then gives an analytic false-positive probability. Such a benchmark is valuable for verifying code, but may be unavailable for more complex trial procedures. A frequentist test with a prespecified critical value is another possible procedure; it has its own rejection rule and must be evaluated under the same null scenario for a direct operating-characteristic comparison. The Bayesian posterior threshold does not automatically imply any particular frequentist error rate.

The principal limitation is that the estimate reflects only the scenarios actually simulated. A rate at \(\theta_0=0\) does not establish the error rate for other null values or nuisance-parameter settings. Nor can a large simulation repair an inappropriate outcome model, prior, or handling rule. Keep the assumed scenario and the resulting conditional interpretation explicit.

Common Mistake