Why Evaluate a Bayesian Rule by Frequentist Error Rates?
A Bayesian trial decision can be stated as a rule: for example, declare the treatment effective if the posterior probability that its effect is positive is at least 0.90. That rule uses the observed data and an analysis prior. It does not, by itself, say how often the trial will declare success when the true effect is zero, or how often it will succeed when a particular beneficial effect is true. Those are frequentist operating characteristics: probabilities calculated under fixed data-generating parameters.
To evaluate them, specify the trial’s data model, the analysis prior, and the posterior decision rule. Then repeatedly generate trial data under a chosen fixed null parameter and under a chosen fixed alternative parameter. Apply the same rule to each generated data set. The fraction of null data sets that lead to success estimates type I error; the fraction of alternative data sets that lead to success estimates power.
The subscript matters. A posterior probability such as \(P(\delta>0\mid \text{data})\) describes uncertainty about the effect conditional on observed data and the analysis model. A frequentist operating characteristic instead averages the success decision over repeated data sets generated at a fixed true parameter. It is not the posterior probability that the rule is correct.
Simulation Procedure and Assumptions
For each parameter value being evaluated, the simulation must reproduce the planned trial and analysis. Fix the sample size, allocation, outcome model, analysis prior, posterior success threshold, and any other features that affect the decision. Generate data from the stipulated model at the fixed parameter value, analyze them as planned, and record whether the rule declares success.
Write down what constitutes success, the analysis prior, and the distribution of the trial data at each fixed parameter value.
For each of \(B\) repetitions, independently generate a complete data set under the chosen fixed parameter value, including randomization or other planned design features.
Calculate the posterior quantity specified by the rule and record \(D=1\) if and only if the success condition is met.
Divide the number of simulated successes by the number of repetitions. Repeat the calculation separately for each fixed null or alternative parameter value.
The basic assumptions are that the simulated data model represents the planned trial under the parameter scenario being studied and that the analysis applied to each simulated data set matches the prespecified analysis. Independent repetitions are the usual simple design for estimating the probabilities. If the simulated rule includes missing-data handling, interim decisions, or other design features, those features must also be represented in the simulation.
Worked Example: A Normal-Endpoint Posterior Rule
Worked Example: A Normal-Endpoint Posterior Rule
Suppose the trial’s estimated treatment effect is \(Y\), measured in endpoint units, and the sampling model is \(Y\mid\delta\sim N(\delta,2)\). Thus, the standard error of \(Y\) is \(\sqrt{2}\) units. Use the analysis prior \(\delta\sim N(0,2)\), with the second argument denoting variance, and declare success when \(P(\delta>0\mid Y)\geq 0.90\). Evaluate type I error at \(\delta=0\) and power at \(\delta=3\).
The normal likelihood and prior give posterior variance \((1/2+1/2)^{-1}=1\) and posterior mean \(Y/2\). Therefore, \(\delta\mid Y\sim N(Y/2,1)\), so
The rule declares success when \(\Phi(Y/2)\geq 0.90\), equivalently when
Under the null value \(\delta=0\), \(Y\sim N(0,2)\). The type I error is the probability that \(Y\) exceeds the cutoff:
Thus, under this model, about 3.50% of repeated trials with a true effect of zero would meet the Bayesian success rule. At the alternative value \(\delta=3\), \(Y\sim N(3,2)\), giving
The power at a true effect of 3 endpoint units is approximately 62.13%. A simulation would generate many \(Y\) values from \(N(0,2)\), apply the cutoff, and count successes; it would then do the same using \(N(3,2)\). For example, in a hypothetical run of 20,000 repetitions per scenario, 699 null successes give \(699/20{,}000=0.03495\), and 12,426 alternative successes give \(12{,}426/20{,}000=0.6213\). These simulated rates are close to the exact probabilities above.
Justification: The Rule Defines a Rejection Region
The normal example illustrates a useful justification. A posterior decision rule translates the data into a binary decision. In that example, the posterior probability is an increasing function of \(Y\), so the rule is equivalent to a simple cutoff \(Y\geq c\). Once this rejection region is identified, its frequentist probability follows from the sampling distribution of \(Y\) at each fixed true value.
More generally, let \(D(y)\) equal 1 when the posterior rule declares success for data \(y\), and 0 otherwise. If \(f(y\mid\theta)\) is the data density at fixed parameter \(\theta\), then the operating characteristic is
where \(I\{\cdot\}\) equals 1 when its condition holds and 0 otherwise. A simulation estimates this integral by generating \(B\) data sets \(y_1,\ldots,y_B\) from \(f(y\mid\theta)\) and averaging the corresponding indicators:
The justification is direct: each indicator is a Bernoulli random variable with success probability \(P_{\theta}(D=1)\). Its average therefore estimates that probability. This argument also explains why the data must be generated at the specified fixed \(\theta\), and why the decision rule must be reapplied to every generated data set.
Worked Example: A Beta-Binomial Trial
Worked Example: A Beta-Binomial Trial
Consider a hypothetical trial with 50 participants per group and a binary endpoint. Let \(p_T\) and \(p_C\) be the treatment and control success probabilities. Use independent \(\operatorname{Beta}(1,1)\) analysis priors and declare success when \(P(p_T>p_C\mid\text{data})\geq 0.95\). To estimate type I error, generate both groups from \(p_T=p_C=0.30\). To estimate power, generate from \(p_T=0.50\) and \(p_C=0.30\).
For every repetition, generate \(X_T\sim\operatorname{Binomial}(50,p_T)\) and \(X_C\sim\operatorname{Binomial}(50,p_C)\) independently. Given counts \(x_T,x_C\), the posterior distributions are
For one illustrative data set, suppose \(x_T=26\) and \(x_C=15\). The posterior distributions are \(\operatorname{Beta}(27,25)\) and \(\operatorname{Beta}(16,36)\). The posterior probability used by the rule is
where \(f_T\) is the treatment posterior density and \(F_C\) is the control posterior cumulative distribution function. Numerical integration gives a value above 0.95, so this data set leads to \(D=1\). The same calculation—not a separate rule—is applied to every simulated pair of counts.
Suppose a hypothetical simulation uses \(B=20{,}000\) repetitions at each scenario. If 1,012 null repetitions meet the criterion, the estimated type I error is \(1{,}012/20{,}000=0.0506\), or 5.06%. If 13,470 alternative repetitions meet it, the estimated power is \(13{,}470/20{,}000=0.6735\), or 67.35%. These are simulation estimates for the stated parameter values, sample sizes, prior, and threshold—not universal properties of the posterior rule.
Worked Example: The Posterior Threshold Changes the Error Rates
Worked Example: The Posterior Threshold Changes the Error Rates
Return to the normal-endpoint model \(Y\mid\delta\sim N(\delta,2)\) with prior \(\delta\sim N(0,2)\). Compare the posterior success thresholds 0.90 and 0.95, evaluating type I error at \(\delta=0\) and power at \(\delta=3\). The posterior remains \(N(Y/2,1)\), so a threshold \(r\) gives the data cutoff \(c_r=2\Phi^{-1}(r)\).
For \(r=0.95\), the cutoff is \(c_{0.95}=2(1.644854)=3.289708\). At the null value, the standardized cutoff is \(3.289708/\sqrt{2}=2.3262\), giving type I error \(1-\Phi(2.3262)\approx 0.0100\). At \(\delta=3\), the standardized cutoff is \((3.289708-3)/\sqrt{2}=0.2049\), giving power \(1-\Phi(0.2049)=\Phi(-0.2049)\approx 0.4188\).
The 0.90 threshold in the preceding example gave type I error about 0.0350 and power about 0.6213. Raising the posterior threshold reduced the null success rate in this model, but also reduced power at the specified alternative. A posterior threshold should therefore not be assumed to correspond to any particular frequentist significance level; its operating characteristics depend on the model, prior, sample size, and true parameter value.
Diagnostics, Comparisons, and Limitations
A useful diagnostic is to confirm that the simulated rejection rate changes sensibly when the fixed true parameter changes. In the normal example, a larger true treatment effect makes values above the fixed cutoff more likely. In a binary-endpoint design, inspect the simulated event counts and verify that they follow the stipulated binomial model. Also verify a selection of individual repetitions by hand or with an independent calculation: check that the posterior parameters use the observed counts correctly and that equality at the threshold is handled consistently.
- Do not generate the true effect from the analysis prior when estimating fixed-parameter error rates. That averages operating characteristics over a distribution of effects and answers a different question, such as prior-averaged assurance.
- Do not change the success threshold after seeing each simulated data set. The rule must be fixed before evaluating its repeated-sampling behavior.
- Do not confuse the posterior probability with power. A posterior probability is computed within one data set; power is the proportion of repeated data sets that satisfy the decision rule at a specified true alternative.
- Check the null definition and nuisance parameters. For a composite null, results at one null parameter value do not establish the maximum type I error across the null. Evaluate the relevant null values, including nuisance-parameter settings that could produce higher error.
A conventional frequentist test is usually constructed around a prespecified type I error target and has a rejection region calibrated to that target under a null model. A Bayesian rule instead specifies a posterior condition, often motivated by evidence or decision consequences. Its frequentist type I error and power can still be calculated, but they are consequences to assess rather than automatic guarantees. Simulation is particularly useful when the posterior rule is complicated or the sampling distribution does not yield a convenient analytic calculation.
These operating characteristics are conditional on the assumptions used to generate and analyze the data. They do not account for model misspecification, departures from the assumed outcome distribution, or unmodeled conduct such as differential missingness unless those features are included. Results at a few parameter values also do not summarize performance over every clinically relevant scenario. Finally, simulated rates are estimates: with too few repetitions, they can vary noticeably from the underlying probabilities.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- In one sentence each, define type I error and power for a Bayesian success rule evaluated at fixed parameter values.
- A simulation generates trial effects from the analysis prior and reports the fraction of trials declared successful. Why is that not, in general, the fixed-parameter power at a specified alternative?
- For the normal model above, calculate the success cutoff when the posterior threshold is 0.90.
- In 20,000 simulated trials generated at a fixed null value, 1,020 meet the success rule. What is the estimated type I error?
Solutions. (1) Type I error is the probability of declaring success under a specified fixed null value; power is the probability of declaring success under a specified fixed alternative. (2) Drawing the true parameter from a prior averages performance over parameter values rather than conditioning on one specified value; that is a prior-averaged quantity, not fixed-parameter power. (3) The posterior is \(N(Y/2,1)\), so \(\Phi(Y/2)\geq 0.90\) is equivalent to \(Y\geq 2\Phi^{-1}(0.90)=2(1.281552)=2.563104\) endpoint units. (4) The estimated type I error is \(1{,}020/20{,}000=0.0510\), or 5.10%.