Why a Bayesian Decision Threshold Needs a Rationale
A rule such as “recommend the treatment if the posterior probability of benefit is at least 0.95” is easy to state. Defending it is harder. Why is 0.95 appropriate for this decision, what clinical consequences does it reflect, how often would the rule recommend treatment in relevant scenarios, and does the resulting evidence address the regulatory question? A posterior threshold is not self-justifying simply because it is a familiar number.
A defensible rationale connects three things: the consequences of the possible decisions, the behavior of the complete decision rule under plausible data-generating conditions, and the evidentiary standard for the intended action. The threshold should be prespecified and interpreted in that context. It is not a universal Bayesian equivalent of a particular frequentist significance level, nor does meeting it automatically determine a regulatory outcome.
For example, \(A\) might be the region where treatment benefit exceeds a clinically meaningful margin, not merely the region where the effect is greater than zero. The definition of \(A\), the posterior model, the threshold \(q\), and the action are all parts of the decision rule. A rationale for \(q\) cannot repair an unclear clinical target or an unsuitable analysis model.
Linking the Threshold to Clinical Consequences
One way to clarify the rationale is to compare the consequences of acting when the desired state is false with those of not acting when it is true. Consider a simplified decision with two states: \(A\), where the treatment has the desired benefit, and \(A^c\), where it does not. Let \(p=P(A\mid D)\). Suppose acting incurs loss \(L_{FP}\) if the state is \(A^c\), while not acting incurs loss \(L_{FN}\) if the state is \(A\). For this simplified calculation, assume the other two outcomes have zero loss.
Choose to act when its expected loss is smaller. Thus \(L_{FP}(1-p) < L_{FN}p\), which rearranges to the threshold below. This derivation makes the direction of the threshold clear: a more consequential false-positive action raises the required posterior probability, while a more consequential missed benefit lowers it.
Worked Example: A Threshold From Simplified Clinical Losses
Suppose a hypothetical decision analysis assigns 9 loss units to acting when the treatment does not provide the desired benefit, and 1 loss unit to withholding action when that benefit is present. The units are a simplified scale for comparing consequences, not measured clinical outcomes.
Here \(L_{FP}=9\) and \(L_{FN}=1\). The rule implied by these assumptions is \(p > 9/(9+1)=0.90\). For a posterior probability \(p=0.92\), the expected loss of acting is \(9(1-0.92)=0.72\) units, while the expected loss of not acting is \(1(0.92)=0.92\) units. Acting has the lower expected loss under this simplified model.
The calculation does not prove that 0.90 is the right threshold for a real submission. The loss values may be difficult to elicit, consequences may not be adequately represented by two states, and regulatory decisions involve more than this stylized comparison. It does show what assumptions would support that threshold and makes those assumptions open to review.
A threshold can also be justified by a prespecified evidentiary standard rather than by explicit numerical losses. In that case, the rationale still needs to say what the probability refers to, why that degree of posterior support is appropriate for the intended decision, and how the analysis accounts for uncertainty and clinical relevance. A threshold should not be chosen after observing trial results to make the decision come out favorably.
Deriving and Interpreting an Operating Characteristic
A posterior threshold describes a decision conditional on observed data. An operating characteristic describes how often that decision rule would act across repeated hypothetical trials under a specified true parameter value. For a rule \(R(D)\), the action probability at a fixed true value \(\theta\) is \(P_\theta\{R(D)=\text{act}\}\). It depends on the full design and analysis, including the sampling model, prior, endpoint, sample size, and decision rule. It is not the posterior probability of benefit in a particular trial.
The distinction matters because a stringent posterior threshold may control action when the treatment has no benefit but also make action uncommon when benefit is clinically important. Operating characteristics reveal that trade-off over a range of parameter values; they do not, by themselves, say whether the trade-off is clinically or regulatorily acceptable.
Worked Example: Translating a Posterior Threshold Into a Data Cutoff
Consider an illustrative normal model. Let \(\theta\) be a treatment effect in endpoint units, with positive values favoring treatment. The observed estimate \(Y\) has sampling distribution \(Y\mid\theta\sim N(\theta,2^2)\), so its standard error is 2 endpoint units. Use a flat prior for this algebraic illustration; the resulting posterior is \(N(Y,2^2)\). This setup is for demonstrating the connection between a posterior threshold and an operating characteristic, not a recommended regulatory model.
Suppose the decision rule is to act when \(P(\theta>0\mid Y)\geq q\), with \(q=\Phi(6.616/2)\), approximately 0.9995. Under the stated model, \(P(\theta>0\mid Y)=\Phi(Y/2)\). Since \(\Phi\) is increasing, the posterior rule is equivalent to \(Y\geq 6.616\) endpoint units.
At a fixed true effect \(\theta\), \(Y\sim N(\theta,4)\). Therefore, the probability of acting is \(P_\theta(Y\geq 6.616)=1-\Phi((6.616-\theta)/2)\). At \(\theta=0\), this is approximately 0.0005. At \(\theta=6\), it is \(1-\Phi(0.308)\), approximately 0.3790. At \(\theta=8\), it is \(1-\Phi(-0.692)\), approximately 0.7555. Rounded to three decimal places, these are 0.0005, 0.379, and 0.756, respectively.
Thus this rule rarely acts when the true effect is zero, but it acts in only about 75.6% of hypothetical trials when the true effect is 8 endpoint units. Whether 8 units is an important effect must be determined clinically; the calculation alone cannot establish that. The example also shows why a high posterior threshold is not synonymous with high probability of acting at every beneficial effect.
Using Simulation to Assess the Rule
For a realistic Bayesian design, the posterior probability may not reduce to a simple cutoff on a normal estimate. Simulation can then estimate operating characteristics by repeatedly generating data under specified parameter values, applying the complete planned analysis, and recording whether the decision rule acts. The simulation should match the proposed design and analysis closely enough to answer the intended question. It should include clinically meaningful benefit values, values near the decision boundary, and values where acting would be undesirable.
The following hypothetical simulation uses the simple model above to illustrate interpretation. It compares a threshold of 0.95, which corresponds to \(Y\geq 2\Phi^{-1}(0.95)\approx 3.290\), with the stringent threshold corresponding to \(Y\geq 6.616\). Each entry is an estimated proportion acting across 20,000 simulated trials at the stated true effect.
Worked Example: Comparing Simulated Operating Characteristics
Suppose the simulation produces the following hypothetical counts. The estimates are the number of trials meeting the rule divided by 20,000. Values at \(\theta=0\) describe action when there is no treatment effect; values at \(\theta=6\) and \(\theta=8\) describe action at two beneficial effects, all in endpoint units.
| Posterior threshold | True effect \(\theta\) | Trials with action | Estimated action probability |
|---|---|---|---|
| 0.95 | 0 | 1,005 of 20,000 | 0.0503 |
| 0.95 | 6 | 18,232 of 20,000 | 0.9116 |
| 0.95 | 8 | 19,812 of 20,000 | 0.9906 |
| Approximately 0.9995 | 0 | 10 of 20,000 | 0.0005 |
| Approximately 0.9995 | 6 | 7,580 of 20,000 | 0.3790 |
| Approximately 0.9995 | 8 | 15,111 of 20,000 | 0.7556 |
For example, at threshold 0.95 and \(\theta=6\), the estimate is \(18{,}232/20{,}000=0.9116\). At the more stringent threshold and \(\theta=8\), it is \(15{,}111/20{,}000=0.75555\), reported as 0.7556. The lower threshold acts more often both at zero effect and at beneficial effects. Neither row alone establishes which rule is preferable: the acceptability of missed benefit, false action, and uncertainty in these estimates must be considered in the decision context.
Simulation estimates also have Monte Carlo error. For \(M\) independent simulated trials and an estimated action probability \(\hat r\), the approximate Monte Carlo standard error is \(\sqrt{\hat r(1-\hat r)/M}\). For \(\hat r=0.0503\) and \(M=20{,}000\), this is about 0.00155. This measures simulation noise, not uncertainty about the clinical effect or the validity of the model. More simulation replicates reduce Monte Carlo error but do not correct misspecified assumptions.
What a Defensible Submission Should Explain
A threshold rationale should make its chain of reasoning reviewable. It should identify the clinical state that counts as success, justify the margin defining that state, explain the threshold in relation to the consequences and evidentiary standard for the intended decision, and show how the complete rule performs across relevant scenarios. The posterior model and prior are part of that rule; the earlier material on prior justification and sensitivity analysis is relevant to assessing whether the resulting posterior probabilities are credible.
Operating characteristics should be reported with enough detail to reproduce and interpret them: the simulated scenarios, data-generating assumptions, design features, analysis applied in each replicate, decision rule, number of replicates, and estimated probabilities. Displaying only a favorable scenario is inadequate. Include scenarios where the treatment has no benefit, effects near the clinical boundary, and effects considered worthwhile. Where estimates are near a desired operating target, report enough precision to show whether simulation error could affect the interpretation.
The intended evidentiary standard is broader than a numerical posterior cutoff. A posterior probability above \(q\) is a result under a specified model and data set; it is not a universal probability that a regulatory decision is correct. Review also concerns whether the clinical question and endpoint are appropriate, whether the analysis is justified, and whether the full evidence supports the requested action. A Bayesian decision rule can organize that evidence, but the threshold alone cannot substitute for the assessment.
Diagnostics, Alternatives, and Limitations
Warning signs include a threshold selected after results are known; a success region defined only as “greater than zero” when a meaningful benefit margin is required; operating characteristics reported at too few or only favorable parameter values; and simulations that omit features of the planned design or analysis. Another warning sign is treating the estimated action probability under a true effect as if it were the posterior probability for an individual trial. These quantities answer different questions.
The expected-loss approach makes trade-offs explicit, but it can be fragile when losses are poorly specified or omit important outcomes. A fixed posterior threshold is simpler to communicate, but its clinical meaning depends on the event whose probability is calculated and the model that produces the posterior. Frequentist type I error and power provide familiar repeated-sampling summaries for a testing procedure; they can complement Bayesian operating characteristics, but they do not by themselves justify a Bayesian posterior cutoff. A credible interval may communicate posterior uncertainty, yet a decision based on whether an interval crosses a margin is still a decision rule whose consequences and operating behavior need examination.
No single threshold is appropriate for every endpoint, product, or regulatory action. Any numerical choice is conditional on the clinical question, model, prior, data quality, and consequences under consideration. The strongest defense is therefore not “this is the standard threshold,” but a transparent account of why this rule is suitable for this decision and what its known limitations are.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- In the simplified two-state loss model, what threshold follows if \(L_{FP}=4\) and \(L_{FN}=6\)?
- Why is \(P_\theta\{R(D)=\text{act}\}\) not the same quantity as \(P(\theta\in A\mid D)\)?
- Under the normal example with \(Y\mid\theta\sim N(\theta,2^2)\), what data cutoff corresponds to a posterior threshold of 0.95 for \(P(\theta>0\mid Y)\)?
- A submission reports favorable simulated action probability at one large beneficial effect, but none under zero effect or near the clinical boundary. What is missing?
Solutions. 1. The threshold is \(4/(4+6)=0.40\), obtained by comparing \(4(1-p)\) with \(6p\). 2. The posterior probability conditions on the observed data and describes uncertainty about \(\theta\); the operating characteristic averages the decision over hypothetical data sets generated at a fixed true \(\theta\). 3. The cutoff is \(2\Phi^{-1}(0.95)\approx 2(1.64485)=3.290\) endpoint units. 4. The simulation omits scenarios needed to evaluate false action and performance near the clinically relevant boundary, so it cannot adequately characterize the trade-offs of the rule.