Selecting Posterior Probability Thresholds

Why Choose a Posterior Probability Cutoff?

For a treatment arm, a posterior probability describes uncertainty about a specified event after observing the data. It does not, by itself, determine what action to take. A decision rule adds a cutoff: for example, declare arm \(j\) successful if the posterior probability that its effect exceeds a clinically meaningful target is greater than \(0.80\). Choosing that cutoff is a substantive decision, not a consequence of Bayes' theorem alone.

The cutoff should reflect what “success” means clinically and what follows from a declaration. A high cutoff may reduce declarations when the evidence is uncertain, but it can also make it harder to identify a genuinely beneficial arm. A lower cutoff does the reverse. The appropriate balance depends on the consequences of false declarations and missed benefits, as well as on the posterior probabilities generated by the model and prior.

Definition: Let \(\theta_j\) denote the effect of arm \(j\), oriented so that larger values are more beneficial, and let \(\delta_j\) be the minimum effect regarded as clinically meaningful. Define the success event \(S_j=\{\theta_j>\delta_j\}\) and its posterior probability \(q_j=P(\theta_j>\delta_j\mid \text{data})\). A posterior-probability cutoff \(c\), with \(0<c<1\), declares arm \(j\) successful when \(q_j\ge c\).

The effect scale, direction, and meaning of \(\delta_j\) must be specified before applying the rule. For example, if \(\theta_j\) is a mean improvement measured in points, then \(\delta_j=2\) defines success as an improvement greater than 2 points. If effects are oriented so that smaller values are better, the event must instead be written in that direction. A probability about mere superiority to control is not interchangeable with a probability of clinically meaningful benefit.

Linking a Cutoff to Clinical Consequences

A simple way to make the tradeoff explicit is to distinguish two actions—declare success or do not declare success—and two states—success event \(S_j\) is true or false. Suppose a false declaration has loss \(L_{\mathrm{FP}}>0\), while failing to declare an arm successful when \(S_j\) is true has loss \(L_{\mathrm{FN}}>0\). Assign zero loss to the two correctly classified outcomes. These losses are simplified representations of consequences, not universal measurements of clinical value.

\begin{aligned} \text{Posterior expected loss of declaring success} &=(1-q_j)L_{\mathrm{FP}},\\ \text{Posterior expected loss of not declaring success} &=q_jL_{\mathrm{FN}}. \end{aligned}

Choose the action with the smaller posterior expected loss. Declaring success is preferred when \((1-q_j)L_{\mathrm{FP}}<q_jL_{\mathrm{FN}}\). Rearranging gives the cutoff:

\begin{aligned} (1-q_j)L_{\mathrm{FP}}&<q_jL_{\mathrm{FN}},\\ L_{\mathrm{FP}}&<q_j(L_{\mathrm{FP}}+L_{\mathrm{FN}}),\\ q_j&>\frac{L_{\mathrm{FP}}}{L_{\mathrm{FP}}+L_{\mathrm{FN}}}. \end{aligned}

Result: Under this two-action, two-state loss model, the loss-minimizing cutoff is \(c=L_{\mathrm{FP}}/(L_{\mathrm{FP}}+L_{\mathrm{FN}})\). A larger loss for a false declaration raises the cutoff; a larger loss for missing a truly successful arm lowers it. Equality gives equal expected losses, so a convention is needed if \(q_j=c\) exactly.

This derivation is useful because it makes the direction of the tradeoff transparent. It does not imply that clinical consequences can always be reduced to two numbers, or that these losses are easy to elicit. A trial may instead set a cutoff by clinical policy or by examining how candidate rules behave under plausible scenarios. Whichever approach is used, the rationale should be stated rather than treating a conventional value such as \(0.90\) as automatically correct.

Worked Example: Applying a Cutoff to a Normal Posterior

Suppose a hypothetical arm has effect \(\theta_j\) measured in points, and an improvement above \(\delta_j=0.10\) points is clinically meaningful. Assume the model and prior yield the posterior distribution \(\theta_j\mid\text{data}\sim N(0.18,0.08^2)\). A prespecified decision rule uses \(c=0.80\). Calculate the posterior probability of success and apply the rule.

Check the calculation conditions. The posterior distribution is explicitly supplied, its standard deviation is \(0.08\) points, and success is defined as \(\theta_j>0.10\) points. Thus a normal tail probability is appropriate for this example; the calculation does not require treating the posterior mean as the probability of success.

Standardize the clinically meaningful threshold:

z=\frac{0.10-0.18}{0.08}=-1.00.

Therefore,

q_j=P(\theta_j>0.10\mid\text{data}) =1-\Phi(-1.00)=\Phi(1.00)\approx0.8413.

Since \(0.8413\ge0.80\), this rule declares the arm successful. In context, conditional on the stated model, prior, and observed data, the posterior probability that the arm's effect exceeds \(0.10\) points is about 84.1%. This is not a claim that the fixed effect has an 84.1% frequentist probability of exceeding the threshold.

Prior Distributions Can Change the Decision

A cutoff is applied to a posterior probability, and that probability depends on both the likelihood and the prior. The same observed result can therefore lead to different decisions under different prior distributions. This is not a reason to choose a prior to obtain a preferred answer; it is a reason to state the prior and understand how it contributes to the decision.

Worked Example: Same Data, Two Prior Means

Consider a hypothetical continuous-outcome model with observed estimate \(y=0.20\) points and known standard error \(0.10\) points, where \(y\mid\theta_j\sim N(\theta_j,0.10^2)\). Compare priors \(\theta_j\sim N(0,0.20^2)\) and \(\theta_j\sim N(0.15,0.20^2)\). Define success as \(\theta_j>0.10\) points and use cutoff \(c=0.80\). The normal likelihood, known standard error, and proper normal priors make the normal-normal posterior calculation applicable.

For a prior \(N(\mu_0,0.20^2)\), the posterior variance and mean are

v=\left(\frac{1}{0.10^2}+\frac{1}{0.20^2}\right)^{-1}=0.008, \qquad m=v\left(\frac{0.20}{0.10^2}+\frac{\mu_0}{0.20^2}\right).

The posterior standard deviation is \(\sqrt{0.008}=0.08944\) points. With prior mean \(\mu_0=0\), \(m=0.16\) points, so

P(\theta_j>0.10\mid y) =\Phi\left(\frac{0.16-0.10}{0.08944}\right) =\Phi(0.6708)\approx0.749.

This is below \(0.80\), so the arm is not declared successful under this prior. With prior mean \(\mu_0=0.15\), \(m=0.19\) points, giving

P(\theta_j>0.10\mid y) =\Phi\left(\frac{0.19-0.10}{0.08944}\right) =\Phi(1.006)\approx0.843.

This exceeds \(0.80\), so the arm is declared successful under the second prior. The data and cutoff are unchanged; the different prior means produce different posterior probabilities and hence different decisions. The example illustrates prior dependence, not which prior is clinically or scientifically appropriate.

Applying the Rule Across Several Arms

A trial with several experimental arms can apply the same rule to each arm's marginal posterior probability. If each arm has its own meaningful threshold \(\delta_j\), calculate \(q_j=P(\theta_j>\delta_j\mid\text{data})\) for each arm and compare each probability with the chosen cutoff. Shared controls or other model features may make arm effects posterior-dependent, but the marginal decision for arm \(j\) still uses its own \(q_j\) under this rule.

The cutoff is not a frequentist familywise error-rate adjustment. Applying \(q_j\ge c\) to several arms does not, by itself, guarantee a specified probability of making no false declarations across the trial. Nor does a marginal cutoff answer a joint question such as whether all selected arms are successful. Those are different decision events, requiring their own probabilities or operating characteristics. The earlier material on Bayesian multiplicity without error-rate correction distinguishes Bayesian arm decisions from automatic frequentist multiplicity control.

Worked Example: A Consequence-Based Cutoff for Two Arms

Suppose a hypothetical trial considers two arms, and the decision-makers assign loss \(L_{\mathrm{FP}}=4\) units to falsely declaring an arm successful and \(L_{\mathrm{FN}}=1\) unit to failing to declare a truly successful arm. For one arm, the posterior probability of exceeding its prespecified meaningful-effect threshold is \(q_j=0.76\). For a second arm it is \(q_k=0.86\). Assume the same losses apply to both arms and that each probability is calculated under the specified model and prior.

The implied cutoff is

c=\frac{4}{4+1}=0.80.

For the first arm, the expected loss of declaring success is \((1-0.76)(4)=0.96\) units, while the expected loss of not declaring success is \((0.76)(1)=0.76\) units. Not declaring success has the smaller expected loss, consistent with \(0.76<0.80\). For the second arm, the corresponding losses are \((1-0.86)(4)=0.56\) and \((0.86)(1)=0.86\) units. Declaring success has the smaller expected loss, consistent with \(0.86>0.80\). Thus the rule declares only the second arm successful. This calculation applies the same per-arm rule; it does not establish a familywise error guarantee for the two-arm trial.

Diagnostics, Alternatives, and Limitations

Before relying on a threshold-based decision, check that the probability being thresholded matches the clinical claim. In particular, verify the direction and units of \(\theta_j\), the value and interpretation of \(\delta_j\), the posterior model and prior, and whether the rule is intended for each arm separately or for a joint event. Recalculate probabilities when a threshold lies near the posterior center, where small changes in the model or inputs may change the decision.

Common signs that the rule is being used poorly include a cutoff selected only because it is familiar, a meaningful-effect threshold chosen after seeing which value makes an arm pass, or language that treats \(q_j\) as a frequentist error probability. Another warning is to report only “pass” or “fail” without the posterior probability and clinically meaningful threshold. Reporting \(q_j\), \(c\), \(\delta_j\), and the resulting action makes the decision more interpretable.

The threshold rule is one approach among several. A posterior probability cutoff gives a transparent yes-or-no decision for a defined event. A frequentist significance rule instead controls specified long-run error rates under its design assumptions; it is not obtained merely by choosing a Bayesian cutoff. Reporting posterior probabilities without a binary declaration preserves more information but may leave action unspecified. The simple loss derivation above offers a rationale for a cutoff when two actions and two error consequences are an acceptable representation; it does not replace a fuller assessment when benefits, harms, costs, or multiple possible actions matter.

The rule also has limits. A posterior probability can be poorly calibrated to the scientific question if the likelihood or prior is unsuitable, and a threshold cannot repair a misspecified model. A fixed cutoff may be inappropriate when consequences differ by arm or when decisions have consequences beyond the two states represented in the simple loss model. In addition, two arms that both cross the threshold need not be equally promising, and an arm just below the cutoff is not thereby proven ineffective. Treat the cutoff as a decision boundary, not as a natural dividing line in the evidence.

Key takeaway: A posterior-probability cutoff turns a probability about a clinically defined event into an action. State the event, justify the cutoff in relation to consequences or an explicit decision policy, and report the posterior probability and assumptions. The cutoff alone is neither a multiplicity correction nor a guarantee about long-run error rates.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

An arm has \(P(\theta_j>2\mid\text{data})=0.92\), where \(\theta_j\) is measured in points. A rule uses \(c=0.90\). What is the decision, and what does the probability mean?
If \(L_{\mathrm{FP}}=3\) and \(L_{\mathrm{FN}}=2\), derive the cutoff under the two-action, two-state loss model.
An analyst defines success as \(\theta_j>0\), although the clinically meaningful target is \(0.5\) points. What is the issue with using \(P(\theta_j>0\mid\text{data})\) for a meaningful-benefit decision?
Two arms each have posterior probability \(0.85\) of success and are assessed with cutoff \(0.80\). Does this establish a 5% familywise error rate? Explain.

Solutions. (1) Declare the arm successful because \(0.92\ge0.90\). Conditional on the specified model, prior, and data, the posterior probability that its effect exceeds 2 points is 92%; this is not a frequentist probability that a fixed effect exceeds 2 points. (2) The cutoff is \(3/(3+2)=0.60\), so declaring success is preferred when the posterior probability exceeds 0.60; equality gives equal expected losses. (3) The event \(\theta_j>0\) describes any positive effect, not an effect exceeding the stated clinically meaningful target. The relevant probability is \(P(\theta_j>0.5\mid\text{data})\). (4) No. Applying a marginal posterior cutoff to each arm does not, by itself, specify or guarantee a frequentist familywise error rate.