Randomized Binary Efficacy With a Posterior Superiority Rule

From Randomization to a Posterior Superiority Probability

In a randomized efficacy trial with a binary endpoint, each participant either experiences the defined response or does not. The main statistical question considered here is whether the response probability under experimental treatment exceeds that under control. A posterior superiority rule expresses the evidence for this direction as a probability conditional on the observed data and the specified model and prior.

The endpoint and analysis population must be defined before calculating the probability. For example, the endpoint might be response by a specified visit, analyzed by randomized assignment. Randomization supports a causal comparison of assigned treatment groups when the trial is properly conducted, but the model still summarizes the response probabilities for the population represented by the trial. It does not repair biased outcome ascertainment, substantial missingness handled inappropriately, or departures from the intended treatment strategy.

Model and Posterior Updating

Suppose \(n_T\) and \(n_C\) participants are analyzed in the experimental and control arms, with \(x_T\) and \(x_C\) responses. A basic model treats outcomes as independent Bernoulli observations within each arm and assumes a common response probability within each arm:

Use independent beta priors \(p_T\sim\operatorname{Beta}(a_T,b_T)\) and \(p_C\sim\operatorname{Beta}(a_C,b_C)\), where all four shape parameters are positive. Independence here is a prior modeling choice: before observing the trial data, the two response probabilities are modeled independently. The likelihood also assumes participants’ binary outcomes are conditionally independent given their arm’s response probability. The beta prior is conjugate to the binomial likelihood, so each posterior is another beta distribution.

For an arm with posterior \(\operatorname{Beta}(a,b)\), its posterior mean is \(a/(a+b)\). Thus the posterior mean risk difference is the experimental-arm posterior mean minus the control-arm posterior mean. The posterior distribution of \(\delta\) is not generally a beta distribution: it is the distribution of the difference between two beta-distributed probabilities.

The superiority probability can be computed by integrating the experimental posterior density against the control posterior cumulative distribution function:

There is generally no simple closed-form expression needed for routine use. Numerical integration or posterior simulation can evaluate this probability. With simulation, draw \(p_T^{(m)}\) and \(p_C^{(m)}\) independently from their respective posterior distributions and estimate the probability by the fraction of draws for which \(p_T^{(m)}>p_C^{(m)}\).

Worked Analyses

Why the Calculation Works

For one arm, if there are \(x\) responses among \(n\) participants, the binomial likelihood as a function of \(p\) is proportional to \(p^x(1-p)^{n-x}\). A \(\operatorname{Beta}(a,b)\) prior has density proportional to \(p^{a-1}(1-p)^{b-1}\). Multiplying prior density by likelihood gives:

This is the kernel of a \(\operatorname{Beta}(a+x,b+n-x)\) density, which justifies the posterior update. In the two-arm model, the likelihood factors into one term for each arm and the priors are independent. The joint posterior therefore also factors into the two beta posteriors. The superiority probability then follows by conditioning on \(p_T=u\): for each value \(u\), the chance that \(p_C<u\) is \(F_C(u\mid\text{data})\), which is averaged over the posterior distribution of \(p_T\).

Applying the Rule Carefully

A posterior probability rule compares \(P(p_T>p_C\mid\text{data})\) with a cutoff selected for the trial. For example, a protocol could specify that the evidence meets its superiority criterion when this probability is at least 0.95. That number is an illustrative choice, not a universal regulatory standard. It should be set before examining the trial result and justified in the design context.

The event \(p_T>p_C\) concerns any positive difference, however small. It is distinct from an event such as \(\delta>\delta_{\min}\), where \(\delta_{\min}>0\) is a clinically meaningful improvement specified in advance. A high probability of any superiority does not by itself imply a high probability of clinically important benefit. Likewise, the posterior probability is not the probability that a null hypothesis is true, nor is it a frequentist \(p\)-value.

The rule’s result depends on the prior as well as the likelihood. The uniform prior used in the examples is simple, but it is not automatically appropriate for every endpoint or trial. Review the prior’s implications for plausible response rates and treatment differences, and assess whether reasonable alternatives change the conclusion. This is especially important with small samples, extreme response counts, or substantial prior information.

Assumptions, Diagnostics, and Alternatives

Before relying on the calculation, verify that the analysis matches the trial question. In particular, check that response is defined at a common time point, the randomized groups are analyzed as planned, and missing outcomes are addressed by a defensible strategy. Treating unobserved outcomes as if they were ordinary nonresponses can distort the response probabilities unless that coding is the prespecified endpoint definition. The simple model also assumes that participants within an arm share a response probability; important center, subgroup, or clustering variation may require a richer model.

Because conjugate beta updating is analytic, there is no Markov chain Monte Carlo convergence problem for the basic model. If posterior simulation is used to calculate superiority, check that the Monte Carlo standard error is small relative to the precision needed for the decision. For \(M\) independent posterior draws and estimated probability \(\hat q\), the approximate Monte Carlo standard error is \(\sqrt{\hat q(1-\hat q)/M}\). Increase \(M\) if this uncertainty could change whether the cutoff is crossed. Also verify the response counts, arm labels, prior parameters, and that the simulated event is \(p_T>p_C\), not its reverse.

A frequentist analysis might instead report an estimated risk difference with a confidence interval and a test of a null hypothesis. Such an analysis answers related but differently framed questions and has different probability interpretations. A Bayesian analysis with a different prior may also produce a different posterior probability. For the present purpose, the essential comparison is that the posterior superiority probability directly summarizes the modeled probability of the directional event, while still depending on the assumed prior and likelihood.

• Common mistake: treating a positive posterior mean as sufficient. A positive mean risk difference can coexist with substantial posterior probability of no superiority, as the second example illustrates. Apply the prespecified probability cutoff rather than judging from the sign of the mean alone.
• Common mistake: calling \(P(p_T>p_C\mid\text{data})\) the probability that treatment works. It is conditional on a model, prior, endpoint, and data, and concerns a particular comparison. State those conditions and avoid implying certainty or universal benefit.
• Common mistake: equating superiority with meaningful benefit. Any positive difference counts in \(p_T>p_C\). Clinical importance requires interpretation of the effect’s size and clinical consequences.

The basic two-binomial model is transparent and useful for an unadjusted randomized comparison, but it does not automatically handle covariate adjustment, missing-data uncertainty, clustering, nonadherence estimands, or heterogeneous treatment effects. If these features matter, the analysis must be expanded and its posterior superiority probability defined for the intended estimand.