Why Three Arms Require More Than Two Comparisons
A randomized trial with two experimental treatments and one control produces two treatment-control comparisons. Each comparison can have a posterior probability of clinically meaningful benefit, but those probabilities are not generally independent: both contrasts use the same uncertain control response rate. The decision rule therefore needs to match the trial’s objective. Declaring each treatment successful separately, requiring both to succeed, and declaring success if either succeeds are different rules.
The focus here is a fixed, binary endpoint measured for each randomized participant, such as response by a prespecified assessment time. The calculations use independent binomial likelihoods and independent beta priors. This is a transparent model for illustrating multiplicity; it is not appropriate without modification for every clinical endpoint or trial design.
With this model, the posterior distribution for each response probability is beta. For \(y_j\) observed responses, the update is \(p_j\mid Y_j=y_j\sim\operatorname{Beta}(a_j+y_j,b_j+n_j-y_j)\). Define \(D_1=p_1-p_0\) and \(D_2=p_2-p_0\), the treatment-control differences in response probability. A clinically relevant margin \(\delta\) is expressed in probability units; for example, \(\delta=0.05\) means a five-percentage-point improvement.
Calculating a Treatment-Control Posterior Probability
The posterior probability \(q_j\) is a probability about the unknown response probabilities conditional on the model, prior, and observed data. Because it concerns a difference of two beta-distributed quantities, it does not in general have a simple beta form. It can be calculated by numerical integration or posterior simulation. A normal approximation based on the posterior mean and variance can be useful for a quick, transparent calculation, but its accuracy should be checked when the posterior is skewed or concentrated near a probability boundary.
Worked Example: One Contrast With a Clinically Relevant Margin
Suppose a hypothetical trial has 60 participants in treatment 1, of whom 30 respond, and 60 control participants, of whom 18 respond. Use independent \(\operatorname{Beta}(1,1)\) priors for the two response probabilities. Calculate approximately the posterior probability that treatment 1 improves response by more than five percentage points.
The assumptions for the beta-binomial update are that each arm’s responses are binomial with a constant arm-specific probability, the outcomes are observed as defined, and the priors are independent and uniform. The posterior distributions are \(p_1\mid\text{data}\sim\operatorname{Beta}(31,31)\) and \(p_0\mid\text{data}\sim\operatorname{Beta}(19,43)\). Their posterior means are \(31/62=0.5000\) and \(19/62=0.3065\), so the mean difference is \(0.1935\).
For a beta random variable with parameters \(a,b\), the variance is \(ab/[(a+b)^2(a+b+1)]\). Thus the two posterior variances are \(31(31)/(62^2\cdot63)=0.003968\) and \(19(43)/(62^2\cdot63)=0.003374\). Under a normal approximation to the posterior difference, its standard deviation is \(\sqrt{0.003968+0.003374}=0.08569\). Standardizing the five-percentage-point margin gives \((0.1935-0.05)/0.08569=1.675\). The resulting normal-approximation probability is \(P(D_1>0.05\mid\text{data})\approx\Phi(1.675)=0.953\).
This approximate calculation puts the posterior probability just above a hypothetical threshold of \(0.95\). It is not an exact evaluation of the difference of beta variables. A final decision near a threshold should use numerical integration or posterior simulation from the beta distributions and report the method and its numerical precision.
The Shared Control Creates Dependence
Even with independent priors and independent arm outcomes, the treatment-control differences are dependent in the posterior because they share \(p_0\). The dependence matters when the question concerns both comparisons together or asks whether at least one treatment is successful.
This result also gives a useful check for a joint normal approximation. The approximate posterior correlation between the differences is \(\operatorname{Var}(p_0\mid\text{data})/ \sqrt{\{\operatorname{Var}(p_1\mid\text{data})+\operatorname{Var}(p_0\mid\text{data})\} \{\operatorname{Var}(p_2\mid\text{data})+\operatorname{Var}(p_0\mid\text{data})\}}\). It is positive under this model. Treating the two differences as independent can misstate the posterior probability that both, or at least one, exceed the margin.
Worked Example: Comparing Both Marginal Probabilities
Consider a second hypothetical trial with 65 participants in each arm. There are 31 responses in treatment 1, 24 in treatment 2, and 19 in control. Again use independent \(\operatorname{Beta}(1,1)\) priors, assume the binomial conditions above, and set \(\delta=0.05\). Calculate approximate marginal probabilities for both treatment-control differences.
The posteriors are \(p_1\sim\operatorname{Beta}(32,35)\), \(p_2\sim\operatorname{Beta}(25,42)\), and \(p_0\sim\operatorname{Beta}(20,47)\), conditional on the data. Their means are \(32/67=0.4776\), \(25/67=0.3731\), and \(20/67=0.2985\). Thus the posterior mean differences are \(0.1791\) for treatment 1 and \(0.07463\) for treatment 2.
The beta posterior variances are \(1120/(67^2\cdot68)=0.003669\) for treatment 1, \(1050/(67^2\cdot68)=0.003440\) for treatment 2, and \(940/(67^2\cdot68)=0.003079\) for control. For treatment 1, the approximate standard deviation of the difference is \(\sqrt{0.003669+0.003079}=0.08215\), giving \(z=(0.1791-0.05)/0.08215=1.572\) and \(q_1\approx\Phi(1.572)=0.942\).
For treatment 2, the approximate standard deviation is \(\sqrt{0.003440+0.003079}=0.08074\). Hence \(z=(0.07463-0.05)/0.08074=0.305\), and \(q_2\approx\Phi(0.305)=0.620\). With a marginal threshold \(c=0.90\), treatment 1 meets the rule and treatment 2 does not. These are separate marginal decisions, not a calculation of the posterior probability that both treatments succeed.
For scale, the approximate correlation between the two differences is \(0.003079/\sqrt{(0.003669+0.003079)(0.003440+0.003079)}\approx0.464\). This calculation uses the shared control variance in the numerator. It is a useful diagnostic against an incorrect independence assumption.
What the Shared Success Rule Does—and Does Not—Say
A rule that compares each \(q_j\) with the same threshold \(c\) is easy to communicate, but its meaning depends on the trial’s decision objective. If the objective is to select any treatment with sufficient evidence of a clinically meaningful advantage, separate marginal rules may be suitable, subject to prespecified decision consequences. If the objective is evidence that both treatments work, the relevant quantity is the joint posterior probability \(P(D_1>\delta,D_2>\delta\mid\text{data})\). If the objective is evidence that at least one works, the relevant probability is \(P(D_1>\delta\text{ or }D_2>\delta\mid\text{data})\). These are not interchangeable.
For the preceding example, suppose posterior simulation produces 10,000 draws, with 5,800 draws in which both contrasts exceed the margin, 3,600 in which only treatment 1 exceeds it, 400 in which only treatment 2 exceeds it, and 200 in which neither does. These hypothetical simulation counts imply estimated marginal probabilities \(q_1=(5800+3600)/10000=0.940\) and \(q_2=(5800+400)/10000=0.620\), an estimated joint probability of \(0.580\), and an estimated at-least-one probability of \((5800+3600+400)/10000=0.980\). The marginal estimates closely match the normal approximations above; the joint and at-least-one probabilities answer different questions.
This example also illustrates why multiplicity is not a single automatic adjustment in Bayesian analysis. A posterior probability for each contrast is conditional on the specified model and data; applying the same threshold to several contrasts does not itself define a familywise false-positive rate. Nor does a high probability that at least one treatment exceeds the margin establish which treatment is best. The desired claim, the loss of an incorrect decision, and any regulatory operating characteristics should be made explicit. Operating characteristics can be assessed under clinically relevant scenarios, including scenarios where neither or only one treatment is effective.
Specify whether success means each treatment separately exceeds the margin, both do, or at least one does.
Update each arm’s response probability using its likelihood and prior, checking outcome definition and arm-specific assumptions.
Evaluate marginal, joint, or at-least-one probabilities as required; preserve the dependence induced by the shared control.
Assess sensitivity to reasonable prior choices, the clinical margin, numerical approximation, and operating characteristics under relevant scenarios.
Diagnostics, Alternatives, and Limitations
Posterior simulation is a direct way to evaluate these probabilities: draw \(p_0,p_1,p_2\) independently from their arm-specific posterior distributions, calculate both differences for each draw, and count the draws satisfying the specified event. For \(M\) draws, the estimate of a posterior event probability is its satisfying-draw count divided by \(M\). Monte Carlo error is important near a decision threshold; for an estimated probability \(\hat q\), its approximate Monte Carlo standard error is \(\sqrt{\hat q(1-\hat q)/M}\). Increase \(M\) or use a more precise integration method when that error could change the decision. Check simulation output against the posterior means, variances, and positive covariance implied by the model.
The normal approximation is a practical alternative for quick calculations, but it can be poor for small samples, extreme response rates, or posterior distributions with substantial skewness. In those settings, beta-based numerical integration or posterior simulation is preferable. A joint multivariate normal approximation can summarize dependence, but should not be imposed as independence when the control is shared. A different alternative is to define a Bayesian hierarchical model that relates arm response probabilities; that changes the model and the borrowing assumptions and should not be treated as a mere multiplicity correction.
The beta-binomial analysis is limited to the stated binary-outcome model. It does not automatically address missing endpoint data, unequal assessment times, treatment discontinuation, covariate adjustment, or outcome definitions that require a different likelihood. A prior or model that borrows information across arms can also change the posterior dependence and the contrast probabilities. Finally, the posterior threshold and clinical margin are decision inputs, not facts determined by the observed response counts. They should be justified for the intended clinical and regulatory use.
Common Pitfall
Check Your Understanding
Answer each question, then compare with the solutions below.
- With independent priors and arm-specific binomial likelihoods, why are the two treatment-control differences positively correlated in the posterior?
- For a posterior simulation with 20,000 draws, 11,000 draws show both contrasts exceeding the margin, 5,000 show only contrast 1 exceeding it, and 2,000 show only contrast 2 exceeding it. What are the estimated marginal probabilities and the estimated at-least-one probability?
- Why is a posterior threshold such as 0.95 not automatically a 5% frequentist familywise error rate?
- Name one diagnostic that helps assess whether a normal approximation is adequate for a treatment-control difference.
Solutions. (1) Both differences subtract the same random control probability, so their posterior covariance equals \(\operatorname{Var}(p_0\mid\text{data})\), which is positive unless the control probability is known with certainty. (2) The marginal estimates are \(q_1=(11000+5000)/20000=0.800\) and \(q_2=(11000+2000)/20000=0.650\). The at-least-one estimate is \((11000+5000+2000)/20000=0.900\). (3) A posterior threshold is a conditional probability decision criterion under a specified model and prior; a frequentist familywise error rate is a repeated-sampling probability under specified null scenarios. They are different quantities. (4) Compare the beta posterior’s shape or simulated difference distribution with the normal approximation, especially checking skewness and whether appreciable probability lies near the response-probability boundaries.