Why the Event Must Match the Decision
A multi-arm trial can pose different questions after observing the data. Is there evidence that at least one experimental arm succeeds? Or is there evidence that every arm in a specified group succeeds? These are not alternative wordings for the same target. They are distinct posterior events, and they can have very different probabilities even when they concern the same arms.
For example, an “at least one” event may be appropriate when the decision is whether to advance one promising arm into further evaluation. An “every arm” event may be appropriate when a decision or claim requires success across all arms in a prespecified set. The event should therefore be defined to match the decision, rather than chosen after looking at which probability is more favourable.
Let \(D\) denote the observed data and let \(\delta_j\) be the treatment effect for arm \(j\) relative to its control, on a scale where larger values represent more benefit. Let \(\Delta_j\) be the prespecified minimum clinically meaningful benefit for that arm. Define the arm-specific success event as \(S_j=\{\delta_j>\Delta_j\}\). Other scientific definitions are possible, but the event must be specified clearly and its direction and clinical margin must suit the outcome.
These definitions concern a specified collection of arms. Changing that collection changes the event. Likewise, if success requires both clinically meaningful benefit and absence of unacceptable harm, define \(S_j\) to include both conditions before evaluating either multi-arm event.
Calculating the Events From Joint Posterior Information
The posterior distribution must describe the arm effects jointly, not just one arm at a time. For two arms, the probability that at least one succeeds follows from inclusion–exclusion: add the two marginal success probabilities, then subtract the probability that both succeed, which was counted twice. The probability that both succeed is their joint posterior probability.
A short justification uses indicator variables. For any outcome, the indicator of a union is the sum of the two event indicators minus the indicator of their intersection. Taking the posterior expectation of both sides gives the probability identity.
For \(K\) arms, the all-arm event is the intersection of all \(K\) success events. The any-arm event is the complement of the event that no arm succeeds:
Neither expression generally reduces to a calculation using only the marginal probabilities. For instance, multiplying marginal probabilities to obtain an intersection probability assumes posterior independence. That assumption must be justified; it should not be inferred from having separate treatment arms. With a shared control, arm-effect posteriors can be dependent because the contrasts share information from the control, and the joint model or joint posterior draws should retain that dependence.
Worked Example: Two Arms and Two Different Decisions
Suppose 1,000 hypothetical draws from the joint posterior classify each of two arms as successful or not. Arm 1 succeeds in 700 draws, arm 2 in 600, and both succeed in 500. The four mutually exclusive patterns are therefore: both succeed, 500 draws; only arm 1 succeeds, 200; only arm 2 succeeds, 100; neither succeeds, 200. These counts sum to 1,000.
The marginal posterior probabilities are \(P(S_1\mid D)=700/1000=0.70\) and \(P(S_2\mid D)=600/1000=0.60\). In particular, the arm 1 total is the 200 draws with only arm 1 success plus the 500 draws with both successes: \(0.20+0.50=0.70\).
For the disjunctive event, use inclusion–exclusion:
The empirical posterior probability that at least one arm succeeds is \(800/1000=0.80\), because all draws except the 200 with neither success satisfy the event. For the conjunctive event, only the 500 draws with both successes qualify, so \(P(S_{\mathrm{all}}\mid D)\approx 500/1000=0.50\).
If a prespecified decision policy uses a posterior probability criterion of 0.75, the any-arm event meets that criterion, while the all-arm event does not. The corresponding conclusions are different: the posterior probability supports an “at least one” decision at that criterion, but it does not support a decision requiring both arms to succeed. The criterion is part of the decision policy, not a consequence of the probability calculation.
More Than Two Arms
For several arms, a joint posterior simulation is a direct way to estimate both events. In each draw, evaluate every arm-specific success condition, then record whether at least one or all of the conditions hold. The fraction of draws satisfying an event estimates its posterior probability. This preserves the dependence among arms automatically, provided the draws come from the joint posterior model.
Worked Example: Three Arms With a Joint Posterior Table
Consider 1,000 hypothetical joint posterior draws for three arms. In each draw, the pattern records which arms meet their prespecified success criteria.
| Successful arms in the draw | Number of draws |
|---|---|
| 1, 2, and 3 | 360 |
| 1 and 2 only | 140 |
| 1 and 3 only | 100 |
| 2 and 3 only | 80 |
| 1 only | 100 |
| 2 only | 70 |
| 3 only | 50 |
| None | 100 |
The counts total \(360+140+100+80+100+70+50+100=1000\). The any-arm event occurs in every draw except the 100 in which none succeeds, so its estimated posterior probability is \(900/1000=0.90\). The all-arm event occurs only in the 360 draws in which all three succeed, giving \(360/1000=0.36\).
As a check, the marginal success probabilities are \(P(S_1\mid D)\approx(360+140+100+100)/1000=0.70\), \(P(S_2\mid D)\approx(360+140+80+70)/1000=0.65\), and \(P(S_3\mid D)\approx(360+100+80+50)/1000=0.59\). None of these marginal values alone gives the all-arm probability; that probability depends on how often all three successes occur together. With a 0.80 criterion, the any-arm event meets the criterion and the all-arm event does not.
Dependence, Diagnostics, and Monte Carlo Error
A frequent diagnostic is to check the event counts against the arm-level counts. For two arms, the intersection cannot exceed either marginal probability, and the union must be at least as large as either marginal probability. These checks follow from set inclusion: \(S_1\cap S_2\) is contained in each success event, while each success event is contained in \(S_1\cup S_2\). Event counts should also partition the draws without omission or double counting.
The following example shows how a plausible-looking independence calculation can change the decision.
Worked Example: Checking an Independence Shortcut
Suppose joint posterior draws for two hypothetical arms estimate \(P(S_1\mid D)=0.74\), \(P(S_2\mid D)=0.67\), and \(P(S_1\cap S_2\mid D)=0.58\). The observed joint estimate is consistent with both marginal probabilities because \(0.58\) is below \(0.74\) and \(0.67\).
Using the joint estimate, the any-arm probability is \(0.74+0.67-0.58=0.83\), and the all-arm probability is \(0.58\). If one incorrectly assumes posterior independence, the estimated intersection would instead be \(0.74\times0.67=0.4958\), and the estimated any-arm probability would be \(0.74+0.67-0.4958=0.9142\). These are not the probabilities indicated by the joint draws. At a 0.90 criterion, the shortcut would put the any-arm probability above the criterion, whereas the joint calculation would put it below.
Thus, dependence is not a technical detail that can be ignored when a decision is near its criterion. Use the joint posterior distribution or joint draws unless independence is a defensible feature of the model.
When probabilities are estimated by simulation, finite simulation size adds Monte Carlo error. For \(M\) independent posterior draws and an event observed in a fraction \(\hat{p}_{\mathrm{MC}}\), an approximate Monte Carlo standard error is \(\sqrt{\hat{p}_{\mathrm{MC}}(1-\hat{p}_{\mathrm{MC}})/M}\). For correlated MCMC draws, the effective sample size for the event indicator, rather than the raw number of iterations, is relevant to precision. If an estimated probability is close to a decision criterion, check whether simulation error could plausibly change which side of the criterion it falls on.
Worked Example: Assessing Simulation Precision
Suppose 5,000 hypothetical posterior draws yield 3,700 successes for arm 1, 3,350 for arm 2, and 2,900 joint successes. The number satisfying at least one success is \(3700+3350-2900=4150\). Thus the estimated any-arm probability is \(4150/5000=0.83\), and the estimated all-arm probability is \(2900/5000=0.58\).
Treating these as independent posterior draws, the approximate Monte Carlo standard error for the any-arm estimate is
For the all-arm estimate, it is
At a 0.80 criterion, the estimated any-arm probability exceeds the criterion by 0.03, several approximate Monte Carlo standard errors. This suggests that ordinary simulation error alone is unlikely to reverse the comparison for independent draws. It does not describe all uncertainty in the analysis: the posterior probabilities still depend on the chosen model, prior, data, and event definitions.
Choosing and Interpreting the Decision Event
A disjunctive decision can be appropriate when the action is to identify or advance one or more promising arms. It does not establish that all arms succeed, nor does the event probability say which arm should be selected; that requires an arm-specific decision rule. A conjunctive decision is appropriate when success of the whole specified set is required. Its probability can be much lower because a single failure makes the event false.
The posterior probability is conditional on the observed data and the model. It is not the probability that the model is correct, and a chosen posterior criterion does not, by itself, state a frequentist error rate. Similarly, a high any-arm probability does not imply high probabilities for every arm. These interpretations must remain tied to the exact event evaluated.
A useful workflow is:
Specify the effect scale, direction, clinical margin, and any additional conditions for \(S_j\).
Use the union if the question is whether at least one specified arm succeeds; use the intersection if every specified arm must succeed.
Calculate from a joint posterior distribution or joint draws, retaining dependence among arms.
Compare the event probability with the prespecified criterion and report which event the resulting decision concerns.
The main alternative is to report only marginal arm probabilities. Those are useful for describing each arm, but they do not answer a joint any-arm or all-arm question unless supplemented by dependence information. Conversely, the joint event alone may conceal which arm contributes most to an any-arm conclusion. Reporting relevant marginal probabilities alongside the target joint probability can make the decision more transparent.
Common Pitfalls and Limitations
- Confusing the two events. A large probability that at least one arm succeeds does not establish that every arm succeeds. State “at least one” or “all” explicitly.
- Multiplying marginals without justification. The product of arm-specific probabilities gives an intersection probability only under posterior independence. Shared controls and joint prior structures can induce dependence.
- Using marginal probabilities to reconstruct an unknown intersection. Without joint information, the intersection is not generally determined. For two events with marginal probabilities \(p_1\) and \(p_2\), it can range from \(\max(0,p_1+p_2-1)\) to \(\min(p_1,p_2)\).
- Changing the event after seeing the results. Selecting the arm set, success margin, or event definition to favour the observed data undermines the intended interpretation.
- Ignoring simulation and model limitations. A precise numerical estimate from posterior draws does not remedy poor convergence, inadequate effective sample size, a misspecified model, or an unsuitable success definition.
These probabilities answer precisely formulated posterior questions; they do not replace scientific judgment about which arms, margins, and actions matter. A sound report names the event, gives its posterior probability, identifies the arm set and success criteria, and states the decision that the probability is intended to inform.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Two arms have posterior success probabilities 0.65 and 0.55, with joint success probability 0.35. Find the posterior probabilities of at least one success and of both successes.
- For three arms, the posterior probability that none succeeds is 0.18. What is the posterior probability that at least one succeeds?
- For two arms, the marginal success probabilities are 0.80 and 0.60, but the joint probability is unknown. Give the possible range for the joint success probability and explain whether the any-arm probability is determined.
- Explain why a decision requiring every one of three arms to succeed should use a different posterior event from a decision that advances the trial if any arm succeeds.
Solutions. (1) The any-arm probability is \(0.65+0.55-0.35=0.85\); the both-success probability is \(0.35\). (2) The any-arm probability is the complement of none succeeding: \(1-0.18=0.82\). (3) The intersection is between \(\max(0,0.80+0.60-1)=0.40\) and \(\min(0.80,0.60)=0.60\). Since the union is \(0.80+0.60-P(S_1\cap S_2\mid D)\), it ranges from 0.80 to 1.00 and is not determined by the marginals alone. (4) The first decision concerns the union of the three success events; the second concerns their intersection. The intersection holds only when all three succeed, so its probability and interpretation differ from those of the union.