A Three-Arm Thompson Sampling Calculation

Why a Three-Arm Calculation Is Different

With three treatments, a posterior best-arm probability must account for uncertainty in all three response rates at once. It is not enough to compare posterior means, nor is it enough to calculate how often one treatment beats each other treatment separately without accounting for their joint comparison. The calculation is still manageable: condition on one treatment’s possible response rate, then use the other two treatments’ posterior cumulative distribution functions.

Assume a binary outcome, coded as success or failure. Treatment \(i\) has an unknown success probability \(p_i\), for \(i=1,2,3\). Given \(p_i\), the outcomes assigned to treatment \(i\) are independent Bernoulli observations with success probability \(p_i\). Suppose treatment \(i\) has \(s_i\) observed successes and \(f_i\) observed failures, and starts with an independent prior \(p_i\sim\operatorname{Beta}(\alpha_i,\beta_i)\), where \(\alpha_i,\beta_i>0\).

The usual likelihood and prior assumptions are important here. The response probability for an arm is treated as stable over the observations being analyzed, and outcomes are conditionally independent given their arm-specific response probabilities. The allocation rule may depend on previously observed outcomes, as in an adaptive trial; when that rule is based on observed history and does not otherwise involve the unknown response probabilities, the response likelihood for the observed assignments and outcomes has the same arm-specific Bernoulli form. The stated independent priors then give independent arm posteriors.

Update Each Arm, Then Compare Posterior Draws

Beta-Bernoulli conjugacy gives the posterior for each arm by adding its observed successes to the prior’s first shape parameter and its failures to the second. Write \(a_i=\alpha_i+s_i\) and \(b_i=\beta_i+f_i\). The resulting posterior distribution is \(\operatorname{Beta}(a_i,b_i)\). Define its density as \(f_i(x)\) and its cumulative distribution function as \(F_i(x)\).

For treatment 1 to be best, its posterior value must exceed both \(p_2\) and \(p_3\). If its value is \(x\), independence makes the probability that both other values are below it \(F_2(x)F_3(x)\). Averaging this conditional probability over treatment 1’s posterior density gives the required probability. The same argument applies to each arm.

This is a one-dimensional integral for each arm, even though there are three treatments. It is essential to use the CDFs of both competing arms: the event being calculated is that arm \(i\) exceeds both of them. A useful check follows because exactly one arm is best with probability one. Thus \(q_1+q_2+q_3=1\), up to numerical rounding.

A Calculation With Three Different Posteriors

Derivation and Checks

The integral formula follows by conditioning on the posterior value of the arm being evaluated. For example, let \(A_1\) be the event that treatment 1 is best. Conditional on \(p_1=x\), the event \(A_1\) requires \(p_2<x\) and \(p_3<x\). Since the posterior distributions are independent, the conditional probability is \(F_2(x)F_3(x)\). Integrating over the possible values \(x\) under treatment 1’s posterior gives:

The independence assumption is used in the middle equality. If posterior dependence between arms is introduced, the joint probability that the competing rates are below \(x\) need not equal the product of their CDFs. The general calculation would then require their joint posterior distribution.

A second justification is the sum-to-one check. The three events “arm 1 is best,” “arm 2 is best,” and “arm 3 is best” are mutually exclusive and exhaustive when the posterior values are continuous: exactly one is largest, and ties have probability zero. Therefore their probabilities sum to one. A computed total far from one signals an algebra, integration, or numerical error; a small discrepancy can arise from rounding.

Interpretation, Alternatives, and Failure Modes

The \(q_i\) values describe posterior uncertainty about which response rate is largest. They are not the posterior means \(E(p_i\mid\text{data})\), and they are not probabilities that the null hypothesis for a treatment comparison is true. In a Thompson sampling allocation, a larger \(q_i\) gives that arm a larger chance of receiving the next assignment, but even an arm with the largest \(q_i\) need not receive every assignment.

A plug-in alternative would rank arms by their posterior means, such as \(a_i/(a_i+b_i)\), and choose the largest. That gives a single ranking and does not preserve the uncertainty reflected in the posterior best-arm probabilities. Pairwise posterior probabilities, such as \(P(p_1>p_2\mid\text{data})\), answer a different question: they do not by themselves determine the probability that treatment 1 beats both treatments 2 and 3. The product-of-CDF integral is the appropriate calculation for the three-way “best” event under independent posteriors.

When the posterior parameters are positive integers, the Beta densities and CDFs are polynomials, so the integrals can often be evaluated exactly as above. For other shape parameters, the same one-dimensional integrals remain valid, but numerical integration may be needed. In either case, check that each integrand is nonnegative on \([0,1]\), that each resulting probability lies between 0 and 1, and that the probabilities sum to one within numerical tolerance.

• Using only posterior means. A posterior mean does not encode the whole posterior distribution, so it cannot replace the best-arm probability calculation.
• Comparing with only one competitor. The event that arm \(i\) is best requires it to exceed both other arms; omitting one CDF overstates the event probability.
• Using a density for a competitor. The competing arm must be below a threshold \(x\), which is a CDF event. Its density alone is not the probability of being below \(x\).
• Ignoring the prior or swapping success and failure counts. The posterior shapes are \(\alpha_i+s_i\) and \(\beta_i+f_i\), respectively. Reversing these counts changes the posterior and the ranking probabilities.
• Assuming independence without justification. The product of two CDFs requires independent posterior arm probabilities. Dependence calls for a joint posterior calculation.

Finally, the calculation describes the posterior conditional on the model, priors, and observed data. If response probabilities change over time, outcomes are not conditionally independent, or the priors are not independent, the stated Beta posteriors and product formula may no longer apply. A numerical result cannot repair a model that does not fit the trial’s assumptions.