Three Probabilities, Three Questions
In a binary-outcome trial, “Which treatment is most likely to be best?” is not the same question as “What is the chance of success on this treatment?” The first concerns uncertainty about the ordering of unknown treatment response rates. The second concerns the response rate itself, averaged over its posterior uncertainty. A third quantity—the probability of success for a future patient assigned to an arm—is a predictive probability. Under the usual Bernoulli model it equals the posterior mean response rate for that arm, but it is not the probability that the arm is best.
This distinction matters in Thompson sampling: the probability an arm is selected is its posterior probability of being best, subject to the method’s treatment of ties. That selection probability is not a forecast of the arm’s response rate. A treatment can be selected often because it has a substantial chance of being best, while its predicted success probability remains quite different.
The weak inequality in the definition of \(q_j\) counts a treatment tied for the highest rate as best. When the joint posterior assigns probability zero to all pairwise equality events, ties have probability zero, and the events “treatment \(j\) is best” partition the posterior probability space. Then \(\sum_{j=1}^K q_j=1\). This holds, for example, for independent continuous Beta posteriors. Continuous marginal posteriors alone do not guarantee it: dependence can place positive posterior probability on equal treatment rates.
Why the Predictive Probability Is a Posterior Mean
Use the standard Beta-Bernoulli model: conditional on \(\theta_j\), outcomes on treatment \(j\) are Bernoulli with success probability \(\theta_j\). Suppose the current history gives a posterior distribution for \(\theta_j\). For the next patient, conditional on that unknown rate, the chance of success is \(\theta_j\). Averaging over posterior uncertainty gives the predictive probability:
This is a prediction for a future response on a specified treatment. It does not compare that treatment’s rate with the other rates, which is what \(q_j\) does. If an allocation rule selects treatment \(j\) with probability \(r_j\), the predictive success probability for a patient assigned under that rule is instead the allocation-weighted average \(\sum_j r_jm_j\), provided the allocation is made using the current history before observing that patient’s response.
For a Beta prior \(\theta_j\sim\mathrm{Beta}(a_j,b_j)\), with \(a_j,b_j>0\), and \(s_j\) successes and \(f_j\) failures, the Bernoulli likelihood gives the posterior \(\mathrm{Beta}(a_j+s_j,b_j+f_j)\). Its mean, and therefore the predictive probability, is \((a_j+s_j)/(a_j+b_j+s_j+f_j)\). This conjugate calculation presumes the stated outcome model is appropriate; the formula does not repair an incorrect likelihood or a poorly specified prior.
Worked Example: Best-Arm Chance and Predictive Success
Suppose two hypothetical treatments have independent posteriors \(\theta_1\sim\mathrm{Beta}(2,1)\) and \(\theta_2\sim\mathrm{Beta}(1,2)\). These are continuous distributions, so the posterior probability of a tie is zero.
The densities are \(f_1(x)=2x\) and \(f_2(x)=2(1-x)\) for \(0<x<1\). The cumulative distribution function of \(\theta_2\) is \(F_2(x)=1-(1-x)^2=2x-x^2\). Thus
With no tie probability, \(q_2=1-q_1=1/6\approx0.1667\). The posterior means are \(m_1=2/(2+1)=2/3\approx0.6667\) and \(m_2=1/(1+2)=1/3\approx0.3333\). A future patient assigned to treatment 1 has predictive success probability \(2/3\), not \(5/6\); \(5/6\) is the posterior chance that treatment 1’s rate exceeds treatment 2’s.
If Thompson sampling selects the treatment with the larger posterior draw, its allocation probabilities are \(r_1=q_1=5/6\) and \(r_2=q_2=1/6\). The predictive success probability for a patient assigned under this randomized rule is
This final value averages response rates over the arm selected by the rule. It is distinct from both arm 1’s predictive probability and arm 1’s probability of being best.
Posterior Updating and a Trial Prediction
The next example separates a direct Beta-Bernoulli prediction from the more involved probability-of-being-best calculation. Assume a hypothetical trial starts each of two treatments with independent \(\mathrm{Beta}(1,1)\) priors. The patients’ binary responses are modeled as conditionally independent given the treatment-specific response probabilities. The allocation of earlier patients may be adaptive, but the response likelihood is still Bernoulli conditional on the assigned treatment and its rate; the allocation rule must not use the current patient’s unobserved response.
Worked Example: Updating Rates and Predicting a Response
Suppose treatment A has 4 successes and 2 failures, while treatment B has 2 successes and 4 failures. The Beta-Bernoulli update gives
The posterior mean response rates are \(m_A=5/(5+3)=0.6250\) and \(m_B=3/(3+5)=0.3750\). Therefore, for a future patient assigned to B, the predictive probability of success is
This calculation does not give \(q_B\). To calculate \(q_B\), one must compare posterior draws or integrate the joint posterior over the region where \(\theta_B\) exceeds \(\theta_A\). The fact that \(m_A=0.6250\) exceeds \(m_B=0.3750\) identifies which posterior mean is larger; it does not, by itself, state either arm’s exact probability of being best.
Ties, Dependence, and Allocation Probabilities
The sum of best-arm probabilities requires care about ties in the joint posterior. If exactly one arm is best with posterior probability one, the best-arm events are disjoint and exhaustive, so their probabilities sum to one. If posterior mass lies on ties and \(q_j\) uses the weak inequality definition, a tied state contributes to multiple \(q_j\)'s. The sum can then exceed one. A Thompson-sampling implementation must specify how it handles ties; uniform random tie-breaking, for example, assigns a tied draw to one arm at random.
Worked Example: Continuous Marginals With a Certain Tie
Consider two hypothetical treatment rates with a perfectly dependent joint posterior: \(\theta_1=\theta_2=\Theta\), where \(\Theta\sim\mathrm{Beta}(2,2)\). Each marginal is a continuous Beta distribution, but the rates are equal with probability one.
Since \(\mathrm{Beta}(2,2)\) has mean \(2/(2+2)=0.5\), both posterior mean response rates and both arm-specific predictive success probabilities are \(0.5\). Under the weak-inequality definition, each treatment is at least as good as the other in every posterior draw, so \(q_1=1\) and \(q_2=1\). Their sum is 2, not 1.
If Thompson sampling breaks every tied draw uniformly at random, its allocation probability is \(1/2\) for each arm. Those allocation probabilities are not the weak-inequality values \(q_1=q_2=1\); they are probabilities after applying the specified tie-breaking rule. This example shows why continuity of each marginal is not enough to establish that ties have probability zero.
Choosing the Quantity That Answers the Question
The quantities are related, but they are not interchangeable. The posterior mean summarizes the rate’s location; the predictive probability forecasts a binary outcome on a specified arm; and the best-arm probability summarizes uncertainty about the ranking across arms. In the Beta-Bernoulli model, the first two have the same numerical value, while the third generally does not.
| Quantity | Question answered | Typical use |
|---|---|---|
| \(m_j=E(\theta_j\mid H)\) | What is the posterior average response rate for arm \(j\)? | Summarizing a treatment’s response rate |
| \(P(Y_{\mathrm{new}}=1\mid H,A_{\mathrm{new}}=j)\) | What is the chance of success for a future patient assigned to arm \(j\)? | Predicting a response on a specified arm |
| \(q_j=P(\theta_j\geq\theta_\ell\ \forall\ell\mid H)\) | What is the posterior chance arm \(j\) has the highest rate? | Thompson-sampling selection when ties have probability zero |
A deterministic choice based on the largest posterior mean answers yet another decision question: which arm has the greatest posterior average rate? It should not be described as Thompson sampling, whose random selection uses posterior uncertainty about which arm is best. Neither a high posterior mean nor a high best-arm probability alone establishes that an arm meets a clinical requirement; that question needs an explicitly defined criterion.
Assumptions, Diagnostics, and Limitations
These interpretations rely on a coherent posterior model for the treatment response rates. In the simple model, each response is binary, outcomes are conditionally independent given the rate for the assigned treatment, and the rates and prior dependence structure are specified. The Beta update also relies on a Bernoulli likelihood and Beta prior. If responses are clustered, outcomes depend on patient characteristics not represented in the model, or the response definition changes during the trial, the simple posterior may not adequately describe uncertainty. Then its means, predictions, and best-arm probabilities can all be misleading.
Useful checks include verifying that success and failure counts match the recorded binary outcomes, confirming that each posterior update uses the patients assigned to that arm, and examining whether the prior and likelihood make sense for the clinical setting. For numerical estimates of \(q_j\), posterior simulation should draw from the joint posterior, record which arm wins each draw under the chosen tie rule, and report Monte Carlo error when relevant. A calculation that estimates each marginal separately but discards their dependence can give incorrect ranking probabilities.
The most common mistake is to read a selection probability as an outcome probability: “Arm A is selected with probability 0.83, so a patient assigned to A has an 83% chance of success.” That conclusion is wrong. In the first example, the selection probability was \(5/6\), while the predictive success probability on arm 1 was \(2/3\). The correction is to name the event in every probability statement: “the posterior probability that arm 1 is best” for \(q_1\), and “the predictive probability of success on arm 1” for \(m_1\).
Check Your Understanding
Answer each question, then compare with the solutions.
- For a future patient assigned to arm \(j\), what quantity gives the predictive probability of a binary success under the Bernoulli model?
- In the first worked example, what is the probability that treatment 2 is best, and what is its predictive success probability?
- Suppose two posterior rates are identical to the same random variable. Why might the weak-inequality best-arm probabilities fail to sum to one?
- If an allocation rule selects arms with probabilities \(r_j\), what is the predictive success probability for a patient allocated by that rule?
Solutions. (1) It is \(E(\theta_j\mid H)=m_j\), by averaging the Bernoulli success probability over the posterior. (2) The first example gives \(q_2=1/6\approx0.1667\) and \(m_2=1/3\approx0.3333\); the former is a ranking probability and the latter predicts success on arm 2. (3) A tied posterior draw satisfies the weak-inequality definition for both arms, so it is counted in both probabilities; if the tie occurs with probability one, both probabilities equal one. (4) It is \(\sum_j r_jm_j\), the allocation-weighted average of the arm-specific predictive probabilities, assuming allocation precedes the future response.