Why Plug-In Greedy Allocation Is Not Thompson Sampling

Two Rules That Can Look Similar

Suppose a trial compares treatments with binary outcomes, such as response versus no response. After observing some outcomes, each treatment has a posterior distribution for its response probability. Two allocation rules may both use this posterior information, but they use it in different ways. Plug-in greedy allocation compares the posterior means and chooses the treatment with the largest mean. Thompson sampling accounts for the full posterior distributions: it draws a response probability for each treatment and allocates to the treatment whose draw is largest.

The distinction is not merely computational. A posterior mean summarizes the centre of a distribution, while a posterior probability of being best also reflects its uncertainty and shape. Plug-in greedy allocation turns a summary into a single choice. Thompson sampling randomizes in proportion to posterior uncertainty about which treatment is best. The allocation probabilities need not be equal to one another, and they are not generally determined by ranking the posterior means alone.

Model and Assumptions

For the examples, assume there are two arms, \(A\) and \(B\), and a patient's outcome on arm \(i\) is \(1\) for response and \(0\) for no response. Conditional on the arm's response probability \(\theta_i\), outcomes assigned to that arm are Bernoulli with success probability \(\theta_i\). Use independent Beta priors for the arms, and assume response probabilities remain stable during the period considered. Under these assumptions, if arm \(i\) has \(s_i\) responses and \(f_i\) nonresponses, a Beta prior with parameters \(a_i,b_i\) gives posterior \(\operatorname{Beta}(a_i+s_i,b_i+f_i)\). Its mean is \((a_i+s_i)/(a_i+b_i+s_i+f_i)\).

The posterior distributions describe uncertainty about the response probabilities, not variability in an individual patient's outcome. A posterior draw of \(\theta_i\) is therefore not a predicted response for the next patient. It is a draw representing uncertainty about the arm's underlying response rate. Thompson sampling compares these draws; plug-in greedy allocation compares posterior means.

The simple comparison assumes that a larger response probability is preferable, that the arms are evaluated on the same binary endpoint, and that the posterior distributions represent the intended model for the trial. If the clinical objective is not to maximize response probability, the relevant quantity for “best” would need to reflect that objective instead. These examples use independent posteriors so the probability calculations can be shown directly.

How the Two Rules Differ

Write the posterior means as \(\mu_A=E(\theta_A\mid\text{data})\) and \(\mu_B=E(\theta_B\mid\text{data})\). If \(\mu_A>\mu_B\), plug-in greedy allocation assigns to \(A\) (apart from any additional design restrictions). Thompson sampling instead selects \(A\) with probability \(P(\theta_A>\theta_B\mid\text{data})\), and selects \(B\) with probability \(P(\theta_B>\theta_A\mid\text{data})\). With continuous Beta posteriors, the probability of an exact tie is zero, so these two probabilities sum to one.

The central calculation for two arms is an average over the posterior distribution of one arm:

where \(f_A\) is the posterior density of \(\theta_A\) and \(F_B(x)=P(\theta_B\leq x\mid\text{data})\) is the posterior cumulative distribution function for arm \(B\). To justify the expression, condition on \(\theta_A=x\). Given that value, the probability that \(B\)'s response probability is below it is \(F_B(x)\). Averaging that conditional probability over the posterior density \(f_A(x)\) yields the integral. This is a probability of being best, not a comparison of means.

Posterior Means and Best-Arm Probabilities

The calculation above also shows why replacing a posterior distribution with its mean changes the rule. A posterior mean is a single number for each arm. Once those numbers are compared, deterministic plug-in greedy allocation discards the rest of the distribution. In contrast, a best-arm probability depends on how much posterior mass each arm places above the other arm's possible values.

The posterior mean and the posterior probability of being best answer different questions. The mean describes the average response probability under the posterior. The best-arm probability describes how often the arm would have the highest response probability across draws from the joint posterior. A treatment can have a high posterior mean without having a correspondingly high probability of being best, particularly when its posterior is uncertain or asymmetric.

A Small Posterior Update Can Change the Sampling Probability

Greedy allocation can be insensitive to new evidence about how far ahead the leading arm is, as long as its posterior mean remains larger. Thompson sampling can respond to such evidence through a change in the probability that the leading arm is best. The next calculation illustrates this using the same pair of arms as before.

Why the Distinction Matters

Thompson sampling does not allocate in proportion to posterior means. It allocates according to posterior best-arm probabilities. This gives some allocation to an arm with a lower posterior mean whenever that arm has a nonzero chance of being best, while favouring arms that are more likely to be best. A deterministic greedy rule, by contrast, can stop exploring an arm as soon as its posterior mean falls below another's, even if substantial uncertainty remains.

This distinction is useful when checking a design implementation. A program that calculates posterior means and always assigns to the largest one implements plug-in greedy allocation, not Thompson sampling. A Thompson sampling implementation must draw from the posterior distributions (or otherwise calculate the corresponding best-arm probabilities) and let the resulting posterior comparison determine the selected arm. The draw-based procedure approximates the best-arm probabilities over repeated draws; finite Monte Carlo computation can add numerical error, but that error is separate from the conceptual difference between the two rules.

Assumptions, Diagnostics, and Limitations

The comparison is meaningful only if the posterior distributions are appropriate for the trial question. For the Beta-Bernoulli calculations here, outcomes must be represented as binary responses, the likelihood must be Bernoulli conditional on the arm-specific response probability, and the Beta update must use the correct counts. Independent posteriors are an assumption of the worked calculations, not a requirement for the general distinction: with dependent posteriors, Thompson sampling still compares draws from the joint posterior, but the best-arm probabilities must respect that dependence.

A useful diagnostic is to inspect the procedure's actual decision rule, not just its reported posterior means. If allocation always goes to the largest mean, then the procedure is greedy, even if it was described as “Bayesian.” If the implementation uses posterior draws, check that it draws from each arm's current posterior and compares those draws, rather than comparing the draws' averages or substituting the posterior means afterward. Also check that the binary outcome coding and success/failure counts match the model.

Neither rule is automatically appropriate for every clinical objective. Plug-in greedy allocation may under-explore arms with uncertain response rates. Thompson sampling can allocate to an arm that is not best in a particular posterior draw, and it can be sensitive to the model and prior used to form the posterior. The examples assume stable response probabilities and do not establish that either design meets any particular trial's ethical, inferential, or operational requirements. Those judgments require considering the trial's goals and assumptions, not just the allocation formula.

Common Mistake