Implementing Posterior Sampling for Trial Allocation

From Posterior Distributions to an Assignment

Thompson sampling turns current uncertainty about treatment response into a randomized assignment. At each allocation decision, it draws one plausible response parameter from each treatment’s current posterior distribution and assigns the next participant to the treatment with the largest draw. The draws are temporary computational devices: they are not observed outcomes, and they do not replace the posterior distributions.

The procedure requires a model for the response and a posterior distribution for each treatment parameter. To make the implementation concrete, begin with a binary beneficial response, such as symptom improvement by a prespecified follow-up time. Let \(K\) be the number of treatments, indexed by \(k=1,\ldots,K\), and let \(\theta_k\) be the probability of a beneficial response under treatment \(k\). For this tutorial, assume responses are conditionally independent Bernoulli observations within and across treatments, and use independent Beta priors for the treatment response probabilities. These assumptions give independent Beta posteriors given the trial data. Other outcome models require draws from their own posterior distributions, but the allocation step is the same: select the treatment with the largest sampled value of the quantity being optimized.

Definition: At an allocation decision, the current posterior for treatment \(k\) is the distribution of \(\theta_k\) given the prior model and the trial information observed so far. One Thompson-sampling allocation draws \(\tilde{\theta}_k\) from this posterior for every treatment and assigns the next participant to an arm whose draw is largest. The tilde marks a temporary posterior draw, not a new estimate or an observed response.

For the Beta-Bernoulli model, if the prior for treatment \(k\) is \(\operatorname{Beta}(a_{0k},b_{0k})\), and the observed data include \(S_k\) beneficial responses and \(F_k\) non-beneficial responses, its posterior is \(\operatorname{Beta}(a_{0k}+S_k,b_{0k}+F_k)\). Thus the algorithm can store the two updated shape parameters for each treatment rather than the full history. This is a sufficient summary under the stated model.

\theta_k\mid \text{current data}\sim\operatorname{Beta}(a_k,b_k), \qquad a_k=a_{0k}+S_k,\quad b_k=b_{0k}+F_k.

Sequential Allocation Algorithm

The essential operation is one draw per treatment per allocation decision. A draw must be generated from each treatment’s current posterior before comparing them. The largest draw determines the treatment assigned to this participant. Once that participant’s response is observed, the data can be incorporated into the relevant posterior before the next decision.

Set the current posterior state.
For each treatment \(k\), obtain its current posterior distribution for the beneficial-response probability \(\theta_k\), based on the model and data observed to date.

Draw one value from every treatment posterior.
Generate \(\tilde{\theta}_k\) from the current posterior of \(\theta_k\), for each \(k=1,\ldots,K\). Use the joint posterior if the model makes treatment parameters dependent; with independent Beta posteriors, the draws can be generated independently.

Select the largest draw.
Choose \(k^\ast\in\arg\max_{1\leq k\leq K}\tilde{\theta}_k\). If there is an exact tie, apply a prespecified tie rule, such as choosing uniformly among the tied treatments.

Assign the participant.
Assign the next participant to treatment \(k^\ast\). The posterior draws determine this allocation; the participant’s response is observed later, not used to make this same allocation.

Prepare for the next decision.
After the response is recorded, incorporate it into the model’s information state and use the resulting current posterior at the next allocation decision.

In compact pseudocode, the allocation operation is:

Algorithm: For the next participant, for each treatment \(k=1,\ldots,K\), draw \(\tilde{\theta}_k\) from treatment \(k\)'s current posterior. Set \(k^\ast=\arg\max_k\tilde{\theta}_k\), resolve any exact tie using the prespecified rule, and assign the participant to \(k^\ast\). After the response is observed, update the trial information state according to the specified response model before running the allocation operation again.

This description specifies an assignment rule, not a complete clinical-trial protocol. Eligibility, consent, safety monitoring, stopping rules, and any constraints on allocation must be handled separately. The mathematical allocation rule also presumes that the model’s response definition and the direction of benefit are clear before sampling.

Why the Draw-and-Maximum Rule Works

The randomization comes from posterior uncertainty. Conditional on the observed trial information, each posterior draw is a random value. The treatment selected by the draw-and-maximum rule is therefore random even when the data currently stored in the trial are fixed. In the independent-posterior case, the probability that treatment \(k\) is selected is the probability that its posterior draw exceeds every other treatment’s draw. Equivalently, it is the posterior probability that treatment \(k\) has the largest response probability, provided the posterior has no ties with positive probability.

P(k^\ast=k\mid \text{current data}) = P\!\left(\tilde{\theta}_k>\tilde{\theta}_j\text{ for every }j\ne k \mid \text{current data}\right) = P\!\left(\theta_k>\theta_j\text{ for every }j\ne k \mid \text{current data}\right).

The last equality follows because the vector of posterior draws, conditional on the current data, has the same joint distribution as the vector of treatment parameters under the posterior. Thus the algorithm samples a treatment in proportion to its posterior chance of being best, without first having to compute that chance explicitly. For continuous Beta distributions, an exact tie has probability zero. In software, a tie rule is still useful for robustness, because finite-precision calculations or discrete models can produce equal draws.

This justification depends on drawing from the current posterior, using a consistent treatment of the joint posterior, and maximizing the parameter that represents the intended benefit. A draw from a posterior mean, a plug-in estimate, or an unrelated distribution does not implement this rule.

Worked Example: Building the Posterior State Before Sampling

Suppose a hypothetical trial compares two treatments for a binary beneficial response. Each treatment has a \(\operatorname{Beta}(1,1)\) prior. Before the next allocation, treatment A has produced 8 beneficial and 4 non-beneficial responses; treatment B has produced 6 beneficial and 6 non-beneficial responses. Find the current posteriors and illustrate one allocation decision using posterior draws.

The stated model is Bernoulli response with independent Beta priors. Under Beta-Bernoulli updating, add beneficial responses to the first shape parameter and non-beneficial responses to the second:

\theta_A\mid\text{data}\sim\operatorname{Beta}(1+8,1+4) =\operatorname{Beta}(9,5),\qquad \theta_B\mid\text{data}\sim\operatorname{Beta}(1+6,1+6) =\operatorname{Beta}(7,7).

For this illustrative allocation, suppose the random-number generator produces the posterior draws \(\tilde{\theta}_A=0.62\) and \(\tilde{\theta}_B=0.71\). These are hypothetical draws from the stated posteriors, not observed response rates. Since \(0.71>0.62\), the rule assigns the next participant to treatment B. It does not assign the participant to A merely because A has more observed beneficial responses; the decision uses the sampled posterior values, which reflect both the data and uncertainty.

Worked Example: Applying the Rule to Three Treatments

Consider a hypothetical three-treatment trial with beneficial-response probabilities \(\theta_1,\theta_2,\theta_3\), independent Beta posteriors based on the trial’s current data. Suppose these current posteriors are \(\operatorname{Beta}(5,5)\), \(\operatorname{Beta}(8,7)\), and \(\operatorname{Beta}(3,4)\), respectively. A single allocation decision generates the hypothetical draws \((0.46,0.59,0.51)\). Which treatment receives the next participant?

All three draws are generated before selection, one from each treatment’s current posterior. Comparing them gives \(0.59>0.51>0.46\), so the largest is the draw for treatment 2. The allocation is therefore to treatment 2. No posterior mean calculation is needed for this decision, and the draws are not three predictions of the next participant’s response. They are draws of the unknown treatment response probabilities.

For comparison, the posterior means are \(5/(5+5)=0.500\), \(8/(8+7)=0.533\), and \(3/(3+4)=0.429\). Those means summarize the posteriors, but selecting the largest mean would be a different, deterministic rule. The actual Thompson-sampling decision in this example follows the sampled values, so it is determined by \(0.46,0.59,0.51\).

Worked Example: Checking the Direction of Benefit

Suppose a hypothetical trial measures a binary adverse event, so a smaller probability is better. The current posteriors for treatments A and B are \(\operatorname{Beta}(4,16)\) and \(\operatorname{Beta}(5,15)\), where the first shape parameter counts adverse events after a \(\operatorname{Beta}(1,1)\) prior. For one allocation, suppose the posterior draws of adverse-event probabilities are \(\tilde{\theta}_A=0.18\) and \(\tilde{\theta}_B=0.24\). Which treatment should receive the next participant if the objective is to reduce adverse events?

Because the parameter is an adverse-event probability, the preferable sampled value is the smaller one. Since \(0.18<0.24\), the assignment is to treatment A. Equivalently, define utility as the negative adverse-event probability, \(U_k=-\theta_k\), draw \(\tilde{U}_k=-\tilde{\theta}_k\), and maximize utility: \(-0.18>-0.24\). This example illustrates why the outcome coding and optimization direction must agree. Blindly applying “choose the largest draw” to a harmful-outcome probability would reverse the intended allocation.

Implementation Checks and Failure Modes

A correct implementation is short, but it is easy to implement a different rule by accident. The following checks focus on the allocation step itself.

Check what is being sampled. Draw a value from each treatment’s posterior distribution, not a treatment’s posterior mean and not a simulated patient outcome.
Check that every treatment gets a draw. Drawing only for the current leader or only for treatments with the largest observed rates prevents the required comparison across all current posteriors.
Check outcome orientation. Use a parameter whose larger value means better response, or explicitly minimize a harmful-outcome parameter. State this convention in the implementation.
Check the information time. All draws for one assignment must use the posterior based on information available before that participant is assigned. Do not use that participant’s not-yet-observed response.
Check the posterior parameterization. In the Beta-Bernoulli model used here, beneficial responses increment \(a_k\) and non-beneficial responses increment \(b_k\). Reversing those counts changes the posterior.
Check ties and reproducibility. Specify a tie rule and, for debugging, record the random seed or the generated draws. A fixed seed can make a test run reproducible; it does not make the allocation rule deterministic in ordinary operation.

A frequent mistake is to calculate all posterior means and assign every participant to the treatment with the largest mean, describing that procedure as Thompson sampling. That is incorrect: a mean is a single summary value, whereas Thompson sampling draws a fresh parameter value from each posterior and selects the largest draw. The difference matters because the draw-and-maximum rule allows a treatment with substantial posterior uncertainty to be selected on some allocations. Another mistake is to interpret \(\tilde{\theta}_k\) as the probability that the next individual responds. It is instead a draw of the treatment-level response probability; the individual response remains a separate random outcome.

Comparison and Scope

Allocation rule	Operation at a decision	How it differs from posterior sampling
Thompson sampling	Draw one parameter from each current posterior; choose the largest beneficial-response draw.	Uses the full posterior uncertainty to induce a randomized choice.
Posterior-mean greedy rule	Compute each posterior mean; choose the treatment with the largest mean.	Uses a fixed summary for the current state rather than posterior draws.
Uniform random allocation	Choose each treatment with equal probability, regardless of the posterior.	Does not use current posterior evidence to favor treatments with higher sampled response probabilities.

These rules answer different design questions and should not be treated as interchangeable. This tutorial describes the mechanics of the posterior-draw allocation for one participant at a time. The model must still be appropriate for the outcome, and practical trial requirements may impose additional constraints. The algorithm itself does not guarantee that every treatment will receive a prespecified minimum number of participants, address safety decisions, or establish that a treatment is clinically superior.

Key takeaway: At each decision, sample one parameter value per treatment from the current posterior, select the treatment with the most favorable draw, and assign the next participant. Correctness depends on the posterior state, the direction of benefit, and using only information available before assignment.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

With a \(\operatorname{Beta}(2,3)\) prior, a treatment has 5 beneficial and 2 non-beneficial responses. What is its posterior distribution?
Two beneficial-response posterior draws are \(0.38\) for treatment A and \(0.44\) for treatment B. Which treatment is assigned under the rule?
Why is choosing the treatment with the largest posterior mean not the same operation as Thompson sampling?
A parameter is the probability of a harmful event, and smaller values are preferable. Should the allocation choose the largest or smallest sampled probability?
When, relative to assignment of the next participant, can that participant’s response be used in the posterior draw for their own assignment?

Solutions. (1) Add responses to the corresponding prior shape parameters: \(\operatorname{Beta}(2+5,3+2)=\operatorname{Beta}(7,5)\). (2) Assign to treatment B because \(0.44>0.38\). (3) The posterior mean is one fixed summary per treatment; Thompson sampling draws a value from each full posterior at the decision, so its selected treatment can vary across draws even with the same current data. (4) Choose the smallest sampled harmful-event probability, or equivalently maximize its negative as a benefit measure. (5) It cannot: the response is not yet observed at assignment. It can inform a later posterior state after it has been recorded.