Why Follow the Trial One Assignment at a Time?
In Thompson sampling, each new participant’s allocation probability depends on the response data available at that moment. The procedure therefore forms a sequence: calculate posterior probabilities, randomize one participant, observe that participant’s response when available, update the posterior, and recalculate. A calculation made before an outcome is observed is not the same as one made after it.
This walkthrough considers a hypothetical trial with two arms, A and B, where a response is binary and success is the outcome of interest. The example isolates the arithmetic of four successive assignments. It assumes responses are available before the next assignment; later sections consider what changes when they are not.
The rule is a randomized decision, not a declaration that the arm with the larger posterior probability must receive the participant. For example, if \(q=0.8\), the next participant is assigned to A with probability 0.8 and to B with probability 0.2. The probability is recalculated using the data available at that assignment, then a random draw determines the actual arm.
Posterior Updates and the Best-Arm Probability
For a Beta prior with parameters \(a_k,b_k\), the density is proportional to \(\theta_k^{a_k-1}(1-\theta_k)^{b_k-1}\). A Bernoulli success contributes a likelihood factor \(\theta_k\), and a failure contributes \(1-\theta_k\). Multiplying the prior density by the likelihood for \(s_k\) successes and \(f_k\) failures gives a density proportional to
which is the kernel of a \(\operatorname{Beta}(a_k+s_k,b_k+f_k)\) distribution. This update uses the correct response coding and assumes the Bernoulli probability is constant within each arm over the period being modeled.
The allocation probability compares two posterior draws, not just the two posterior means. In general, it can be written as an integral over the posterior density of A and the posterior distribution function of B:
Here \(f_A\) is the posterior density for \(\theta_A\), and \(F_B(x\mid\mathcal D)=P(\theta_B<x\mid\mathcal D)\). The integral follows by conditioning on \(\theta_A=x\): the conditional probability that \(\theta_B<x\) is \(F_B(x\mid\mathcal D)\), and averaging that quantity over the posterior distribution of \(\theta_A\) gives the result. For the simple Beta distributions in the walkthrough, these probabilities can be evaluated directly.
Worked Example: Four Successive Assignments
Hypothetical setup. Begin with independent uniform priors, \(\theta_A,\theta_B\sim\operatorname{Beta}(1,1)\). This is a valid prior for each Bernoulli success probability. Assume independent responses conditional on the arm probabilities, no response delays, and no minimum allocation floor. The four outcomes below are one possible realization of the random assignment and response process.
Assignment 1. Initially, both posterior distributions are uniform and symmetric, so \(P(\theta_A>\theta_B)=0.5\). The first participant is assigned to A in this hypothetical run and has a success. Updating gives A a \(\operatorname{Beta}(2,1)\) posterior and leaves B at \(\operatorname{Beta}(1,1)\).
The equality holds because, conditional on \(\theta_A=x\), a uniform \(\theta_B\) is below \(x\) with probability \(x\). Thus participant 2 is assigned to A with probability \(2/3\) and to B with probability \(1/3\). Suppose the random draw assigns participant 2 to B and the response is a failure. B’s posterior becomes \(\operatorname{Beta}(1,2)\); A remains \(\operatorname{Beta}(2,1)\).
Assignment 3. For \(\theta_B\sim\operatorname{Beta}(1,2)\), the posterior distribution function is \(F_B(x)=1-(1-x)^2\). Since \(\theta_A\sim\operatorname{Beta}(2,1)\), its density is \(2x\), so
Therefore participant 3 is assigned to A with probability \(5/6\) and to B with probability \(1/6\). Suppose the random draw assigns participant 3 to A and the response is a success. The updated posteriors are A: \(\operatorname{Beta}(3,1)\), and B: \(\operatorname{Beta}(1,2)\).
Assignment 4. The density for A is now \(3x^2\). Using the same distribution function for B,
Participant 4 is thus assigned to A with probability 0.9 and to B with probability 0.1. Suppose the random draw assigns participant 4 to B, and the response is a success. The updated posteriors are A: \(\operatorname{Beta}(3,1)\), and B: \(\operatorname{Beta}(2,2)\).
After the fourth response, the probability for the next assignment can also be recalculated. The distribution function for \(\operatorname{Beta}(2,2)\) is \(F_B(x)=3x^2-2x^3\), while the density for A is \(3x^2\). Hence
The sequence of probabilities for assigning to A, before participants 1 through 4, was \(0.5,\ 0.6667,\ 0.8333,\ 0.9\). After the fourth response, the probability for the next participant would be \(0.8\). The change from 0.9 to 0.8 is not a contradiction: the last success was observed on B, so the posterior evidence for B improved.
Keeping the Sequential Bookkeeping Straight
A practical way to avoid mixing up the information available at different assignments is to record the data counts and posterior parameters after every observed response. The table summarizes the hypothetical run. Each allocation probability is calculated before that row’s assignment; the last row shows the probability recalculated after the fourth response.
| Participant | Probability assigned to A | Observed assignment and response | Posteriors after response |
|---|---|---|---|
| 1 | 0.5000 | A, success | A: Beta(2,1); B: Beta(1,1) |
| 2 | 0.6667 | B, failure | A: Beta(2,1); B: Beta(1,2) |
| 3 | 0.8333 | A, success | A: Beta(3,1); B: Beta(1,2) |
| 4 | 0.9000 | B, success | A: Beta(3,1); B: Beta(2,2) |
| Next assignment | 0.8000 | Not part of the four assignments | A: Beta(3,1); B: Beta(2,2) |
Worked Example: Calculating a Probability From Posterior Counts
Suppose a separate hypothetical snapshot, again using \(\operatorname{Beta}(1,1)\) priors, has two successes and no failures on A, and no successes and two failures on B. The resulting posteriors are A: \(\operatorname{Beta}(3,1)\), and B: \(\operatorname{Beta}(1,3)\). This is a valid update from the stated counts: each success increments the first Beta parameter and each failure increments the second.
For B, \(F_B(x)=1-(1-x)^3\), and the density for A is \(3x^2\). Therefore the probability of assigning the next participant to A is
So the next assignment probabilities are 0.95 for A and 0.05 for B. This is a probability based on the full posterior distributions. It is not obtained by simply comparing posterior means, even though in this particular example the posterior means are \(3/4\) for A and \(1/4\) for B.
Responses That Are Not Yet Available
The main walkthrough assumes that each response arrives before the next participant is allocated. In a trial with delayed responses, the same principle applies, but the update uses only responses that have actually become available. A pending response is not a success or a failure and must not be entered into either count. If the allocation algorithm is intended to use all available outcome information, its data state should distinguish assigned participants from participants with observed outcomes.
For example, with uniform priors, one observed success on A and one observed failure on B give posteriors \(\operatorname{Beta}(2,1)\) and \(\operatorname{Beta}(1,2)\), respectively, and an allocation probability of \(5/6\) for A. If another participant has already been assigned to A but has no recorded response, that pending case does not change the probability. Once a success is observed on A, the posterior becomes \(\operatorname{Beta}(3,1)\) and the probability for A becomes 0.9. This distinction matters because an unobserved outcome contains no realized Bernoulli response to add to the likelihood.
Worked Example: Checking the Available-Response Update
Consider the delayed-response situation just described. There is one observed success on A and one observed failure on B, plus a pending participant assigned to A. Starting from independent \(\operatorname{Beta}(1,1)\) priors, the observed responses give A: \(\operatorname{Beta}(2,1)\), and B: \(\operatorname{Beta}(1,2)\). The probability for A is
The pending participant has no observed success or failure, so the correct allocation probability remains \(0.8333\). If that participant’s response later arrives as a success, A becomes \(\operatorname{Beta}(3,1)\), and the recalculated probability is
Entering the pending participant as a failure instead would also be wrong: it would add evidence that has not been observed and produce a different, unjustified posterior.
Diagnostics, Alternatives, and Limits
For a short sequential run, a transparent audit is often more useful than a sophisticated diagnostic. After each outcome, verify that the correct arm’s count changed by one, that a success increments the first Beta parameter, that a failure increments the second, and that the two allocation probabilities sum to one. Also check that the allocation probability is calculated from the posterior state before the next random draw, not from a posterior that includes a future response.
A common mistake is to interpret \(q=0.9\) as meaning that A is certainly better and assign every subsequent participant to A. The correction is that \(q=0.9\) is the probability used in the randomization: A receives the next participant with probability 0.9, while B retains probability 0.1. Another mistake is to assign according to which arm has the larger posterior mean. That is a different rule; Thompson sampling uses the posterior probability that an arm is best. With two arms and no ties, the posterior best probabilities are \(q\) and \(1-q\), but they are not generally determined by comparing posterior means alone.
Compared with fixed equal randomization, this procedure changes the assignment probabilities as outcome information accumulates. Compared with a plug-in rule that always selects the arm with the larger estimated success probability, Thompson sampling retains a chance of assigning to the other arm. These are different allocation strategies, with different consequences for how participants are assigned; the four-person calculation alone does not establish which strategy is preferable for a particular trial.
The calculation also depends on its modeling assumptions. A single fixed \(\theta_k\) for each arm may be inadequate if success probabilities vary substantially across time or across participant subgroups and those differences are not modeled. Independent arm priors are an assumption, not a requirement for every possible model; with dependent posteriors, the joint posterior must be used to calculate \(P(\theta_A>\theta_B\mid\mathcal D)\). Finally, allocation floors or other protocol constraints change the probability actually used for randomization. The values above apply to the stated unconstrained rule, not to a version with modified probabilities.
Check Your Understanding
Answer each question using the assumptions and notation in this tutorial. Brief solutions follow the questions.
- With independent \(\operatorname{Beta}(1,1)\) priors, one success on A, and one failure on B, what are the posteriors and the probability of assigning the next participant to A?
- Suppose the current probability for A is 0.72. What are the randomization probabilities for A and B, and does this make assignment to A certain?
- With uniform priors, suppose A has one success and one failure, while B has one failure. What are the posterior distributions for the two arms?
- Why should a pending response not be counted as either a success or a failure?
Solutions. (1) A has posterior \(\operatorname{Beta}(2,1)\), and B has posterior \(\operatorname{Beta}(1,2)\). As calculated above, \(P(\theta_A>\theta_B\mid\mathcal D)=5/6\approx0.8333\), so the probability for B is \(1/6\approx0.1667\). (2) Assign to A with probability 0.72 and to B with probability \(1-0.72=0.28\); neither assignment is certain. (3) A has posterior \(\operatorname{Beta}(2,2)\), because it has one success and one failure. B has posterior \(\operatorname{Beta}(1,2)\), because it has one failure and no success. (4) A pending response has not yet provided a Bernoulli outcome, so it contributes no success or failure likelihood factor and does not change the posterior counts.