Updating the Allocation State After Each Response

Why the allocation state changes after a response

In a sequentially updated Thompson sampling trial, the allocation for the next participant depends on the evidence available when that participant is assigned. A response therefore does more than add a row to the trial database: it changes the posterior distribution for the treatment received, which can change the probability that each treatment is selected next. The update is local to the arm whose response was observed, but the resulting allocation probabilities compare all arms.

The focus here is the transition from one observed response to the next allocation state. It is useful to keep three quantities distinct: the posterior response rate for an arm, the predictive probability of the next response on that arm, and the probability that the arm is best under the current posterior. The last of these is the Thompson sampling allocation probability; it is not generally the same as either of the first two.

Definition: Consider \(K\) treatment arms with unknown binary-response probabilities \(\theta_1,\ldots,\theta_K\). Before participant \(t\) is assigned, let \(H_t\) denote the observed trial history. Define \(q_{k,t}=P(\theta_k=\max_j\theta_j\mid H_t)\) as the posterior probability that arm \(k\) has the largest response probability. Under Thompson sampling, \(q_{k,t}\) is the probability of assigning participant \(t\) to arm \(k\).

State representation and assumptions

For arm \(k\), let \(S_{k,t}\) and \(F_{k,t}\) be the numbers of observed successes and failures on that arm before participant \(t\) is assigned. Suppose the prior response probability is \(\theta_k\sim\operatorname{Beta}(a_k,b_k)\), with independent priors across arms. Assume that, conditional on the arm-specific response probabilities, responses are independent Bernoulli observations, and that each arm’s response probability remains stable during the period being modeled. A success is coded \(Y_t=1\), and a failure \(Y_t=0\).

These assumptions give a compact state: the current Beta parameters for each arm. If \(Z_t\) is the arm assigned to participant \(t\), the observed response updates only that arm’s success or failure count. The response model must be appropriate for the trial: if response probabilities change over time, depend on participant covariates, or are linked across treatments, this independent Beta-Bernoulli state is not adequate as stated.

Update rule: If participant \(t\) receives arm \(k\) and has a success, replace that arm’s posterior \(\operatorname{Beta}(a_k+S_{k,t},b_k+F_{k,t})\) by \(\operatorname{Beta}(a_k+S_{k,t}+1,b_k+F_{k,t})\). For a failure, replace it by \(\operatorname{Beta}(a_k+S_{k,t},b_k+F_{k,t}+1)\). The posterior distributions for the other arms do not change from this response.

Once the posterior state is updated, the allocation probabilities for the next participant are recalculated from the updated joint posterior. In practice, this can be done by evaluating posterior best-arm probabilities or by drawing parameter values from the updated posteriors and estimating the fraction of draws in which each arm is best. Any such estimate is conditional on the updated history, not on the history before the response.

One response, two different probabilities

The predictive probability of success on arm \(k\), given the current data, is the posterior mean of \(\theta_k\). If that arm’s posterior is \(\operatorname{Beta}(a'_k,b'_k)\), then the next response on that arm has success probability \(a'_k/(a'_k+b'_k)\). This probability answers, “If the next participant receives this arm, how likely is a success?” By contrast, \(q_{k,t}\) answers, “How likely is this arm to be assigned next under Thompson sampling?” It compares uncertainty and response-rate beliefs across all arms.

Worked Example: One Success Changes the Next Allocation

Suppose there are two arms, A and B, each with a \(\operatorname{Beta}(2,2)\) prior and no observed responses. Both posteriors are identical, so symmetry gives current allocation probabilities \(q_A=q_B=1/2\). Participant 1 is assigned A and has a success. Find the updated posterior and the next participant’s allocation probabilities.

Update. The success adds one to A’s success parameter, so A’s posterior is \(\operatorname{Beta}(3,2)\). B remains \(\operatorname{Beta}(2,2)\). To find the next allocation, calculate \(P(\theta_A>\theta_B)\); the continuous Beta distributions have zero probability of an exact tie.

The density of \(\theta_A\sim\operatorname{Beta}(3,2)\) is \(12x^2(1-x)\), and the cumulative distribution function of \(\theta_B\sim\operatorname{Beta}(2,2)\) is \(3x^2-2x^3\). Therefore:

\begin{aligned} q_A &= \int_0^1 12x^2(1-x)(3x^2-2x^3)\,dx \\ &= 12\int_0^1 (3x^4-5x^5+2x^6)\,dx \\ &= 12\left(\frac{3}{5}-\frac{5}{6}+\frac{2}{7}\right) = \frac{22}{35} \approx 0.6286. \end{aligned}

Thus \(q_B=1-22/35=13/35\approx0.3714\). The success makes A more likely to be best, so A has the larger probability of receiving participant 2. This does not say that A’s response probability is \(0.6286\): its posterior mean is \(3/(3+2)=0.6000\). The allocation probability and predictive response probability answer different questions.

Sequential updating and the next participant

A sequential trial repeats the same transition: assign one participant using the current state, observe that participant’s response, update the state, and use the new state for the next assignment. The assignment mechanism itself does not update the Beta response parameters. Under the stated policy, assignment is randomized using the history-based probabilities; conditional on the history, the fresh randomization does not reveal the unknown true response probabilities. The observed response supplies the new likelihood information.

Start from the current history.
Record each arm’s current posterior parameters, obtained from its prior and its observed successes and failures.

Assign the participant.
Use the current posterior best-arm probabilities \(q_{k,t}\) as the probabilities of assignment to the respective arms.

Observe and code the response.
For the assigned arm, record a success as \(1\) or a failure as \(0\), with the participant and arm correctly matched.

Update and recompute.
Increment the assigned arm’s success or failure count, leave other arm posteriors unchanged, and recalculate the best-arm probabilities for the next assignment.

Worked Example: A Failure Reverses Some of the Change

Continue from the state in the preceding calculation: A has posterior \(\operatorname{Beta}(3,2)\) and B has posterior \(\operatorname{Beta}(2,2)\). The next participant is assigned A. The predictive success probability on A is its posterior mean, \(3/5\), so the predictive failure probability is \(1-3/5=2/5=0.4000\). Suppose the observed response is a failure.

The failure increments A’s second Beta parameter, giving A the posterior \(\operatorname{Beta}(3,3)\). B remains \(\operatorname{Beta}(2,2)\). Both posterior distributions are symmetric around \(1/2\). If independent draws \(X\) and \(Y\) are taken from these distributions, the pair \((1-X,1-Y)\) has the same joint distribution as \((X,Y)\). Consequently \(P(X>Y)=P(X<Y)\); because the continuous distributions tie with probability zero, each probability is \(1/2\).

The updated allocation probabilities for the next participant are therefore \(q_A=q_B=0.5000\). The response did not make B’s posterior stronger; rather, the failure changed A’s posterior, and the resulting comparison between the two arms is now symmetric.

Why the expected allocation probability is unchanged

A response can make the next allocation probability rise or fall, as the examples show. Nevertheless, averaging over the possible response on the assigned arm gives a useful check: before that response is observed, the expected updated probability that any particular arm is best equals its current probability of being best. This is a posterior-probability identity, not a claim that every observed response leaves allocation unchanged.

Result: Conditional on the current history \(H_t\) and the assigned arm \(Z_t\), let \(B_k\) be the event that arm \(k\) is best. If the response model and posterior are correctly specified, then \(E[P(B_k\mid H_t,Z_t,Y_t)\mid H_t,Z_t]=P(B_k\mid H_t)\).

To justify the identity, condition on the two possible responses. Let \(\pi_y=P(Y_t=y\mid H_t,Z_t)\) and \(q_{k,y}=P(B_k\mid H_t,Z_t,Y_t=y)\). Bayes’ rule and the law of total probability give:

\begin{aligned} \sum_{y\in\{0,1\}}\pi_y q_{k,y} &= \sum_{y\in\{0,1\}}P(Y_t=y\mid H_t,Z_t) P(B_k\mid H_t,Z_t,Y_t=y) \\ &= P(B_k\mid H_t,Z_t) = P(B_k\mid H_t). \end{aligned}

The last equality holds because the randomized assignment, conditional on the observed history, supplies no additional information about the unknown response probabilities. This identity can flag implementation errors: if a calculation of the outcome-weighted updated probabilities fails to recover the current probability, check the response probabilities, posterior updates, and event probabilities. It is a mathematical expectation, so estimates based on a finite number of simulations need not match it exactly.

Worked Example: Checking Both Possible Responses

Return to two \(\operatorname{Beta}(2,2)\) arms. Participant 1 receives A. Before observing the response, A’s predictive success probability is \(1/2\), and its predictive failure probability is also \(1/2\). After a success, the first worked example gives \(q_A=22/35\). After a failure, A has \(\operatorname{Beta}(2,3)\) while B remains \(\operatorname{Beta}(2,2)\). Reflection about \(1/2\) shows the failure-state probability is \(q_A=13/35\), the complement of \(22/35\).

The success-weighted average of the updated allocation probability for A is:

\frac{1}{2}\left(\frac{22}{35}\right) +\frac{1}{2}\left(\frac{13}{35}\right) =\frac{35}{70} =\frac{1}{2}.

This equals A’s pre-response allocation probability. The calculation checks the update logic while also showing why a particular observed response still matters: the next allocation is \(22/35\) after success and \(13/35\) after failure, even though their predictive average is \(1/2\).

Diagnostics, alternatives, and limitations

Reliable adaptation depends on the state used for each assignment matching the responses actually observed before that assignment. A practical audit can compare the stored Beta parameters with prior parameters plus the recorded success and failure counts, and can verify that every response is attributed to the correct participant and arm. Posterior predictive probabilities provide another check: for a \(\operatorname{Beta}(a',b')\) state, the next-response success probability must be \(a'/(a'+b')\). Separately, the allocation probabilities across arms must sum to one, up to numerical error.

Updating the wrong parameter. A success increments the first Beta parameter; a failure increments the second. Reversing this coding reverses the intended evidence.
Updating the wrong arm or updating all arms. Under independent arm-specific Beta-Bernoulli models, only the assigned arm’s posterior changes from this response.
Using a stale state. If the next assignment is computed before the latest available response is incorporated, it is conditional on an outdated history. Delayed or missing outcomes require a clearly defined rule for which data are available at each decision.
Confusing response and allocation probabilities. The posterior mean predicts a response conditional on treatment; the posterior best-arm probability governs assignment under Thompson sampling.
Treating a numerical estimate as exact. Monte Carlo estimates of best-arm probabilities have simulation error. Repeated computations can fluctuate, especially when few posterior draws are used.

The main alternatives make different decisions. Fixed randomization uses prespecified assignment probabilities that do not change in response to outcomes. A greedy rule assigns according to the largest posterior mean and does not use the posterior probability of being best as its randomization probability. Sequential Thompson sampling instead recomputes its probabilities after each available response. If several participants are assigned from one unchanged state, their assignments do not each incorporate the preceding participant’s new response; that is a different timing rule from the sequential update considered here.

The update is only as appropriate as its model and data flow. Independent Beta posteriors do not represent shared treatment effects, covariate-dependent response rates, or time-varying outcomes without further modeling. Nor does the posterior update alone specify operational safeguards such as allocation floors or how to handle unavailable responses. Those decisions must be made consistently with the trial’s design; they should not be improvised by altering the posterior counts.

Common mistake: “After a success on A, add one success to every arm because all arms are being compared.” The response is evidence about the assigned arm’s response probability, not a response observed on the other arms. Under the independent-arm model, update A’s posterior only, then recompute the best-arm probabilities using the full collection of updated and unchanged posteriors.

Worked Example: Three Arms After One Success

Suppose A, B, and C each have an independent \(\operatorname{Beta}(1,1)\) prior and no data. The initial allocation probabilities are \(1/3\) each by symmetry. A participant is assigned A and responds successfully. A’s posterior becomes \(\operatorname{Beta}(2,1)\), while B and C remain uniform. The predictive success probability on A before the response was \(1/2\), the mean of its uniform prior.

Let \(x\) be a draw from A’s updated posterior, which has density \(2x\). For A to be best, independent uniform draws from B and C must both be below \(x\), an event with probability \(x^2\). Thus:

q_A=\int_0^1 2x\cdot x^2\,dx =2\left(\frac{1}{4}\right) =\frac{1}{2}.

For B to be best, condition instead on B’s uniform draw being \(y\). A’s draw is below \(y\) with probability \(y^2\), and C’s draw is below \(y\) with probability \(y\). Therefore:

q_B=\int_0^1 y^2y\,dy=\frac{1}{4}, \qquad q_C=\frac{1}{4}.

The updated allocation probabilities are \((0.5000,0.2500,0.2500)\), which sum to one. A’s single success increases its chance of being best, while the other arms’ posteriors remain unchanged. Their allocation probabilities nevertheless change because each is being compared with A’s updated posterior.

Check Your Understanding

Answer each question using the sequential state definition above. Solutions follow the questions.

An arm has posterior \(\operatorname{Beta}(4,3)\). What is its posterior after a failure, and what is its predictive success probability before that response?
Two arms have identical continuous posterior distributions. What are their Thompson sampling allocation probabilities?
Three arms begin with independent uniform priors. A response is a failure on arm A. What is A’s posterior?
Why can the expected updated allocation probability equal the current one even though an observed response changes the next allocation?

Solutions. (1) The failure increments the second parameter, so the posterior becomes \(\operatorname{Beta}(4,4)\). Before the response, the predictive success probability is the posterior mean \(4/(4+3)=4/7\approx0.5714\). (2) By symmetry and the absence of ties for continuous distributions, each arm has probability \(1/2\) of being best and is assigned with probability \(1/2\). (3) A failure changes A’s \(\operatorname{Beta}(1,1)\) prior to \(\operatorname{Beta}(1,2)\); the posteriors for the other two arms remain \(\operatorname{Beta}(1,1)\). (4) The updated probability is conditional on the realized response, so it can differ for success and failure. Averaging those conditional probabilities with the predictive probabilities of the responses recovers the original posterior probability by the law of total probability.