Trial Information States for Thompson Sampling

Why the Information State Matters

Thompson sampling assigns the next patient using uncertainty about the treatment responses, not just the observed averages. To do that coherently, the trial must keep track of what has happened, how each outcome relates to its assigned treatment, and what the current posterior distribution says about the unknown treatment parameters. If any of these pieces is omitted or misrepresented, the resulting assignment probabilities can be wrong even when the sampling step is implemented correctly.

The central object is the information state: a representation of the observed trial history and the current posterior distribution needed to make the next randomization. The complete history is useful for auditing and checking assumptions; a posterior distribution or an exactly sufficient summary can be the more compact state used by the allocation algorithm.

Definition: For the first \(n\) randomized patients, let \(A_i\in\{1,\ldots,K\}\) denote patient \(i\)'s assigned arm and let \(Y_i\) denote the outcome when it is observed. The trial history \(H_n\) contains the assignments, observed outcomes and their observation status, relevant baseline or stratification variables, and the recorded randomization information through patient \(n\). The arm-specific observed data are \(D_{n,a}=\{(Y_i,X_i):1\le i\le n,\ A_i=a,\ Y_i\text{ observed}\}\), with \(X_i\) denoting any modeled covariates. The posterior state is \(\Pi_n=P(\theta\in\cdot\mid H_n)\), the conditional distribution of the unknown response parameters \(\theta\) given the history.

Model and Assumptions Before Sampling

Suppose each treatment arm \(a\) has an unknown parameter \(\theta_a\), and the outcome model for a patient assigned to that arm has likelihood \(f_a(y\mid x,\theta_a)\). The parameter might represent a response rate, a mean response, or another feature of the outcome distribution. The prior distribution \(\Pi_0\) describes uncertainty about the parameters before trial outcomes are observed. A prior may factor across arms, but independence is not automatic: shared parameters, hierarchical models, or correlated prior beliefs can make the joint prior dependent.

For a simple arm-specific model, assume outcomes are conditionally independent given their assigned arms, covariates, and parameters, and that the model correctly describes the endpoint being analyzed. Also assume the assignment rule at each step depends only on information available at that time, not on future outcomes. Under these conditions, the outcome likelihood contribution from observed data is the product of the appropriate arm-specific likelihoods. If the assignment probabilities are known functions of the observed past and do not introduce additional parameter-dependent information, they do not alter that outcome likelihood when conditioning on the realized history. More complicated designs, informative missingness, or allocation mechanisms depending on unknown parameters may require a joint model instead.

\Pi_n(d\theta) \ \propto\ \Pi_0(d\theta) \prod_{a=1}^{K}\prod_{i:A_i=a,\;Y_i\ \mathrm{observed}} f_a(Y_i\mid X_i,\theta_a).

This expression describes how the history informs the posterior; it does not imply that the full raw history can always be discarded. A summary is safe to use as the algorithm’s state only if it retains all information needed for the posterior calculation and the next allocation. In particular, patient-level covariates, endpoint timing, or missingness indicators must be retained when the model uses them.

Worked Example: Organizing the Observed Arm Data

Consider a hypothetical three-arm trial with six patients whose primary binary outcomes have all been recorded. The assignments and outcomes are: patient 1, arm 1, response; patient 2, arm 2, no response; patient 3, arm 3, response; patient 4, arm 1, no response; patient 5, arm 2, response; patient 6, arm 3, response. The protocol has specified a prior distribution and outcome model for each arm.

The arm-specific observed data are \(D_{6,1}=(1,0)\), \(D_{6,2}=(0,1)\), and \(D_{6,3}=(1,1)\), where 1 denotes response and 0 denotes no response. The counts of observed responses are therefore 1, 1, and 2, respectively, and each arm has two observed outcomes. These summaries might suffice for a particular model, but that must follow from its likelihood; it is not true for every model or covariate structure.

The history \(H_6\) also records who received each assignment, the order of assignments, the outcomes’ observation status, and the randomization information. With the stated model assumptions, the posterior is obtained by combining the specified prior with each arm’s observed-outcome likelihood. The example does not determine a numerical posterior because no prior or likelihood parameters have been specified. Patient 7’s allocation must use the actual current posterior state, not just the response totals viewed without their model.

Posterior State and the Allocation Rule

For response-seeking allocation, Thompson sampling draws a parameter value from the current posterior, identifies the arm with the best response under that draw, and assigns the next patient accordingly. Equivalently, the conditional allocation probability for an arm is the posterior probability that the arm is selected as best, after the stated tie-breaking rule is applied. This probability is conditional on the available trial history; it is not the probability that the arm is truly best in an absolute or frequentist sense.

Let \(r_a(\theta)\) be the probability that the tie-breaking rule selects arm \(a\) when the sampled parameter value is \(\theta\). For every \(\theta\), the rule assigns its total probability across the arms that achieve the best value, so \(\sum_{a=1}^K r_a(\theta)=1\). If ties are broken uniformly, for example, \(r_a(\theta)\) is the reciprocal of the number of tied best arms when \(a\) is among them, and is zero otherwise. A deterministic rule is also possible, provided it selects exactly one arm at a tie.

q_{n+1,a} = P\{\text{the tie-broken best arm is }a\mid H_n\} = \int r_a(\theta)\,\Pi_n(d\theta), \qquad \sum_{a=1}^{K}q_{n+1,a}=1.

When the posterior assigns probability zero to ties, this is also the probability that \(\theta_a\) is strictly larger in response value than every other arm’s value. But when ties have positive posterior probability, an unqualified equality-event probability \(P(\theta_a=\max_b\theta_b\mid H_n)\) can count one tied draw as a win for several arms. It is therefore not a valid allocation rule in general. The tie-broken event, or the integral using \(r_a\), gives a proper probability distribution.

Worked Example: Allocation Probabilities With Posterior Ties

Suppose a hypothetical posterior for three response parameters has three possible states. State 1 has probability 0.40 and values \((0.8,0.8,0.3)\); state 2 has probability 0.35 and values \((0.6,0.7,0.7)\); state 3 has probability 0.25 and values \((0.4,0.5,0.9)\). Use a fair tie-break among arms with the largest value.

In state 1, arms 1 and 2 tie for best, so each is selected with probability \(1/2\). In state 2, arms 2 and 3 tie, so each is selected with probability \(1/2\). In state 3, arm 3 is uniquely best. Thus,

q_{n+1,1}=0.40(1/2)=0.200, \qquad q_{n+1,2}=0.40(1/2)+0.35(1/2)=0.375,

q_{n+1,3}=0.35(1/2)+0.25(1)=0.425.

The probabilities sum to \(0.200+0.375+0.425=1.000\), as required. Counting every tied arm as a winner instead would give \(0.40\), \(0.75\), and \(0.60\), which sum to 1.75 and cannot be used as assignment probabilities. After calculating the valid \(q\)'s, the randomization system assigns the next patient to one arm according to this categorical distribution.

A Posterior State Can Be Compact

A posterior state need not be a stored list of every possible parameter value. It can be a distribution represented by parameters, or by sufficient statistics, when the model permits that reduction. The point is not to compress data for its own sake: the representation must preserve the posterior that the allocation rule needs. The next example shows such a representation in a normal model; it is illustrative of state bookkeeping, not a prescription for a particular trial endpoint.

Worked Example: Updating a Normal-Model State

In a hypothetical trial, suppose outcomes in each arm follow \(Y\mid\theta_a\sim N(\theta_a,1)\), independently conditional on their arm-specific means. Each arm has prior \(\theta_a\sim N(0,1)\), and the two priors are independent. Assume outcomes are fully observed and there are no covariates. Arm 1 has observed outcomes 1 and 3; arm 2 has observed outcome 2.

For this known-variance normal model, the posterior variance is \(1/(1+n_a)\), and the posterior mean is \(\sum_iY_i/(1+n_a)\), because both prior variance and observation variance equal 1. For arm 1, \(n_1=2\) and the outcome sum is 4, giving posterior variance \(1/3\) and mean \(4/3\). Thus \(\theta_1\mid H_n\sim N(4/3,1/3)\). For arm 2, \(n_2=1\) and the outcome sum is 2, giving posterior variance \(1/2\) and mean 1. Thus \(\theta_2\mid H_n\sim N(1,1/2)\).

Under the specified independent priors and likelihood, the current posterior state can therefore be represented by these two normal distributions. This compact state preserves the information needed to draw parameter values under this model. The raw outcomes and assignments should still be retained in the trial record for verification, and the state representation would need to change if the model included additional covariates, shared parameters, or informative outcome observation.

State Transitions, Diagnostics, and Alternatives

After a new patient’s outcome becomes available, the posterior state is updated using that observation and its assigned arm, subject to the model. In a trial with delayed endpoints, the next patient may be randomized before the previous patient’s outcome is known. The allocation state must then distinguish patients already assigned from patients with observed outcomes: a pending outcome belongs in the history, but it is not an observed response that can be inserted into the outcome likelihood as if known. If the design uses interim outcomes or partial follow-up, their definitions and likelihood contributions must be specified explicitly.

Useful checks include verifying that the stored arm labels and outcome counts agree with the trial log, that the posterior uses only information available at the allocation time, and that posterior simulations reproduce the stored posterior model. Every computed allocation vector should have entries in \([0,1]\) and sum to 1 within numerical tolerance. Large discrepancies between recomputed and stored states, unexpected zero allocation probabilities, or treatment of pending outcomes as observed are reasons to stop and investigate rather than proceed automatically.

Thompson sampling differs from fixed randomization, which uses prespecified probabilities regardless of the accumulating posterior, and from a greedy rule, which assigns only to the arm with the best current point estimate. Thompson sampling uses uncertainty: an arm with an uncertain response may still be selected when posterior draws make it best. The method’s allocation probabilities depend on the chosen prior, likelihood, response objective, and tie rule. It does not by itself guarantee good estimation of treatment effects, adequate minimum exposure to each arm, or robustness to model misspecification. Trial protocols may therefore require allocation floors, caps, or other safeguards, which form part of the design and must be reflected in the actual randomization probabilities.

A frequent error is to treat the current observed mean or response proportion as the information state and then assign according to which estimate is largest. That discards posterior uncertainty and turns the procedure into a plug-in or greedy rule, not Thompson sampling. A second error is to equate “best” with a unique largest parameter without specifying what happens at ties. The correction is to retain the posterior state and define the selected-best event using a tie rule whose probabilities sum to one.

Key takeaway: A valid Thompson-sampling decision uses the posterior conditional on the recorded history and a fully specified best-arm and tie-breaking rule. The induced arm probabilities must form a distribution before they are used to randomize the next patient.

Check Your Understanding

Answer the questions, then compare your reasoning with the solutions below.

What distinguishes the full history \(H_n\) from the arm-specific observed data \(D_{n,a}\)?
Under what condition can an arm-specific summary replace the raw outcomes in the allocation algorithm?
Two arms are tied for best in a posterior draw. What must a tie-breaking rule ensure?
Why should a pending endpoint not be included as an observed outcome in the posterior likelihood?
In the normal-model example, what is the posterior mean and variance for arm 2?

Solutions. (1) \(H_n\) includes the broader recorded sequence, including assignments, observation status, and randomization information; \(D_{n,a}\) isolates the observed outcomes and modeled covariates for arm \(a\). (2) The summary may replace raw outcomes only when it preserves all information needed for the specified posterior calculation and allocation rule. (3) It must select one best arm, possibly at random, with selection probabilities across tied arms summing to one. (4) Its response is not yet known, so treating it as observed inserts unavailable information into the likelihood. (5) Arm 2 has \(n_2=1\) and sum 2, so its posterior is \(N(1,1/2)\): mean 1 and variance \(1/2\).