What Must Be Specified?
A clinical bandit problem is a sequential allocation setup: patients arrive over time, a treatment is assigned to each patient, and that patient’s outcome becomes available to inform later decisions. Before discussing any allocation rule, the problem needs a clear description of what can be assigned, what is observed, how outcomes are distributed, and how many decisions are made. These choices determine what a proposed policy is actually operating on.
The basic formulation below treats patients as arriving one at a time and uses a fixed number of enrollment decisions. It does not prescribe a particular allocation algorithm or decide which clinical objective is most important. Those are separate choices. The central task here is to specify the decision problem precisely enough that an allocation rule and its performance can be interpreted.
Arms, Decisions, and History
Let \(K\) be the number of available treatment arms, indexed by \(a\in\{1,\ldots,K\}\). At patient arrival \(t\), where \(t=1,\ldots,T\), let \(A_t\) denote the arm assigned to that patient, and let \(Y_t\) denote the outcome subsequently observed. The decision is made using information available before that assignment. Write the observed history before patient \(t\) as
If baseline characteristics are used by the allocation rule or outcome model, denote the information available for patient \(t\) by \(X_t\), and include it in the pre-assignment information. A randomized allocation policy can be described by probabilities \(\pi_t(a\mid H_{t-1},X_t)\), where \(\pi_t(a\mid H_{t-1},X_t)\) is the probability of assigning arm \(a\) for patient \(t\), given the information then available. These probabilities are nonnegative and sum to one over the available arms. A deterministic rule is the special case in which one arm has probability one.
The history matters because allocation can adapt: the probability of assigning a treatment to a later patient may depend on responses from earlier patients. This does not change the definition of the arms or outcomes, but it changes the joint distribution of the observed assignments and responses. In particular, the full sequence of assignments is not generally independent of earlier outcomes.
The Sequential Outcome Model
For a simple stationary model, let \(F_a\) be the distribution of the outcome under arm \(a\). The essential assumption is sequential: conditional on the model parameters, the information available before patient \(t\), and the assignment \(A_t=a\), the outcome \(Y_t\) has distribution \(F_a\) and does not otherwise depend on the past. The same arm-specific distribution is used at each decision time. If outcomes depend on patient characteristics, a more appropriate assumption specifies a conditional distribution \(F_a(\,\cdot\mid X_t)\), rather than pretending every patient has the same reward distribution.
Here \(B\) is any set of possible outcomes and \(\theta\) denotes fixed model parameters. In the stationary version without covariates, the right-hand side is simply \(F_a(B\mid\theta)\). This assumption says that, once the current patient’s relevant information, current arm, and model parameters are specified, earlier outcomes do not add further information about that patient’s outcome. It does not say that observed outcomes are independent after conditioning on the entire adaptive assignment sequence.
The distinction is important. Under an adaptive policy, an assignment later in the trial can reveal information about earlier outcomes because the policy used those outcomes to make its decisions. The sequential outcome model concerns what happens at the moment each patient is assigned; it should not be replaced by an independence claim conditional on the completed sequence of assignments.
The outcome may itself be the quantity of interest, such as a binary response or a count. Alternatively, the analysis may define a reward \(R_t=g(Y_t)\), where \(g\) maps the observed clinical outcome to a numerical value representing a specified utility. If reward is used, the formulation must state what \(g\) is and which direction is preferred. A distribution for \(R_t\) is then induced by the outcome distribution and this mapping. The utility definition is part of the problem, not a harmless relabelling.
Worked Example: Two Arms and a Fixed Patient Horizon
Suppose, hypothetically, a trial compares standard care (arm 1) with a new regimen (arm 2). The outcome is whether each patient has a defined clinical response by day 30, coded \(Y_t=1\) for response and \(Y_t=0\) otherwise. Assume a stationary model with response probabilities \(p_1=0.60\) and \(p_2=0.75\). The planned horizon is \(T=4\) patients.
Each patient-level decision is \(A_t\in\{1,2\}\). Under the stated model, for either arm \(a\), \(Y_t\) is Bernoulli with response probability \(p_a\), conditional on the pre-assignment history and \(A_t=a\). The assumption requires that response probabilities do not change across these four patients, after accounting for the model, and that the current response distribution does not otherwise depend on earlier history.
For illustration, consider a nonadaptive schedule assigning two patients to each arm. The expected total number of responses is
The corresponding expected response proportion across the four assignments is \(2.70/4=0.675\), or 67.5%. This calculation describes the specified schedule and model; it does not imply that an adaptive policy will assign exactly two patients to each arm. The arms, outcome, model, and horizon are explicit, while the allocation policy is a separate specification.
From a Clinical Outcome to a Reward Distribution
A reward model should preserve the clinically relevant distinctions in the outcome. For instance, if response and toxicity both matter, coding response alone as the reward would omit toxicity. A utility mapping can combine these features, but its values and interpretation must be stated before comparing arms. The following example shows that defining reward is a modeling choice with direct consequences.
Worked Example: Defining Utility for Response and Toxicity
Consider a hypothetical two-arm trial. For each patient, record response and whether a specified toxicity occurs. Define the reward as response indicator minus \(0.4\) times the toxicity indicator. Thus response without toxicity has reward 1, response with toxicity has reward 0.6, toxicity without response has reward \(-0.4\), and neither event has reward 0. These are hypothetical utility units, not clinical estimates.
Suppose the joint outcome probabilities are as follows. In arm A, the probabilities for response with toxicity, response without toxicity, no response with toxicity, and neither are \(0.30, 0.40, 0.10,\) and \(0.20\), respectively. They sum to \(1.00\). In arm B, the corresponding probabilities are \(0.45, 0.25, 0.20,\) and \(0.10\), also summing to \(1.00\).
For arm A, the expected reward is
For arm B, it is
Under this particular reward definition and these hypothetical distributions, arm A has the larger expected reward by \(0.54-0.44=0.10\) utility units per patient. This comparison is not a statement that arm A has a higher response probability: arm B has more response-with-toxicity outcomes in the specified table. It is a comparison under the declared utility scale. A different clinically justified penalty for toxicity could change the comparison.
Horizon and Joint-Distribution Justification
A fixed patient horizon \(T\) means that the trial makes exactly \(T\) sequential assignment decisions in the basic formulation. For example, \(T=200\) refers to 200 enrolled patients, not necessarily 200 calendar days. A calendar-time horizon can interact with recruitment rates, delayed outcomes, and patient mix, and therefore requires additional specification. Likewise, a rule that stops early when a condition is met is a different stopping design from a fixed-\(T\) problem.
The sequential assumptions imply a useful factorization of the joint law when the covariate sequence is treated as fixed, or is exogenous such that conditioning on the entire sequence leaves the sequential assignment and outcome laws valid. Let \(\pi_t(a_t\mid h_{t-1},x_t)\) describe the allocation policy, and let \(f_{a_t}(y_t\mid x_t,\theta)\) denote an outcome probability mass or density under the current arm. Applying the chain rule one decision at a time gives
The factorization follows because, at each time, the assignment is drawn according to the policy using the available history, and then the outcome is generated from the specified conditional outcome distribution for the assigned arm. It is a sequential factorization, not a claim that \(Y_1,\ldots,Y_T\) are independent conditional on the completed assignments \(A_1,\ldots,A_T\). That distinction is essential when assignments depend on earlier responses.
Worked Example: Adaptation Can Select on an Earlier Outcome
Consider a hypothetical two-patient trial with two arms. Patient 1 is assigned arm 1. The response is Bernoulli with probability \(0.50\) under either arm, and each new patient’s response follows the arm-specific distribution conditional on the pre-assignment history. The policy assigns patient 2 to arm 1 if \(Y_1=1\), and to arm 2 if \(Y_1=0\). The horizon is \(T=2\).
Since \(A_2=1\) exactly when \(Y_1=1\), we have \(P(A_2=1)=0.50\), and
Thus the first observed outcome does not retain its original Bernoulli distribution after conditioning on the full assignment sequence, because the second assignment was selected using that outcome. This does not violate the sequential outcome model. For patient 2, conditional on the history and chosen arm, the new response still has probability \(0.50\). For example, the probability of the path \(Y_1=1,A_2=1,Y_2=0\) is
The path probability is computed in the order decisions and outcomes occur. Treating the two observed outcomes as independent conditional on the completed assignments would not describe this adaptive design correctly.
Assumptions, Diagnostics, and Alternatives
A simple formulation is useful only if its assumptions are plausible for the trial being described. The stationary arm-specific model is most credible when the response definition and assessment window are consistent, relevant patient differences are either limited or modeled, and there are no important time trends in outcome generation. If treatment effects vary by patient type, specifying \(F_a(\,\cdot\mid X_t)\) may be more appropriate than one distribution \(F_a\) for everyone. If care changes during enrollment, outcomes are delayed, or a treatment’s effect depends on earlier treatment exposure, a basic stationary model may fail.
Useful checks include reviewing outcome coding and assessment timing, plotting response summaries against enrollment time and important baseline groups, and comparing observed patterns with the stated model. Such checks are diagnostic rather than definitive: with small samples, an apparent absence of change does not establish stationarity. If assignment probabilities become extremely small for an arm, the data may also provide little information about that arm’s outcome distribution in later parts of the trial. The problem specification should state whether all arms remain available and whether there are any assignment constraints.
A fixed-allocation randomized trial uses assignment probabilities that do not adapt to prior outcomes, whereas a bandit formulation permits the probabilities to depend on the evolving history. Fixed allocation can make planned arm comparisons and sample sizes straightforward; an adaptive policy changes the distribution of arm counts and the dependence structure of the observed data. Neither description alone specifies the clinical objective. A trial may prioritize patient outcomes during enrollment, learning about treatments, or other design criteria, and these should not be conflated.
Common failures begin with an underspecified problem: “compare the treatments” leaves unclear which outcomes count, “use the best response” leaves the utility undefined, and “run for six months” does not state how many patients or what happens to delayed outcomes. Another error is to claim that outcomes are independent conditional on all observed assignments. Under an adaptive policy, later assignments can encode earlier outcomes; the correct assumption is the sequential conditional outcome model stated above. If the model’s conditional distribution changes with history in ways not represented by \(X_t\) or \(\theta\), the simple formulation is inadequate and should be revised rather than defended by an independence claim.
Check Your Understanding
Answer each question using the definitions and assumptions in this tutorial. Brief solutions follow.
- A trial has three treatments and enrolls 120 patients. What are \(K\), \(T\), and the patient-level decision?
- State the sequential outcome assumption for a binary response under arm \(a\), with response probability \(p_a\).
- Why can conditioning on a later assignment alter the distribution of an earlier outcome in an adaptive trial?
- If a reward is defined as response minus \(0.5\) times a toxicity indicator, what reward is assigned to a patient with no response and toxicity?
Solutions: (1) \(K=3\), \(T=120\), and each decision is the assignment \(A_t\in\{1,2,3\}\) for patient \(t\). (2) Conditional on the model parameters, pre-assignment history, and \(A_t=a\), \(Y_t\) is Bernoulli with probability \(p_a\), and it does not otherwise depend on the past under the stated model. (3) If the policy uses earlier outcomes to choose a later arm, observing that later assignment provides information about those earlier outcomes; it therefore need not leave their marginal distribution unchanged. (4) The reward is \(0-0.5(1)=-0.5\) utility units.