Two Different Questions for an Adaptive Trial
An adaptive allocation policy can be judged by what it accomplishes during enrollment or by the treatment it selects at the end. These goals are related, but they are not interchangeable. A policy that quickly assigns most participants to a promising treatment may earn high reward during the trial, yet leave too little comparative information to identify the best treatment reliably. Conversely, a policy may deliberately spend enrollment on comparisons that improve the final choice, even when those assignments yield less reward in the short run.
The distinction matters whenever a trial is described as “learning which treatment is best” or as “maximizing patient benefit.” Those phrases can refer to different targets. The first is a terminal decision problem; the second is a reward-accrual problem over the sequence of assignments. Before comparing policies, specify which target is primary and how it will be measured.
Here \(A_t\) is the arm assigned at time \(t\), \(Y_t\) is its observed reward, and \(N_i(T)=\sum_{t=1}^T I(A_t=i)\) is the number of assignments to arm \(i\). The recommendation \(\hat{i}_T\) is a terminal decision based on the information collected by the policy; it need not equal the arm assigned most often. If several arms attain \(\mu^*\), recommending any of them has zero simple regret.
Assumptions and the Cumulative-Reward Identity
For the basic comparison, assume the arm means are stationary: the same \(\mu_i\) applies throughout enrollment. Let \(\mathcal{H}_{t-1}\) denote the information available just before assignment \(t\). The bandit reward assumption is that, for each arm \(i\),
This condition says that, given the pre-assignment history and the selected arm, the expected reward at that assignment is its arm mean. It does not require conditioning on the entire assignment sequence. That distinction is essential: later assignments can depend on earlier rewards, so conditioning on all assignments may reveal information about those rewards.
The expected reward identity follows by conditioning at each time on the information available when the current assignment is made. In particular, sum over the possible selected arms, apply the conditional-mean assumption, and then use linearity of expectation:
The second equality is valid because \(I(A_t=i)\) is determined by the pre-reward assignment and is measurable given \(\mathcal{H}_{t-1}\) and \(A_t\). Summing over \(t\), and using \(\mathbb{E}[N_i(T)]=\sum_{t=1}^T\mathbb{P}(A_t=i)\), gives:
Consequently, expected cumulative regret has the arm-gap decomposition
This is an expected-reward result; it does not say that each realized reward equals its arm mean. It also does not require independent rewards across time. The conditional-mean assumption and finite expectations suffice for the displayed expectation identities. Stronger distributional assumptions, such as independent Bernoulli rewards with fixed success probabilities, are needed for the probability calculations in particular models.
Worked Example: A Two-Arm Pilot-and-Commit Policy
Worked Example: A Two-Arm Pilot-and-Commit Policy
Suppose, hypothetically, that two treatments produce independent Bernoulli rewards with means \(\mu_1=0.60\) and \(\mu_2=0.50\). There are \(T=10\) assignments. A policy assigns one participant to each arm, then assigns the remaining eight to whichever arm had the larger pilot reward; if the pilot rewards tie, it chooses arm 1. The terminal recommendation is that same selected arm.
The true best arm is arm 1, and the reward gap for arm 2 is \(0.60-0.50=0.10\). The policy selects arm 2 only if the pilot reward from arm 1 is 0 and the pilot reward from arm 2 is 1. By independence,
Thus the probability of correctly identifying the best arm is \(1-0.20=0.80\), and expected simple regret is
Arm 2 is assigned once in the pilot and receives the remaining eight assignments with probability 0.20. Therefore, \(\mathbb{E}[N_2(10)]=1+8(0.20)=2.60\). The expected cumulative regret and expected total reward are
The result separates two summaries: the terminal recommendation is wrong with probability 0.20, while the expected cumulative reward is 5.740 Bernoulli successes over ten assignments. A wrong terminal recommendation has simple regret 0.10 in this example; assignments to arm 2 incur a 0.10 expected gap each.
Worked Example: More Reward Can Mean Less Reliable Selection
Worked Example: More Reward Can Mean Less Reliable Selection
Consider a hypothetical comparison with \(T=20\), \(\mu_1=0.70\), and \(\mu_2=0.50\). Suppose a design evaluation reports the following operating characteristics for two policies. These are stipulated summaries of their behavior for this example, not quantities inferred from the arm means alone.
| Policy | \(\mathbb{E}[N_2(20)]\) | Probability terminal choice is arm 2 |
|---|---|---|
| Policy A | 1 | 0.35 |
| Policy B | 4 | 0.10 |
Arm 1 is the best arm and the gap is \(0.70-0.50=0.20\). Under Policy A, expected cumulative regret is \((0.20)(1)=0.20\), so expected cumulative reward is \(20(0.70)-0.20=13.80\). Its expected simple regret is \((0.35)(0.20)=0.070\), and its probability of correct terminal selection is \(1-0.35=0.65\).
Under Policy B, expected cumulative regret is \((0.20)(4)=0.80\), giving expected cumulative reward \(20(0.70)-0.80=13.20\). Its expected simple regret is \((0.10)(0.20)=0.020\), and its probability of correct terminal selection is \(0.90\). Thus Policy A earns 0.60 more expected reward over enrollment, while Policy B has lower expected simple regret by 0.050 and a 0.25 higher probability of a correct terminal choice.
Neither policy is uniformly better without specifying the goal. The calculation illustrates why enrollment reward and terminal identification need separate performance summaries.
Worked Example: Identification Error and Simple Regret Are Not the Same
Worked Example: Identification Error and Simple Regret Are Not the Same
Suppose, hypothetically, there are three arms with means \(\mu_1=0.55\), \(\mu_2=0.53\), and \(\mu_3=0.40\), and \(T=12\). An allocation policy has expected assignment counts \((6,4,2)\). Its terminal recommendation has probabilities 0.55, 0.40, and 0.05 of selecting arms 1, 2, and 3, respectively. Treat these as stipulated policy-evaluation summaries.
Expected cumulative reward is
An oracle receiving the best mean at all 12 assignments would have expected reward \(12(0.55)=6.60\). Hence expected cumulative regret is \(6.60-6.22=0.38\). For terminal identification, the error probability is \(0.40+0.05=0.45\), because either arm 2 or arm 3 is not optimal. But simple regret weights these mistakes by their gaps:
The identification error probability is substantial, yet the expected simple regret is small because arm 2 is close in mean to the best arm. An error indicator treats choosing arm 2 and choosing arm 3 alike; simple regret does not.
Choosing and Diagnosing the Objective
A design should be evaluated against the decision it is meant to support. If the main purpose is to recommend a treatment after enrollment, report terminal identification measures, such as the probability of selecting an optimal arm and expected simple regret. If the main purpose is to improve reward during enrollment, report expected cumulative reward or expected cumulative regret. When both aims matter, report both rather than allowing one to stand in for the other.
| Evaluation target | What it asks | Typical summary | What it can miss |
|---|---|---|---|
| Best-arm identification | Is the final recommendation optimal, or close to optimal? | Correct-selection probability; expected simple regret | Reward forgone during enrollment |
| Cumulative reward | How much reward was accrued across all assignments? | Expected total reward; expected cumulative regret | Whether the final recommendation is reliable |
| Hybrid objective | How well does the policy perform on both goals? | A pair of metrics or an explicitly defined combined utility | Trade-offs hidden by an unspecified weighting |
The main alternatives are therefore objective choices, not merely different names for the same policy. Policies emphasizing exploitation of currently promising arms may reduce cumulative regret, while policies that maintain comparisons can improve terminal decisions. Earlier material on explore-then-commit, UCB, and successive elimination illustrates allocation strategies; their evaluation still depends on whether the target is reward accrued, final identification, or both. A hybrid criterion is meaningful only when its combination rule is stated—for example, a pre-specified weighted utility with a justified scale and weight.
Several diagnostics help reveal objective mismatch. Check that the reported metric corresponds to the stated purpose; compare terminal error and cumulative regret rather than assuming one determines the other; and examine performance across plausible reward gaps. Near-tied treatments can yield frequent selection errors but little simple regret, whereas a wrong choice across a large gap can be consequential. In practical evaluation, uncertainty in estimated operating characteristics should also be reported, especially when they are obtained by simulation.
The basic identities have limitations. They rely on fixed arm means and a well-defined reward for each assignment. If treatment effects change over calendar time, outcomes are delayed, participants interfere with one another, or the reward definition omits important clinical outcomes, the stationary-mean comparison may not answer the relevant question. In addition, the oracle benchmark assumes the best arm could be assigned to everyone and ignores constraints; if clinical, ethical, or operational constraints apply, the benchmark must reflect them. Maximizing a numerical reward is not automatically the same as maximizing clinical benefit.
Common Mistake
- Wrong reasoning: “The policy has low cumulative regret, so it must identify the best treatment well.” Low regret can result from assigning often to a treatment with a mean close to the best, even if the final ranking remains uncertain.
- Correction: Cumulative regret weights each assignment by its reward gap; terminal identification evaluates a final choice. Calculate and report the relevant measures separately.
- Wrong conditioning: “Given the full sequence of assignments, each observed reward has mean \(\mu_i\) for its assigned arm.” Later assignments may depend on earlier outcomes, so the full sequence can reveal information about those outcomes.
- Correction: Use the conditional mean given the history before assignment and the arm selected at that time, then apply iterated expectation. Do not use conditioning on the completed adaptive assignment sequence to justify the cumulative-reward identity.
Check Your Understanding
Work through each question, then compare your reasoning with the solutions below.
- With two arms, \(\mu_1=0.8\) and \(\mu_2=0.7\), a policy has \(\mathbb{E}[N_2(10)]=3\). Find its expected cumulative regret relative to always receiving the best mean.
- In the same setting, the terminal recommendation selects arm 2 with probability 0.25. Find the probability of correct selection and expected simple regret.
- Why is conditioning on the complete adaptive assignment sequence not generally a valid justification for treating every assigned reward as having conditional mean \(\mu_i\)?
- Two policies have the same expected cumulative regret, but one recommends an optimal arm more often. Can either be declared better without specifying the goal? Explain.
Solutions. (1) The gap is \(0.8-0.7=0.1\), so expected cumulative regret is \((0.1)(3)=0.30\) reward units. (2) The correct-selection probability is \(1-0.25=0.75\); expected simple regret is \((0.25)(0.1)=0.025\) reward units. (3) Future assignments may depend on earlier rewards, so observing the full assignment sequence can reveal information about those rewards; the valid conditional mean is given the pre-assignment history and the selected arm. (4) No. The first policy is not distinguished by the stated cumulative-regret measure, while the other is better on terminal identification; the preferred policy depends on which objective is primary or how a hybrid objective is defined.