Regret Relative to the Best Treatment

Why Compare an Adaptive Trial With the Best Treatment?

An adaptive trial assigns treatments as information accumulates. Early participants may receive a treatment that later appears less effective, while assignments made to learn about alternatives can also reduce the reward gathered during the trial. To describe this shortfall in a defined way, compare the trial’s expected cumulative reward with a benchmark: the expected reward that would have been obtained by assigning every participant to the treatment with the highest expected reward.

This benchmark answers a focused question: how much expected reward did the allocation policy leave unrealized, relative to consistent assignment to the best-mean treatment? It does not identify the best treatment from the trial’s observed outcomes, and it does not by itself measure the trial’s overall clinical or scientific value.

Definition and Assumptions

Consider $K$ available treatments and a fixed horizon of $T$ participants, indexed by $t=1,\ldots,T$. Let $A_t\in\{1,\ldots,K\}$ denote the treatment assigned to participant $t$, and let $Y_t(a)$ denote the reward that participant $t$ would produce under treatment $a$. The observed reward is $Y_t=Y_t(A_t)$. Rewards must use a common, consistently oriented scale: larger values represent better outcomes.

For this tutorial, assume a stationary stochastic reward model: each treatment $a$ has a constant mean reward $\mu_a$, and the reward for the current participant, given the assigned treatment and available history, has mean $\mu_a$. Thus $E[Y_t(a)]=\mu_a$ for every $t$. The allocation policy may use previous assignments and observed rewards to choose $A_t$, but the current reward is not known when that assignment is made. All treatments in the benchmark must be available at every decision point.

Definition: The best-treatment mean is $\mu^*=\max_{a\in\{1,\ldots,K\}}\mu_a$. The fixed-horizon best-treatment benchmark is $T\mu^*$, the expected cumulative reward from assigning all $T$ participants to a treatment attaining $\mu^*$. The expected regret relative to this benchmark is $$ R_T=T\mu^*-\sum_{t=1}^{T}E[Y_t] =T\mu^*-\sum_{t=1}^{T}E[\mu_{A_t}]. $$ It is the expected reward gap, in reward units, between the benchmark and the allocation policy over the horizon.

The second expression follows from the model: conditional on assigning treatment $a$, the expected reward is $\mu_a$. The policy’s assignments can be random and depend on earlier data; the expectation averages over that adaptive assignment process as well as reward variation. If $N_a(T)=\sum_{t=1}^T I(A_t=a)$ is the number assigned to treatment $a$, the same quantity can be written $$ R_T=T\mu^*-\sum_{a=1}^{K}\mu_a E[N_a(T)]. $$ This form makes clear that the benchmark depends on the treatment means, while the policy’s expected reward depends on how its assignments are distributed across treatments.

Why the Expected Gap Is Nonnegative

At each decision, the difference between the best-treatment mean and the mean of the assigned treatment is nonnegative, because $\mu^*$ is the largest of the treatment means. The expected total gap is therefore nonnegative as well. The following derivation also establishes why the benchmark can be interpreted as an opportunity cost in expected reward.

\begin{aligned} R_T &=T\mu^*-\sum_{t=1}^{T}E[\mu_{A_t}]\\ &=\sum_{t=1}^{T}\left(\mu^*-E[\mu_{A_t}]\right)\\ &=\sum_{t=1}^{T}E[\mu^*-\mu_{A_t}]\\ &\geq 0, \end{aligned}

The first equality uses the definition of the benchmark and the model-based expected reward. The second uses $T\mu^*=\sum_{t=1}^{T}\mu^*$. In the third, $\mu^*$ is constant, so $E[\mu^*-\mu_{A_t}]=\mu^*-E[\mu_{A_t}]$. Finally, $\mu^*-\mu_{A_t}\geq 0$ for every treatment that could be assigned. This proves nonnegativity without assuming that the policy knows which treatment is best.

Worked Example: A Policy With Unequal Expected Allocation Counts

Suppose a hypothetical trial has three treatments with mean rewards measured in symptom-free days per participant: $\mu_A=6.0$, $\mu_B=7.5$, and $\mu_C=5.0$. The horizon is $T=20$. Across repeated runs of a specified adaptive policy, its expected allocation counts are $E[N_A]=6$, $E[N_B]=10$, and $E[N_C]=4$. These expected counts sum to 20.

Treatment B has the highest expected reward, so $\mu^*=7.5$ symptom-free days per participant. The benchmark is $$ T\mu^*=20(7.5)=150 $$ expected symptom-free days.

The policy’s expected cumulative reward is $$ \sum_a \mu_a E[N_a] =6.0(6)+7.5(10)+5.0(4) =36+75+20=131 $$ expected symptom-free days. Therefore, $$ R_{20}=150-131=19 $$ expected symptom-free days.

Under the stated model, the policy’s expected total reward is 19 symptom-free days below the fixed best-treatment benchmark over 20 participants. This is an average expected-performance comparison, not a claim that every run loses exactly 19 days or that treatment B should automatically be assigned to every future patient.

Worked Example: Randomized Assignments With Known Probabilities

In a second hypothetical example, two treatments have constant mean rewards of $\mu_A=4$ and $\mu_B=6$ points per participant. There are five participants. At each assignment, the policy gives treatment A with probability 0.40 and treatment B with probability 0.60, independently of the previous history. The probabilities sum to one, and each treatment is available for every participant.

The best-treatment mean is $\mu^*=6$ points per participant, so the benchmark is $5(6)=30$ expected points. For each participant, the expected reward is $$ E[\mu_{A_t}]=0.40(4)+0.60(6)=1.6+3.6=5.2 $$ points. Across five assignments, the policy’s expected cumulative reward is $5(5.2)=26$ points. Thus, $$ R_5=30-26=4 $$ expected points.

Equivalently, treatment A has an expected count of $5(0.40)=2$, and treatment B has an expected count of $5(0.60)=3$. The expected gap is $2(6-4)+3(6-6)=4$ points. Both calculations compare expected rewards under the stated assignment probabilities with always assigning B.

Worked Example: Equal Best Means Give a Zero Gap

Suppose three hypothetical treatments have mean rewards, in quality-adjusted days per participant, of $\mu_A=8$, $\mu_B=8$, and $\mu_C=6$. For a horizon of 12, a policy has expected counts $E[N_A]=3$, $E[N_B]=5$, and $E[N_C]=4$. This policy assigns eight expected participants to treatments A and B, both of which attain the best mean.

Here $\mu^*=8$, so the benchmark is $12(8)=96$ expected quality-adjusted days. The policy’s expected reward is $$ 8(3)+8(5)+6(4)=24+40+24=88 $$ expected quality-adjusted days. Its expected gap is $96-88=8$ days. The same result follows from the expected assignments to C: $4(8-6)=8$ days. Allocations between A and B do not contribute to the gap because their means tie for best.

If the expected count assigned to C were zero, the gap would be zero even if the policy split participants between A and B. That would mean the policy matches the benchmark’s expected reward, not that A and B produce identical outcomes for every participant or are interchangeable for all clinical purposes.

What This Benchmark Measures—and What It Does Not

The comparator is a single treatment with the largest expected reward under the specified model. It is not the treatment with the largest observed sample mean: noisy trial data can make an inferior treatment look best in a particular run. Nor is it necessarily the optimal adaptive policy, which could use accumulated information to change assignments over time. The benchmark asks how the policy compares with consistently assigning the best-mean treatment, not how it compares with every possible decision rule.

The expected gap is most interpretable when reward units have direct meaning, such as symptom-free days or a clearly defined utility score. If the reward is a binary outcome coded as 1 for success and 0 for failure, the gap is in expected successes. If a harmful outcome is coded so that larger numbers are worse, the direction must be reversed or the reward must be redefined before applying this formulation. A numerical answer has no useful clinical interpretation until the reward scale and direction are stated.

The best-treatment benchmark also differs from a fixed control benchmark. A control comparison can be important for clinical or regulatory reasons even if the control is not the treatment with the highest expected reward. Likewise, maximizing reward does not automatically account for safety constraints, uncertainty, equity, treatment burden, or the value of learning for future patients. Those considerations may motivate a different objective or additional constraints.

Assumptions, Diagnostics, and Limitations

The calculation relies on assumptions that deserve explicit scrutiny:

Stable treatment means. The same $\mu_a$ is assumed for every participant and time point. If patient mix, clinical practice, or treatment effects change during enrollment, a single best mean may not represent the trial horizon.
Comparable rewards. Each treatment’s reward must be defined on the same scale and oriented so higher is better. Comparing an outcome measured in days for one treatment with an unconverted score for another is not meaningful.
Available benchmark treatment. The treatment attaining $\mu^*$ must be feasible for every participant in the horizon. If a treatment is contraindicated for some patients, assigning it to everyone is not a valid comparator.
Reliable means. The benchmark depends on the $\mu_a$ values. If they are estimated imprecisely, the computed gap inherits that uncertainty; substituting sample averages does not make the best treatment known.
Aligned time horizon. Both quantities must cover the same $T$ participants and the same reward definition. Comparing one policy’s full-trial reward with a shorter benchmark is not a valid gap.

Useful checks include confirming that expected allocation counts sum to $T$, that assignment probabilities at each decision sum to one, and that $\mu^*\geq\mu_a$ for every arm. The calculated gap should not be negative under this definition. A negative value usually signals a reversed reward direction, an arithmetic error, or a mismatch between the benchmark and policy quantities. If the calculation uses estimated means, report that dependence rather than presenting the result as if the true means were known.

With patient-specific treatment effects, the identity of the best treatment may vary across participants. A single global $\mu^*$ still defines a best-average-treatment benchmark under the stated population, but it need not equal the reward from selecting the best treatment separately for each person. That individualized comparator requires additional assumptions and information about treatment effects. Similarly, if means vary over time, a fixed best treatment and a time-specific benchmark are different choices; the comparator must be stated before interpreting the gap.

Common Mistake

Common mistake: “The treatment with the largest observed average in this trial is the benchmark, so the gap is the difference between that sample average and the policy’s reward.” This uses a noisy, data-selected quantity in place of the model’s highest expected reward and can make the benchmark misleading. The correction is to define the comparator using the treatment means $\mu_a$ under the specified reward model, then compare expected cumulative rewards over the same horizon. If the means are estimated, acknowledge the uncertainty in the benchmark.

Check Your Understanding

Answer each question using the expected-reward benchmark defined above. Solutions follow the questions.

Two treatments have means of 3 and 5 points per participant. For a horizon of 8, a policy has expected counts of 3 and 5, respectively. What is the benchmark, the policy’s expected reward, and the expected gap?
Why is the treatment with the largest observed sample average not automatically the best-treatment benchmark?
If two treatments tie for the highest mean, does assigning participants to either one create a gap relative to this benchmark? Explain.
Name two assumptions that would need reconsideration if treatment means changed over calendar time or differed by patient subgroup.

Solutions. (1) The benchmark is $8(5)=40$ points. The policy’s expected reward is $3(3)+5(5)=9+25=34$ points, so the gap is $40-34=6$ points. (2) An observed sample average varies with random outcomes and may be high by chance; the benchmark is defined using expected rewards $\mu_a$, not the winner in one noisy sample. (3) No. A treatment tied at $\mu^*$ has $\mu^*-\mu_a=0$, so assigning to it creates no expected gap under this benchmark. (4) The constant-mean assumption and the definition of a single best treatment for the full population and horizon would need reconsideration. A time-varying or subgroup-specific benchmark must be defined to match the changed setting.