Three Questions, Not One
A response-adaptive trial can be evaluated in more than one way. Did patients assigned during the trial experience many favorable outcomes? How many favorable outcomes might have been lost relative to assigning everyone to the truly better treatment? And how likely was the planned statistical test to detect a treatment difference? These are distinct questions. A design can deliver substantial benefit while having modest power, or have good power without assigning most patients to the treatment that would have been best.
The distinction matters when judging a design: improving the experience of current participants is not the same objective as learning reliably about treatment effects. The calculations below use a binary favorable outcome for clarity. The same distinctions apply to other outcomes if their clinical value and direction are defined appropriately.
For a binary favorable outcome, \(Y_i=1\) denotes success and \(Y_i=0\) denotes failure. Then cumulative benefit is the number of successes, and its unit is favorable outcomes. If the allocation counts are \(N_A\) and \(N_B\), with \(N_A+N_B=n\), and outcomes have treatment-specific means \(\mu_A\) and \(\mu_B\), expected cumulative benefit is
This expression assumes the treatment-specific means remain applicable to the participants assigned each treatment. In an adaptive design the counts can themselves be random; in that case, the expected benefit averages over the possible allocation histories as well as outcomes. A calculation conditional on observed counts is not automatically the design's unconditional expected benefit.
Regret Relative to the Best Treatment
Regret is a comparison with an oracle benchmark: the treatment with the greatest true mean is treated as if it were known in advance and assigned to everyone. That benchmark is useful for evaluating a design, but it is not an achievable allocation rule when the best treatment is unknown. In a trial, investigators have to learn about treatment effects while assigning participants.
For fixed allocation counts, the expected regret is
If \(A\) is truly best, so that \(\mu_A\geq\mu_B\), then this simplifies to \(R=N_B(\mu_A-\mu_B)\). Each participant assigned to \(B\) contributes the expected shortfall \(\mu_A-\mu_B\), while a participant assigned to \(A\) contributes none. This is an expected shortfall, not a claim that the outcomes those participants would have had under the other treatment can be observed.
When allocation counts are random, expected regret averages that shortfall over the allocation process. For example, if \(A\) is best and participant \(i\) receives \(B\) with probability \(q_i\), conditional on the information available at assignment, the expected regret is \((\mu_A-\mu_B)\sum_{i=1}^{n}E(q_i)\), provided the treatment means are stable. Realized outcomes can differ from these expectations.
Worked Example: Benefit and Regret From Allocation Counts
Suppose, hypothetically, a 50-person trial compares treatments \(A\) and \(B\). The favorable-outcome probabilities are \(\mu_A=0.70\) and \(\mu_B=0.60\). For this calculation, suppose 30 participants receive \(A\) and 20 receive \(B\). Treat these counts as fixed; the example is about the arithmetic, not a particular allocation algorithm.
Expected cumulative benefit. The expected number of favorable outcomes is
Thus this allocation is expected to produce 33 favorable outcomes. The actual number could be higher or lower because individual outcomes vary.
Best-treatment benchmark. Since \(0.70>0.60\), treatment \(A\) is best in this hypothetical scenario. Assigning all 50 participants to \(A\) would yield an expected \(50(0.70)=35\) favorable outcomes.
Expected regret. The difference is \(35-33=2\) expected favorable outcomes. Equivalently, \(20(0.70-0.60)=20(0.10)=2\). This benchmark does not tell us which outcomes the 20 participants would actually have experienced on \(A\); those counterfactual outcomes are unobserved.
Power Is a Property of a Test
Statistical power answers a different question: under a specified alternative treatment effect and data-generating model, what is the probability that the planned statistical test rejects \(H_0\)? For example, if \(H_0\) states equal response probabilities for \(A\) and \(B\), power is calculated under a specified alternative such as \(p_A=0.70\) and \(p_B=0.60\). Power depends on the test, its significance level \(\alpha\), the sample size, the allocation process, and the assumed outcome model. It is not the probability that a treatment works, nor is it the proportion of participants who benefit.
Power is not calculated by subtracting the trial's observed success count from an oracle benchmark. It is a probability over repeated trials, each conducted using the specified design and analyzed with the specified test. In a response-adaptive trial, those repetitions must reproduce the allocation updates, because assignment probabilities depend on accumulated information. The inferential procedure must also be appropriate for that design; an ordinary fixed-allocation calculation should not be assumed valid merely because it is familiar.
Worked Example: Power Under a Specified Two-Treatment Model
For a transparent approximation, consider a hypothetical 100-person comparison with a fixed allocation of 60 participants to \(A\) and 40 to \(B\). Suppose the true success probabilities are \(p_A=0.70\) and \(p_B=0.60\). Use a two-sided, large-sample \(z\)-test of equality, at \(\alpha=0.05\), and approximate its rejection boundary using the common-null planning value \(p_0=0.65\). This is an illustrative normal approximation, not an exact power calculation.
Under the common-null planning value, the standard error for the difference in sample proportions is
The two-sided 5% rejection boundary for the estimated difference is approximately \(1.96SE_0=1.96(0.09736)=0.1908\). Under the specified alternative, the estimated difference has approximate mean \(0.70-0.60=0.10\) and standard error
Approximating the estimated difference by a normal distribution under the alternative, the power is the probability that it exceeds \(0.1908\) or falls below \(-0.1908\):
So this approximation gives about 17.7% power for the specified alternative and test. The fixed allocation's expected cumulative benefit is \(60(0.70)+40(0.60)=66\) successes, and its expected regret is \(100(0.70)-66=4\) successes. Those two outcome summaries do not determine the power: power also depends on the rejection rule and sampling variability. In an actual adaptive design, power should be evaluated using the design's allocation mechanism and an inference procedure calibrated for it.
Why the Measures Can Point in Different Directions
Cumulative benefit is an absolute outcome summary. Regret is a relative outcome summary, with the best-treatment benchmark as its reference. Power is an inferential operating characteristic. Their units and interpretations therefore differ: benefit and regret may be expressed in expected favorable outcomes, whereas power is a probability.
For example, if both treatments have high success probabilities and are close in effectiveness, an allocation that gives many participants the slightly inferior treatment may still yield high cumulative benefit and only small regret. Yet a small difference can be difficult to detect, so power may be low. Conversely, a design can have high power for a sufficiently large treatment difference even if many participants were allocated to the inferior treatment early in the trial. Neither example is a contradiction; the measures address different goals.
A useful comparison makes the allocation assumption explicit. Consider again 50 participants and success probabilities \(0.70\) for \(A\) and \(0.60\) for \(B\). If 25 receive each treatment, expected benefit is \(25(0.70)+25(0.60)=32.5\), and expected regret is \(35-32.5=2.5\). If 35 receive \(A\) and 15 receive \(B\), expected benefit is \(35(0.70)+15(0.60)=33\), and expected regret is \(35-33=2\). The second allocation has greater expected benefit and lower expected regret under these assumed means. That arithmetic alone does not establish which design has greater power; power requires a specified test and the distribution of its test statistic under the design.
| Measure | Question answered | Typical scale | Key reference |
|---|---|---|---|
| Cumulative patient benefit | How many favorable outcomes does the trial deliver, on average? | Outcome units, such as successes | Absolute outcomes under the design |
| Regret | How much expected benefit is lost versus assigning everyone to the truly best treatment? | Same outcome units as benefit | Oracle best-treatment benchmark |
| Power | How often does the specified test reject under a specified alternative? | Probability from 0 to 1 | Test, significance level, alternative, and design |
Assumptions, Diagnostics, and Limitations
These calculations need a clearly defined outcome and a meaningful treatment-specific mean. For the binary examples, the favorable event must be defined consistently, and the response probabilities must be interpreted for the participants and conditions represented by the model. The regret calculation assumes the treatment means used to define the best treatment are the true means for the comparison; if the best treatment changes across participant subgroups or over time, a single fixed benchmark may be inappropriate. The examples also treat one participant's outcome as unaffected by other participants' assignments. If there is interference or a changing treatment effect, the simple sum of fixed treatment means may not describe the design's expected outcomes.
In an adaptive design, the allocation counts are random and related to observed outcomes. Report whether an operating characteristic is conditional on a particular allocation sequence or averaged over repetitions of the full design. For power, specify the alternative, test, \(\alpha\), analysis method, and how adaptive assignments are reproduced. For benefit and regret, state the outcome scale, time horizon, assumed treatment means, and benchmark. Simulation can estimate these quantities, but its results are only as relevant as the scenarios and analysis procedures simulated.
A common mistake is to say, “The design has lower regret, so it must have higher power.” This does not follow. Lower regret means the design's expected patient outcomes are closer to the best-treatment benchmark under the assumed means. Power is the rejection probability of a specified test under an alternative. To compare power, calculate or simulate that rejection probability; do not infer it from patient benefit or regret.
A second pitfall is to call the best-treatment benchmark an observed control group. It is not: it represents a counterfactual allocation in which everyone receives the truly best treatment, and for each participant only the outcome under the assigned treatment is observed. The benchmark is useful for design evaluation, but it cannot reveal the unobserved individual outcomes under alternative assignments.
Check Your Understanding
Answer each question using the definitions and distinctions above. Solutions follow.
- In a hypothetical 40-person trial, 24 participants receive \(A\) and 16 receive \(B\). The success probabilities are \(0.75\) and \(0.65\), respectively. Calculate expected cumulative benefit and expected regret.
- Why is regret relative to assigning everyone to the truly best treatment not the number of observed failures among participants assigned to the inferior treatment?
- A design produces an expected 70 successes, compared with 72 under the oracle benchmark. Is its power 70/72? Explain.
- Two designs have the same expected regret under a specified scenario. Does that establish they have the same power? Give a reason.
Solutions. (1) Expected benefit is \(24(0.75)+16(0.65)=18+10.4=28.4\) expected successes. The oracle benchmark is \(40(0.75)=30\), so expected regret is \(30-28.4=1.6\) expected successes, also \(16(0.75-0.65)=1.6\). (2) Regret is an expected difference against a counterfactual best-treatment allocation. Observed failures are actual outcomes under assigned treatments and include random variation; they do not reveal what those same participants would have experienced under the other treatment. (3) No. The ratio \(70/72\) is neither the definition of power nor a rejection probability. Power is the probability that a specified test rejects \(H_0\) under a specified alternative. (4) No. Equal expected regret does not specify either design's test statistic distribution or rejection rule. Their allocation processes and inferential procedures can differ, so power must be evaluated separately.