Why Compare Designs Across More Than One Goal?
A response-adaptive design can direct more participants to a treatment that appears promising as outcomes accrue. That may improve expected patient benefit, but it does not automatically improve estimation or power. The same outcomes that influence later assignments can also influence which observations enter each treatment group. A fair comparison therefore separates at least three questions: how participants are allocated, how many responses are expected, and how well the design supports a treatment comparison.
The calculations below use a small, deliberately transparent example to expose the dependencies. A separate larger-sample calculation gives a power benchmark. Neither example is evidence about a real trial; both are hypothetical.
Outcome Model and Assumptions
Suppose there are two treatments, A and B, and participant outcomes follow a Bernoulli model. For each participant, the potential response under A has probability \(p_A\), and the potential response under B has probability \(p_B\). The treatment effect on the response-probability scale is \(p_B-p_A\). For the calculations here, participants’ potential outcomes are independent across participants, each participant’s response probability is stable within treatment, and the observed outcome is the outcome under the assigned treatment. There is no interference between participants.
In the adaptive example, each response is observed before the next assignment. Thus each assignment may depend on past observed outcomes, but not on the current participant’s unobserved outcome. This is important: randomizing the current participant before observing that participant’s response allows the assignment probability to depend on the past without allowing it to select the current participant’s response.
Under these assumptions, the expected total number of responses has a useful unconditional expression. Write \(N_A\) and \(N_B\) for the numbers assigned to A and B. Since each participant’s assignment is made before that participant’s outcome is observed,
This identity averages over the allocation process and the outcomes together. It does not imply that the expected response count conditional on the final allocation totals is \(p_A N_A+p_B N_B\). In an adaptive design, those totals may themselves reveal information about earlier outcomes. Conditional expectations given the final totals must account for that selection.
Worked Example: Allocation and Expected Patient Benefit
Worked Example: Allocation and Expected Patient Benefit
Suppose \(p_A=0.30\) and \(p_B=0.70\), and three participants are enrolled. The first participant is assigned to A and the second to B. For the third participant, assign B with probability 0.80 if the second participant responded, and with probability 0.20 if the second participant did not respond. Otherwise assign A. Compare this rule with a fixed design that assigns the third participant to B with probability 0.50, independently of outcomes.
Let \(I=1\) when participant 3 is assigned to B. Under the adaptive rule,
Consequently, \(E[N_B]=1+0.62=1.62\) and \(E[N_A]=3-1.62=1.38\). The expected total responses, calculated unconditionally, are
For the fixed design, \(E[N_A]=E[N_B]=1.50\), so the expected total is \((0.30)(1.50)+(0.70)(1.50)=1.500\). In this particular model and rule, adaptation increases expected responses by \(1.548-1.500=0.048\), or 0.016 responses per enrolled participant on average. That is a modest expected benefit, not a guarantee that any individual trial will have more responses.
The calculation uses expected allocation totals, not a response-count calculation conditional on the realized totals. For example, the number assigned to B is larger when the second participant responds, so conditioning on the final number assigned to B changes the distribution of the outcomes that helped determine that number.
Outcome-Driven Allocation Can Affect the Estimator
An often-reported treatment contrast for binary outcomes is the difference between the observed sample response proportions, \(\hat p_B-\hat p_A\). Under fixed random allocation, each arm’s sample size is determined independently of outcomes, and the arm-specific sample means are unbiased for their response probabilities under the model above. Under response adaptation, random sample sizes and outcome-dependent assignment can make the expectation of this ratio estimator different from \(p_B-p_A\). This does not mean that every adaptive estimator is biased; it means that unbiasedness cannot be assumed from the treatment randomization alone.
Worked Example: Exact Bias in a Three-Participant Adaptive Rule
Use the same three-participant rule and probabilities as in the previous example. Let \(Y_{B1}\) be the response of the participant assigned to B at the second assignment, and let \(Y_{B2}\) be the third participant’s potential response under B. The third participant’s B-assignment indicator \(I\) has \(P(I=1)=0.62\). Because assignment depends on \(Y_{B1}\),
When \(I=1\), the B sample mean is \((Y_{B1}+Y_{B2})/2\); when \(I=0\), it is \(Y_{B1}\). The third participant’s potential outcome is independent of the earlier response and assignment decision, so \(E[IY_{B2}]=P(I=1)p_B=(0.62)(0.70)=0.434\). Therefore,
The A sample contains the first participant and, when \(I=0\), the third participant. Since \(I\) depends on the earlier B outcome and not on either A potential outcome, the expected A sample mean is \(E[\hat p_A]=0.30\). Thus
For this particular estimator and rule, the expectation is 0.063 below the true response-probability difference. The cause is visible in the rule: when the first B response is a failure, B is less likely to receive the third participant, leaving a one-person B sample that is especially likely to have a low observed rate. This exact calculation concerns the expectation across all trials; it is not a calculation conditional on final arm sizes.
Power: A Benchmark, Not a Shortcut
Power is the probability that a specified test rejects \(H_0\) when a specified alternative is true. It depends on the test, significance level \(\alpha\), sample size, outcome model, and allocation rule. For an adaptive design, a power calculation must respect the joint process that generates outcomes and assignments. Replacing random adaptive sample sizes with their expectations can be useful as a rough benchmark, but it does not in general give the design’s actual power.
Worked Example: Approximate Power Under Unequal Expected Allocation
Consider a hypothetical trial with 120 participants, \(p_A=0.30\), and \(p_B=0.50\). Compare equal fixed allocation of 60 per arm with a notional adaptive allocation averaging 42 to A and 78 to B. For a transparent benchmark, use the large-sample two-sided score-test approximation for \(H_0:p_A=p_B\), with \(\alpha=0.05\). This calculation treats the displayed arm sizes as fixed; it is not an exact power calculation for a response-adaptive design.
Under the alternative, the estimated standard deviation of the difference in sample proportions is approximated by \(\sqrt{p_A(1-p_A)/n_A+p_B(1-p_B)/n_B}\). Under the null, use the pooled probability \(\bar p=0.40\) to approximate the score-test standard error and its two-sided critical boundary.
For equal allocation, the null standard error is \[ SE_0=\sqrt{0.40(0.60)(1/60+1/60)}=0.08944, \] so the approximate critical boundary is \(1.96(0.08944)=0.1753\). Under the alternative, \[ SE_1=\sqrt{0.30(0.70)/60+0.50(0.50)/60}=0.08756. \] The estimated difference has mean \(0.20\). Its probability of exceeding the positive boundary is approximately \(\Phi((0.20-0.1753)/0.08756)=\Phi(0.282)\approx0.611\); the lower-tail probability is negligible. The approximate power is therefore about 0.611.
For the 42/78 allocation, the null standard error is \[ SE_0=\sqrt{0.40(0.60)(1/42+1/78)}=0.09376, \] giving a critical boundary of \(1.96(0.09376)=0.1838\). The alternative standard error is \[ SE_1=\sqrt{0.30(0.70)/42+0.50(0.50)/78}=0.09058. \] The positive-tail approximation is \(\Phi((0.20-0.1838)/0.09058)=\Phi(0.179)\approx0.571\), again with a negligible lower tail. This benchmark is lower than the equal-allocation benchmark, despite directing more participants to B. Unequal allocation can reduce precision for a treatment contrast, particularly when the less-allocated arm contributes substantially to its variance.
These numbers are fixed-count normal approximations, not a conclusion that every adaptive design has power 0.571. Actual adaptive power must be evaluated under the complete allocation rule, with an inferential procedure appropriate to that rule, typically by repeated-trial simulation. Such simulation should also evaluate type I error under \(H_0\); a nominal 5% test is not necessarily a 5% test when its usual assumptions fail.
How to Read the Comparison
| Goal | Equal fixed allocation | Response-adaptive allocation |
|---|---|---|
| Expected patient benefit | Can be lower if one treatment is substantially better, because allocation does not shift toward it. | May improve under a favorable model and rule, but is not guaranteed and depends on early outcomes. |
| Estimation | Arm sizes are set independently of outcomes, simplifying standard estimation. | Outcome-driven arm sizes can affect the distribution and bias of naive sample-mean contrasts. |
| Power | Often benefits from balanced allocation for comparing two treatments with similar outcome variances. | Depends on the allocation path and analysis; a larger count on the better treatment does not by itself imply greater power. |
| Operating-characteristic assessment | Often admits direct calculations under standard models. | Requires evaluating the sequential assignment rule and outcome process jointly. |
The comparison is model-dependent. A larger treatment difference, different outcome variability, a different reward definition, or a different adaptation rule can change the balance among benefit, bias, and power. Expected benefit is also not the same objective as statistical precision: enrolling more participants on one treatment may increase expected responses while making the treatment contrast less precise.
Diagnostics, Pitfalls, and Limitations
A useful operating-characteristic assessment evaluates the whole trial repeatedly under plausible models, rather than reporting only the final allocation ratio. At minimum, examine the distribution of treatment counts, expected responses, estimator bias, interval coverage, type I error under \(H_0\), and power under meaningful alternatives. Examine variability across trial replications as well as averages: two designs with similar mean allocations can have very different probabilities of extreme imbalance.
- Conditioning on final totals as if they were fixed in advance. The incorrect reasoning is that, if a trial ends with \(N_A\) and \(N_B\) assignments, its expected response count must be \(p_AN_A+p_BN_B\). In an adaptive design, the final totals can be selected by earlier outcomes. The correction is to calculate unconditional expectations using the sequential assignment rule, or to derive the conditional distribution given final totals explicitly.
- Equating more responses with more power. Extra expected responses can improve patient benefit but do not guarantee a more precise treatment contrast. Compare the variance and the test’s operating characteristics, not just the number of successes.
- Using a fixed-design test without checking its assumptions. Outcome-dependent assignments can alter the distribution of common estimators and test statistics. Check type I error, interval coverage, and power under the actual design; do not infer validity from the test’s nominal \(\alpha\) alone.
- Trusting one allocation path. A single simulated or observed trial can look unusually favorable or unfavorable. Assess repeated-trial distributions and sensitivity to the response probabilities and allocation rule.
The small exact example isolates one source of estimator bias, but it does not establish that all response-adaptive designs produce the same direction or magnitude of bias. Likewise, the power benchmark isolates the effect of unequal fixed counts, but it omits the dependence between adaptive assignments and past outcomes. A design comparison should state which quantities are exact, which are approximate, and which require simulation under the full procedure.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- In the three-participant example, what is the probability that the third participant receives B when \(p_B=0.70\)?
- Why is \(p_A N_A+p_B N_B\) not generally the conditional expected response count given the final adaptive allocation totals?
- In the exact bias example, what is the bias of \(E[\hat p_B-\hat p_A]\) relative to \(p_B-p_A\)?
- Why is the 42/78 power calculation a benchmark rather than the actual power of a response-adaptive design?
Solutions. (1) The probability is \((0.70)(0.80)+(0.30)(0.20)=0.62\). (2) Final totals depend on earlier outcomes, so conditioning on those totals can select outcome histories; the conditional expectation requires the allocation rule and that selection. (3) The bias is \(0.337-0.400=-0.063\), or 0.063 below the true difference. (4) It treats the arm sizes as fixed and therefore omits their outcome-dependent variation and the resulting joint distribution of assignments and outcomes.