Why a Worked Comparison Needs More Than a Success Count
A response-adaptive rule can allocate more participants to an arm that has performed better so far. That may increase the expected number of observed successes, but it also makes arm sizes random and links later assignments to earlier outcomes. A fair comparison with fixed allocation therefore needs to track several quantities: expected successes, the variability of the allocation, and the precision available for estimating the treatment effect.
The calculations below use one deliberately simple hypothetical trial. Its purpose is to make the comparison transparent, not to recommend this particular rule. The assumed treatment effects are known for evaluating the designs; in a real trial, they would not be known when assignments are made.
The fixed design assigns exactly 9 participants to each treatment. In the response-adaptive design, an initial pair is assigned one participant to each treatment, with treatment order randomized. After both outcomes are observed, each of the remaining 16 participants is independently assigned to B with probability \(q=0.75\) if B had a success and A a failure; with probability \(q=0.25\) if A had a success and B a failure; and with probability \(q=0.50\) if the two initial outcomes match. Thus both treatments retain a positive chance of assignment at every step.
Expected Successes Under the Two Designs
For fixed allocation, the expected number of successes is the sum of each arm’s sample size multiplied by its success probability. For the adaptive design, first average the assignment probability \(q\) over the possible initial outcomes. Independence of the initial outcomes gives the probabilities of the three cases in the rule.
Worked Example: Expected Adaptive Allocation and Successes
The probability that B succeeds while A fails in the initial pair is \(0.60(1-0.40)=0.36\). The probability that A succeeds while B fails is \(0.40(1-0.60)=0.16\). The outcomes match with probability \(1-0.36-0.16=0.48\). Therefore,
The expected number assigned to B is one initial participant plus 16 times the expected later assignment probability. The expected number assigned to A is calculated in the same way:
For the fixed design, expected successes are \(9(0.40)+9(0.60)=3.6+5.4=9.00\). For the adaptive design, the initial pair contributes \(0.40+0.60=1.00\) expected successes. A later participant has average success probability \(E(q)p_B+[1-E(q)]p_A=0.55(0.60)+0.45(0.40)=0.51\). Thus,
Under these assumptions, the adaptive rule yields \(0.16\) more expected successes over 18 participants. This is an expected-count difference, not a guarantee for any particular trial, nor evidence that the estimated treatment effect will be more precise.
Arm-Size Variability Is Part of the Design
The fixed design has no arm-size variability: \(n_B=n_A=9\) on every trial. In the adaptive design, arm size depends both on which initial outcomes occur and on the random assignments made afterward. The law of total variance separates these sources. Conditional on \(q\), the number of later assignments to B is binomial with 16 trials and success probability \(q\).
Worked Example: Variance of the Adaptive Arm Size
The three possible values of \(q\) have probabilities \(0.36\), \(0.16\), and \(0.48\), respectively. First calculate the second moment and variance of \(q\):
Since \(n_B=1+\operatorname{Binomial}(16,q)\) conditional on \(q\), the conditional mean is \(1+16q\) and the conditional variance is \(16q(1-q)\). Applying the law of total variance gives
The standard deviation of the B-arm size is therefore \(\sqrt{11.16}\approx3.34\) participants. Because \(n_A=18-n_B\), the A-arm size has the same variance. The term \(16E\{q(1-q)\}\) reflects assignment randomness within a given interim state; the term \(256\operatorname{Var}(q)\) reflects variation across interim states. The fixed design’s arm-size variance is zero.
This calculation illustrates why expected arm sizes alone do not describe an adaptive design. Although the adaptive design averages 9.8 participants in B and 8.2 in A, an individual trial can depart substantially from those averages. Very unbalanced realized counts can make a treatment comparison less precise, even when the expected allocation appears reasonable.
Approximate Treatment-Effect Power
For a transparent benchmark, consider a two-sided normal approximation for testing \(H_0:p_B-p_A=0\) at significance level \(\alpha=0.05\), against \(H_a:p_B-p_A\ne0\). Under the specified alternative, the true difference is \(0.60-0.40=0.20\). For arm sizes \(n_A,n_B\), an approximate standard error under that alternative is \(SE\approx\sqrt{p_A(1-p_A)/n_A+p_B(1-p_B)/n_B}\). If the estimated difference is approximated by a normal random variable centered at 0.20 with this standard error, the corresponding approximate power is the probability it falls outside the two-sided null rejection limits, \(\pm1.96SE\).
Worked Example: Approximate Power Under Fixed Allocation
With 9 participants per arm, the alternative standard error is
The standardized difference is \(0.20/0.23094\approx0.8660\). Under the normal approximation, power is
Thus the benchmark power is about \(13.9\%\): if this fixed design were repeated under the specified alternative and the approximation applied, roughly 13.9% of tests would reject the null hypothesis at the 5% level. This is low power, reflecting the small total sample size and the stipulated effect size.
Worked Example: Approximate Power Using Expected Adaptive Arm Sizes
A simple plug-in comparison substitutes the expected adaptive arm sizes, \(n_A=8.2\) and \(n_B=9.8\), into the same alternative standard-error formula:
The standardized difference is \(0.20/0.23186\approx0.8626\). The plug-in power is therefore
This rough calculation gives about \(13.9\%\) power for the adaptive design as well, with fixed allocation slightly higher before rounding. It is a benchmark, not an exact power calculation for the response-adaptive trial. Replacing random arm sizes by their expectations does not account for their variability or their dependence on observed outcomes.
Derivation and Interpretation of the Approximation
The power calculation follows by treating the estimated difference \(\hat{p}_B-\hat{p}_A\) as approximately normal under the alternative, with mean \(p_B-p_A\) and standard deviation \(SE\). If the null-based rejection limits are approximated by \(\pm1.96SE\), the standardized statistic has approximate mean \((p_B-p_A)/SE\) and standard deviation 1. Rejection occurs in either tail. Subtracting the lower rejection limit from the alternative mean and the upper limit from that mean yields the two tail probabilities used above:
This is a deliberately simplified normal approximation using the specified alternative probabilities in the standard error. It is useful for a first comparison, but is not necessarily the exact power of a particular test. In particular, a conventional test’s standard error and critical value may be estimated from the observed data, and an adaptive assignment mechanism can affect the sampling distribution. To report actual operating characteristics, specify the analysis procedure and evaluate its power under the full allocation and outcome process, for example by enumeration when feasible or by simulation.
Comparison, Diagnostics, and Limitations
| Quantity | Balanced fixed allocation | Response-adaptive rule |
|---|---|---|
| Expected participants in B / A | 9 / 9 | 9.8 / 8.2 |
| Expected total successes | 9.00 | 9.16 |
| Variance of B-arm size | 0 | 11.16 |
| Approximate power benchmark | 0.1393 | 0.1386, plug-in only |
The result is specific to the rule, sample size, assumed probabilities, and test approximation. It does not establish that response-adaptive allocation generally increases successes or reduces power. The small expected success-count gain here comes at the cost of variable arm sizes, and the plug-in calculation suggests no power advantage in this example. The earlier material on power and design comparisons distinguishes such operating characteristics from patient-benefit summaries; neither should be treated as a substitute for the other.
A useful diagnostic is to inspect the distribution of realized arm sizes, not only their means. In a simulated evaluation, tabulate the frequency of strongly unbalanced allocations and check that both arms remain adequately represented. Also check whether the assumed response probabilities are plausible and stable over time: changing case mix, delayed outcomes, or misspecified success rates can make the rule’s apparent response to performance misleading. This particular rule uses only two initial outcomes, so its assignment probability can swing substantially based on a very small amount of information.
The main alternative here is balanced fixed allocation, which guarantees equal sample sizes and gives a straightforward benchmark for precision. The adaptive rule may be motivated by a desire to assign more participants to the arm with better observed outcomes, but in this example its expected allocation is only modestly unequal and its expected successes only slightly higher. Neither design is uniformly preferable: the decision depends on the trial’s objective, ethical considerations, analysis plan, and the consequences of allocation variability. These calculations do not derive an optimal allocation target or establish which allocation minimizes variance.
Common Mistake
Check Your Understanding
Answer each question, then compare with the brief solutions below.
- What is the probability that the initial outcomes match, and how does that probability enter \(E(q)\)?
- Why is the fixed design’s arm-size variance zero while the adaptive design’s is positive?
- Calculate the adaptive design’s expected total successes if the expected later assignment probability to B were \(0.50\), leaving all outcome probabilities and sample sizes unchanged.
- Why does substituting expected arm sizes into a standard-error formula not give exact adaptive-design power?
Solutions. (1) The match probability is \(1-0.36-0.16=0.48\); it contributes \(0.48(0.50)\) to \(E(q)=0.55\). (2) The fixed design always assigns 9 to each arm, whereas the adaptive design randomizes later assignments and its probability \(q\) also varies with initial outcomes. (3) With \(E(q)=0.50\), the later average success probability is \(0.50(0.60)+0.50(0.40)=0.50\), so expected total successes are \(1+16(0.50)=9.00\). (4) The realized arm sizes vary and depend on observed outcomes; replacing them by their means omits that joint variation and does not evaluate the test’s actual rejection probability under the adaptive assignment process.