Why a Numerical Comparison Needs More Than Success Counts
A response-adaptive design can assign more patients to a treatment that has appeared more successful. If success probabilities differ, that may increase the expected number of successes during the trial. But the same outcome-dependent assignment can make a simple treatment-effect estimator biased, and a power calculation that treats the final sample sizes as fixed can be misleading. A useful comparison therefore keeps these quantities separate: expected successes, estimator bias, and power.
We use a deliberately small hypothetical trial so that the allocation rule and estimator bias can be worked out explicitly. Its small size also makes an important limitation visible: a normal-approximation power calculation can be computed, but it is not automatically a reliable or calibrated test procedure.
The trial enrolls four patients. The first two form a pilot, with one assigned to each treatment. For each of the remaining two patients, the probability of assignment to A is \(q=0.75\) if the pilot produced an A success and a B failure, \(q=0.25\) if it produced an A failure and a B success, and \(q=0.50\) if the pilot outcomes matched. The two later assignments are independent conditional on the pilot outcomes. The fixed design instead assigns two patients to each treatment, independently of outcomes.
The treatment success probabilities differ by \(p_A-p_B=0.30\), or 30 percentage points. This is the treatment-effect scale used in the example; it is not a claim that the difference is causal in every setting. A causal interpretation requires the assignment and outcome assumptions above, together with a well-defined treatment and outcome.
Expected Assignments and Expected Successes
Under fixed allocation, each arm has two patients. The expected number of successes is therefore \(2p_A+2p_B\). In the adaptive design, the pilot determines the later assignment probability. The four possible pilot patterns have probabilities \(0.28\), \(0.18\), \(0.42\), and \(0.12\), respectively, obtained from independence of the two pilot outcomes.
Worked Example: Comparing Expected Successes
First calculate the pilot-pattern probabilities. An A success and B success has probability \(0.70(0.40)=0.28\); two failures has probability \(0.30(0.60)=0.18\); A-only success has probability \(0.70(0.60)=0.42\); and B-only success has probability \(0.30(0.40)=0.12\). Thus the expected probability of assigning a later patient to A is
There are two later patients, so the expected total number assigned to A is \(1+2(0.575)=2.15\); the expected number assigned to B is \(4-2.15=1.85\). The pilot contributes \(0.70+0.40=1.10\) expected successes. For each later patient, the marginal probability of success is \(0.575(0.70)+0.425(0.40)=0.5725\). Hence
For fixed allocation, \(E(S_{\mathrm{fixed}})=2(0.70)+2(0.40)=2.20\). In this model the adaptive rule yields \(0.045\) more expected successes per four-patient trial. This is an expectation under the stated probabilities, not a guarantee that any particular adaptive trial will have more successes.
The comparison is driven by the rule favoring A after an A-only pilot success. It does not establish that this rule is optimal, nor does the expected count alone summarize patient benefit or the quality of treatment-effect estimation. In particular, a design can improve expected outcomes while changing the sampling behavior of the estimator.
Why the Naive Treatment-Effect Estimate Can Be Biased
Let \(\hat p_A\) and \(\hat p_B\) be the observed sample proportions of successes in the two arms, and consider the naive difference \(\hat p_A-\hat p_B\). Under fixed allocation, the arm sizes are fixed and each sample proportion has expectation equal to its arm’s success probability. Under the adaptive rule, the final arm sizes depend on pilot outcomes. The pilot successes therefore influence both the numerators and the denominators of the final proportions.
A direct way to see the consequence is to condition on the pilot outcomes and on \(K\), the number of the two later patients assigned to A. Given these quantities, the expected naive difference is
where \(i,j\in\{0,1\}\) are the pilot outcomes on A and B. The denominators include the pilot patient in each arm. Averaging this expression over the design gives the unconditional expectation.
Worked Example: Deriving the Adaptive Estimator's Bias
For pilot outcomes \((i,j)=(1,0)\), the assignment probability is \(q=0.75\), so \(K\sim\operatorname{Binomial}(2,0.75)\). Its probabilities for \(K=0,1,2\) are \(0.0625,0.375,0.5625\). The conditional expected differences are \(0.7333,0.6500,0.8000\), respectively. Their weighted mean is
For \((i,j)=(0,1)\), \(q=0.25\). The corresponding conditional expected differences for \(K=0,1,2\) are \(-0.6000,-0.3500,-0.5333\), giving a weighted mean of \(-0.5021\). For either matching pilot pattern, \(q=0.50\); the conditional expectation averaged over \(K=0,1,2\) is \(0.1250\).
Weight these results by the pilot-pattern probabilities:
The true difference is \(0.30\), so the naive estimator's bias under this design and model is \(0.3079-0.30=0.0079\), about 0.79 percentage points upward. With fixed allocation, \(E(\hat p_A-\hat p_B)=0.70-0.40=0.30\), so its bias is zero under the model. This calculation does not say every adaptive rule biases this estimator in the same direction; the result depends on the allocation rule and outcome model.
A Qualified Power Comparison
For orientation, one can form a plug-in normal-approximation calculation for a one-sided test of \(H_0:p_A-p_B=0\) against \(H_a:p_A-p_B>0\) at \(\alpha=0.05\). In this calculation, the fixed design uses its actual arm sizes. The adaptive calculation substitutes expected arm sizes and uses the unconditional mean of the naive estimator just derived. This is a rough planning proxy, not an exact power calculation for a calibrated adaptive test.
Worked Example: Plug-In Normal-Approximation Power
For fixed allocation, the variance of the difference in sample proportions under the specified alternative is approximated by
Thus \(SE=\sqrt{0.2250}=0.4743\). Using the one-sided normal critical value \(z_{0.95}=1.645\), the approximate power is \(1-\Phi(1.645-0.30/0.4743)=0.156\), or about 15.6%.
For the adaptive proxy, substitute the expected arm sizes \(2.15\) and \(1.85\) into the same binomial variance expression:
Using the adaptive estimator mean \(0.3079\), the corresponding approximate power is \(1-\Phi(1.645-0.3079/0.4769)=0.159\), or about 15.9%. These two values are approximate normal-tail calculations. Their difference is not evidence that the adaptive design has higher valid power: the adaptive estimator is biased here, the sample is extremely small, and the calculation does not account exactly for the random arm sizes or calibrate the test's null distribution.
Justification, Diagnostics, and Limitations
The expected-success calculation follows by conditioning on the pilot pattern. For each later assignment, the probability of success is \(q p_A+(1-q)p_B\) conditional on the pilot. Averaging over pilot patterns gives \(E(q)p_A+\{1-E(q)\}p_B\), because the treatment-specific success probabilities are constant and later assignment is randomized using the stated probability. Multiplying by the two later patients and adding the pilot expectation gives the total. This argument depends on the outcome model: if success probabilities change over time or vary with patient characteristics related to assignment, this calculation must be revised.
The bias calculation uses a different key fact: a sample proportion is a ratio with a random denominator. In general, the expectation of a ratio is not the ratio of expectations. Here the pilot outcomes help determine subsequent assignments, and those same pilot outcomes remain in the arm-specific proportions. Averaging the conditional expectations over all pilot patterns and later assignments, rather than treating expected arm sizes as fixed, reveals the bias.
Several checks should accompany any such comparison:
- Check the allocation mechanism. Verify that each assignment probability is implemented as stated and that both arms remain represented. In this example each arm has one pilot patient, so neither sample proportion has an empty-arm denominator.
- Check the outcome assumptions. Compare the assumed success probabilities across time and relevant patient groups. Trends, changing eligibility, or treatment-effect heterogeneity can make a simple two-probability model inadequate.
- Check estimator behavior. Under repeated simulated trials from the assumed model, compare the average naive estimate with the true difference. The calculation above predicts an average of \(0.3079\), not \(0.30\).
- Calibrate testing under the null. A valid test should have rejection probability near its intended \(\alpha\) when \(H_0\) holds. The design's outcome-dependent allocation must be reproduced in that calibration; a fixed-design standard error alone does not establish Type I error control.
Fixed allocation offers predictable arm sizes and a simple unbiased difference in proportions under the stated independent-outcome model. Response-adaptive allocation can shift assignments toward an arm that appears better and, in this example, raises expected successes slightly. But this particular rule is based on only two pilot outcomes, so random early variation can drive later assignments. The resulting estimator bias and test calibration matter even when the expected success count is favorable.
The normal approximation is especially limited here: four observations are far too few for its tail probability to be treated as dependable. A more serious power analysis would specify the exact test statistic and rejection rule, then evaluate its null rejection probability and alternative rejection probability under the complete adaptive randomization procedure, for example by exact enumeration or a carefully validated simulation. Those calculations would need to include the random assignments and outcomes, not just expected arm sizes. The plug-in values above are useful for illustrating the arithmetic and the danger of interpreting a rough power gain as a design guarantee.
Check Your Understanding
Answer each question, then compare with the brief solutions.
- Under the stated adaptive rule, what is the expected number assigned to A?
- What is the fixed design's expected number of successes?
- Why can the naive difference in proportions be biased even though each patient's treatment is randomized?
- What does the adaptive plug-in power value of 0.159 represent, and what does it fail to establish?
Solutions. (1) The expected A assignments are \(1+2E(q)=1+2(0.575)=2.15\). (2) The fixed design has \(2(0.70)+2(0.40)=2.20\) expected successes. (3) Random assignment probabilities depend on pilot outcomes, and those outcomes also enter the final proportions; the resulting random denominators make the naive ratio estimator biased in this example. (4) It is a rough normal-tail approximation under the alternative using expected arm sizes and the calculated estimator mean. It does not establish exact power or Type I error control for a properly calibrated adaptive test.