Two Goals That Can Pull in Different Directions
A response-adaptive trial uses accumulating outcome information to change the probabilities of assigning future participants to treatment arms. With delayed outcomes, the information available at an allocation decision may describe earlier participants rather than the current state of evidence. The rule therefore has to be judged on more than one dimension: how participants are expected to fare, how informative the treatment comparison will be, and when that comparison can be made.
These objectives are not interchangeable. Allocating more participants to a treatment believed to be better can raise expected participant benefit. But if fewer participants receive the other treatment, the estimated treatment contrast will generally be less precise. Waiting for outcomes to resolve may improve the information used in a decision but can slow enrollment decisions or delay conclusions. Predicting pending outcomes can make decisions sooner, but those predictions introduce model dependence.
There is no allocation rule that is automatically best on all these dimensions. A useful assessment states the target, the assumptions, and the cost being accepted. The calculations below are illustrations under explicit assumptions, not claims about a real trial.
Assumptions for Simple Comparisons
Use two arms, labelled 1 and 0, and let \(Y\) be an outcome measured after treatment assignment. For a success outcome, take \(Y=1\) for success and \(Y=0\) otherwise; for a continuous outcome, use its value in the stated units. Let \(p_1\) and \(p_0\) denote success probabilities in a binary-outcome illustration, and let \(\mu_1\) and \(\mu_0\) denote mean outcomes in a continuous-outcome illustration.
The basic calculations require care. For expected outcome calculations, assume that the stated arm-specific outcome means or probabilities are appropriate for the participants being considered, treatment assignment is implemented as specified, and the outcome scale and follow-up are comparable across arms. These are simplifying assumptions: in a real trial, treatment effects may vary among participants.
For a standard-error comparison using the formula below, assume independent outcomes across participants, a common outcome variance \(\sigma^2\) in the two arms, and fully observed outcomes from the counts being used. If outcomes are pending, plugging in expected resolved counts is only a rough approximation. It additionally assumes that resolution is not selectively related to outcome in a way that changes the observed comparison. When these conditions are doubtful, the simple calculation is not a substitute for a model that addresses the actual data and delay process.
An allocation rule may use outcome information available at the time of each assignment. With delayed feedback, that information can be stale. The participant-benefit calculations below treat a stated belief about which arm is better as a premise; they do not assume that a stale estimate is necessarily correct.
Expected Outcomes and Comparison Precision
Suppose a fraction \(r\) of participants is assigned to arm 1 and the rest to arm 0. If arm-specific success probabilities are \(p_1\) and \(p_0\), the expected success probability averaged over assignments is the allocation-weighted average. For a continuous outcome, the same argument applies to the means.
This identity follows by conditioning on assigned arm: arm 1 occurs with probability \(r\) and has expected outcome \(p_1\), while arm 0 occurs with probability \(1-r\) and has expected outcome \(p_0\). It describes expected outcomes under the assumed probabilities; it does not establish that the rule has correctly identified the better treatment.
For a comparison of arm means with \(n_1\) and \(n_0\) independent observations and common variance \(\sigma^2\), the variance of the difference in sample means is \(\sigma^2(1/n_1+1/n_0)\). Therefore its standard error is:
The formula shows why extreme allocation ratios can harm comparison precision: reducing either arm's sample size increases its contribution \(1/n_a\) to the variance. The formula assumes a common known standard deviation for this illustration; in an analysis, the variance is generally estimated, and inference must reflect the design and the data actually collected.
Worked Example: More Participants on the Better-Performing Arm
Suppose, hypothetically, that 100 participants will be assigned between two treatments with success probabilities \(p_1=0.60\) and \(p_0=0.40\). Assume these probabilities are known for this illustration and do not vary across participants. Compare equal allocation with 75% allocation to arm 1.
Under equal allocation, \(r=0.50\). The expected success probability is:
Thus 100 participants have 50 expected successes. Under 75:25 allocation, \(r=0.75\), so:
That is 55 expected successes among 100 participants, or 5 more than under equal allocation. The gain follows because the assumed better-performing arm receives more assignments.
Now assess the precision of a continuous-outcome comparison with common standard deviation \(\sigma=4\) points. Under 75:25 allocation, \(n_1=75\) and \(n_0=25\):
Under equal allocation, \(n_1=n_0=50\), giving:
The 75:25 rule improves expected success in this stipulated scenario, but gives a less precise mean comparison: its standard error is about 0.924 points rather than 0.800 points. Neither calculation says which objective should take priority; that depends on the trial's purpose and ethical and design constraints.
Why Delay Changes the Trade-Off
An adaptive rule can direct assignments using the latest available outcomes, but “latest available” need not mean “most relevant to current treatment performance.” If outcomes resolve slowly, many assignments may be made before new information arrives. Decisions during that interval may behave much like decisions under a fixed allocation, or may continue to reflect a possibly outdated ranking of treatments.
A delay can also affect the timing of comparison separately from its eventual precision. A trial may ultimately observe many outcomes in both arms, yet not have enough resolved outcomes for a useful interim decision at the desired calendar time. Conversely, waiting for more outcomes can improve the evidential basis for a decision but postpones it. A precision calculation based on total eventual sample size does not describe when that precision becomes available.
One way to express the effect of decision accuracy in a simple binary setting is to let \(q\) be the probability that a rule selects the truly better arm when it commits to one arm. Suppose the better arm has success probability \(p_B\), the worse arm \(p_W\), and \(p_B>p_W\). If the selected arm is the better one with probability \(q\), the expected success probability per assignment is \(q p_B+(1-q)p_W\). This is an illustrative measure of the quality of the treatment ranking, not a general property of every adaptive allocation rule.
Worked Example: A Delayed Ranking Has Little Benefit
Suppose, hypothetically, that the better treatment has success probability \(0.60\) and the worse treatment \(0.40\). A rule uses delayed feedback to select one arm for a group of 20 incoming participants. Assume the rule identifies the better arm with probability \(q=0.55\), and otherwise selects the worse arm. For comparison, an equal random allocation assigns half to each arm.
For the delayed rule, the expected success probability for each participant is:
The 20 assignments therefore have \(20(0.51)=10.2\) expected successes. Under equal allocation the expected probability is \(0.50\), giving \(20(0.50)=10.0\) expected successes. The delayed ranking's expected gain is only 0.2 successes over this group.
If the rule instead had \(q=0.80\), its expected success probability would be:
That corresponds to 11.2 expected successes, or 1.2 more than equal allocation. The difference illustrates why information timeliness matters: a rule that is only slightly more likely than chance to identify the better arm may produce little participant benefit, even if a highly accurate ranking would produce more.
The calculation assumes the stated \(q\) applies to the whole group, both outcomes are correctly represented by the assumed probabilities, and there are no other differences among participants. It does not establish a practical value for \(q\) in a real trial; that would require evidence and assumptions about the outcome and delay process.
Resolved Outcomes and a Precision Check
The standard-error formula can help compare the information available from arm counts, but delayed resolution calls for a qualified interpretation. If the expected number of resolved outcomes is used in place of the actual count, the result is an approximate planning calculation, not the realized standard error. It also does not measure the calendar time needed to reach those counts.
Worked Example: Approximate Precision With Outcomes Pending
Imagine a hypothetical interim assessment after 28 participants have been assigned to arm 1 and 24 to arm 0. Suppose 80% of outcomes in each arm are expected to be resolved by that assessment, with resolution unrelated to the outcome within each arm. Assume the continuous outcome has common standard deviation \(\sigma=5\) points. Estimate the standard error using expected resolved counts.
The expected resolved counts are \(28(0.80)=22.4\) in arm 1 and \(24(0.80)=19.2\) in arm 0. Using these expected counts as a rough approximation:
For comparison, if the same 52 assignments had been split equally, with 26 per arm and the same 80% resolution rate, the expected resolved count would be \(26(0.80)=20.8\) per arm. The corresponding approximation is:
The approximate standard errors are close, with the unequal allocation slightly larger. This small difference should not be overinterpreted: the calculations use expected, potentially fractional resolved counts and assume equal resolution rates and common variance. Actual resolved counts, unequal delay patterns, or selective resolution can change the comparison. The calculation also says nothing about whether either assessment occurs soon enough to support a timely decision.
Diagnostics, Alternatives, and Limitations
A practical assessment should track participant outcomes and decision information separately. Useful checks include the allocation proportion over time, the number of assigned and resolved outcomes in each arm, the age of unresolved outcomes, and the interval between enrollment and outcome resolution. These summaries help show whether the trial is receiving information at the rate assumed by its allocation plan.
Warning signs include a widening imbalance in resolved counts, allocation probabilities that continue to favor an arm despite little recent outcome information, and a growing backlog that delays comparisons. If resolution differs by treatment arm, a pooled resolution rate can conceal an important imbalance. If outcomes resolve selectively within an arm, the resolved observations may not represent the full set of assigned participants. In those cases, an expected-count standard error is not an adequate diagnostic of comparison quality.
The main alternatives answer different priorities:
| Approach | Potential advantage | Trade-off under delays |
|---|---|---|
| Equal allocation | Balances assignment counts and is a useful precision benchmark for a two-arm mean comparison. | Does not direct more participants to a treatment believed to be better; unresolved outcomes may still delay comparisons. |
| Unequal or adaptive allocation | Can increase expected participant benefit if the favored treatment is genuinely better. | Can reduce comparison precision, and delayed information can make the favored ranking less reliable. |
| Wait for more outcomes before changing allocation | Allows a decision to use more resolved information. | May postpone allocation changes or decisions; waiting does not itself ensure the data are unbiased or sufficient. |
| Use predictions for pending outcomes | May permit decisions before outcomes fully resolve. | Decisions depend on the prediction model and its assumptions, rather than on observed outcomes alone. |
No row is universally preferable. Equal allocation is not always optimal for every scientific or participant-benefit objective, and adaptive allocation does not automatically improve participant outcomes. A comparison's standard error is only one aspect of decision quality; bias, relevance of the estimand, and the time at which evidence is available also matter.
The key limitation of the numerical illustrations is that they treat treatment outcomes and, where needed, the accuracy of the treatment ranking as given. In an actual response-adaptive trial, these quantities are uncertain and may change as data accumulate. Delay can make the observed information lag behind the trial's current state. Assessment should therefore distinguish what is known from what is predicted, and should not treat pending outcomes as if they had already been observed.
Common Mistake
Wrong reasoning: “The adaptive rule assigns more participants to the treatment with the higher observed response, so it must improve both patient benefit and the treatment comparison.”
Correction: A higher observed response does not guarantee that the treatment is truly better, especially when feedback is delayed and the estimate is based on limited resolved data. Even when the favored arm is genuinely better, a more imbalanced allocation can reduce precision for the between-arm comparison. Evaluate expected participant outcomes, comparison quality, and timeliness as separate outcomes, under clearly stated assumptions.
Check Your Understanding
Work through each question, then check the solutions below.
- A rule selects one treatment for a group. The better treatment has success probability 0.42, the worse has 0.30, and the rule chooses the better treatment with probability 0.65. What is the expected success probability per participant? How does it compare with equal allocation?
- With common standard deviation 3 points, compare the approximate standard errors for \(n_1=n_0=40\) and for \(n_1=60,\ n_0=20\). State which allocation is more precise under the stated model.
- There are 30 assignments to arm 1 and 20 to arm 0. If 70% and 90% of outcomes, respectively, are expected to be resolved, what is the approximate standard error for a mean difference when \(\sigma=4\) points?
- Why can an expected-count standard error fail to describe comparison quality at a particular interim date?
Solutions. (1) The expected success probability is \(0.65(0.42)+0.35(0.30)=0.273+0.105=0.378\). Equal allocation gives \(0.50(0.42)+0.50(0.30)=0.36\), so the stated rule is higher by 0.018, or 1.8 percentage points per participant under these assumptions. (2) Equal allocation gives \(3\sqrt{1/40+1/40}=3\sqrt{0.05}\approx0.671\) points. The 60:20 allocation gives \(3\sqrt{1/60+1/20}=3\sqrt{0.0666667}\approx0.775\) points. Equal allocation is more precise under the common-variance independent-outcome model. (3) Expected resolved counts are \(30(0.70)=21\) and \(20(0.90)=18\). Thus \(SE\approx4\sqrt{1/21+1/18}=4\sqrt{0.103175}\approx1.285\) points. This is an approximation based on expected counts. (4) Actual resolved counts may differ from their expectations, and unequal or outcome-related resolution can make resolved observations unrepresentative. The calculation also does not indicate when the required outcomes will resolve.