Two Objectives, Not One
A response-adaptive allocation rule changes treatment assignment probabilities as outcome information accumulates. It may assign more participants to a treatment that appears to produce better outcomes, with the aim of improving the outcomes experienced during the trial. But a trial also aims to estimate a treatment effect precisely. These aims are related, but they are not identical: an allocation can increase expected participant outcomes while making the treatment comparison less precise.
Patient benefit concerns the outcomes experienced by participants under an allocation policy. For a binary outcome coded as success or failure, one simple measure is the expected number of successes among enrolled participants. Inferential efficiency concerns the precision with which the trial estimates a specified treatment contrast. It can be described using the estimator’s sampling variance or standard error, with smaller values indicating greater precision when the estimand and analysis are held fixed. Power is related to precision, but it also depends on the effect being tested, the test, and the significance level.
The distinction matters when a design is judged. Expected successes count outcomes across participants, whereas an estimate of a treatment contrast uses outcomes in both treatment groups to compare them. A policy can shift participants toward a treatment with a higher response probability and thereby raise the expected total number of successes. At the same time, having fewer participants in one group can increase uncertainty in the contrast estimate.
Model and Assumptions for a Simple Comparison
For the calculations below, consider two treatments, A and B, and a binary outcome \(Y\), where \(Y=1\) denotes success. Let \(p_A\) and \(p_B\) be the success probabilities under A and B. The treatment-effect estimand is the risk difference \(\Delta=p_A-p_B\). Assume outcomes are measured consistently, participants’ outcomes do not affect one another, and each participant’s outcome under an assigned treatment follows the corresponding Bernoulli distribution. These assumptions make the expected-success calculations meaningful; further assumptions may be needed for a particular trial or clinical outcome.
If the numbers assigned to the two treatments are fixed at \(n_A\) and \(n_B\), and outcomes are independent within and between groups, the difference in observed success proportions is \(\hat{p}_A-\hat{p}_B\). Its variance is
To justify this expression, the sample proportion in group A is an average of \(n_A\) independent Bernoulli outcomes, each with variance \(p_A(1-p_A)\). Therefore its variance is \(p_A(1-p_A)/n_A\). The corresponding variance for group B is \(p_B(1-p_B)/n_B\). Independence between the groups means the variance of their difference is the sum of these two variances. Taking the square root gives the standard error.
This formula is a fixed-count benchmark, not automatically an exact standard error for a response-adaptive trial. Under adaptive assignment, group sizes can be random and depend on earlier outcomes; that dependence must be accounted for in the inferential analysis. The formula is useful for understanding how allocation counts can affect precision, but applying it mechanically to an adaptive trial can give an incomplete account of uncertainty.
Worked Example: More Expected Successes, Less Precision
Suppose a hypothetical planning calculation uses \(p_A=0.60\) and \(p_B=0.40\) for 100 participants. Compare equal allocation, 50 to each treatment, with an allocation of 70 to A and 30 to B. Treat the probabilities as known for this illustration; they are not estimates from a real study.
Expected participant outcomes. Under equal allocation, the expected number of successes is
With 70 participants assigned to A and 30 to B, the expected number is
Thus this allocation has four more expected successes among 100 participants under the stated probabilities.
Precision benchmark. For equal allocation, the fixed-count variance of the difference in proportions is
so the standard error is \(\sqrt{0.0096}=0.0980\). With 70 participants on A and 30 on B, the corresponding calculations are
giving a standard error of \(\sqrt{0.01143}=0.1069\). In this fixed-count benchmark, the allocation with more expected successes has a larger standard error for the treatment contrast. This is an illustration of the trade-off, not an exact analysis of a response-adaptive trial: the calculation does not include the dependence created when allocation counts respond to trial outcomes.
Evaluating the Objectives Separately
A sound design comparison states what is being optimized and how each objective will be measured. For a binary outcome, expected successes under a specified allocation can be calculated by summing the expected successes in each treatment group. For inferential efficiency, specify the treatment-effect estimand and use a variance or standard error appropriate to the design and analysis. Comparing a total success count under one policy with a standard error under another is not meaningful: the quantities answer different questions.
The distinction also helps when outcomes appear promising early in an adaptive trial. Early observed responses may inform subsequent allocation, but they do not establish the true treatment success probabilities. A policy that reacts strongly to noisy early results can send many later participants toward a treatment that only appeared superior by chance. Patient outcomes and precision should therefore be evaluated across the full allocation-and-analysis procedure, not inferred from one observed allocation path.
Worked Example: Auditing a Design Comparison
A hypothetical planning report compares two policies for a trial of 200 participants. Repeated simulated trials under the specified outcome and allocation model give an average of 116 successes under policy F and 124 under policy R. The reported standard errors for the estimated treatment contrast, calculated using each policy’s intended analysis, are 0.090 for F and 0.110 for R.
Patient-outcome comparison. Policy R has \(124-116=8\) more average successes across the 200 participants in these simulations. This is an average difference under the simulated assumptions, not a guarantee of eight additional successes in an individual trial.
Precision comparison. Policy F has the smaller reported standard error: \(0.090<0.110\). For a rough scale comparison, multiplying each standard error by 1.96 gives half-widths of \(1.96(0.090)=0.176\) and \(1.96(0.110)=0.216\). These are normal-approximation half-widths, not guaranteed interval widths or a substitute for checking the actual interval method.
The results therefore point in opposite directions: R produces more average successes in the simulation, while F gives a more precise treatment-effect estimate according to the reported standard errors. A decision-maker must weigh these distinct objectives and verify that both were evaluated under comparable assumptions. The average success counts alone do not establish which policy is preferable overall.
Worked Example: Choosing the Right Question
A hypothetical trial team is considering a response-adaptive policy. In a planning analysis, the team estimates that the policy would raise the expected success rate among enrolled participants from 0.52 to 0.55. The design analysis also predicts standard errors of 0.12 for the treatment contrast under the adaptive policy and 0.10 under a fixed policy.
For 160 participants, the expected success-count difference is
This means the adaptive policy is projected to yield 4.8 more successes on average under the planning model; it does not mean a particular trial can have a fractional number of successes. The standard-error comparison is \(0.12>0.10\), so the adaptive policy is projected to be less precise for the treatment contrast in that analysis. These figures do not by themselves determine power or clinical desirability. The team should ask separately whether the projected participant-outcome improvement is important and whether the loss of precision is acceptable for its inferential goal.
Diagnostics, Alternatives, and Limitations
A design comparison is only as credible as its assumptions and calculations. Check that the policies are compared using the same total sample size, outcome definition, follow-up period, treatment contrast, and underlying response model. If one policy is evaluated with known response probabilities and another with estimated probabilities, the comparison may reflect different information rather than a design effect. For adaptive policies, inspect the distribution of allocation counts across repeated trials, not only their average: highly variable counts can produce unstable estimates or very small groups.
Also check that the inferential standard error corresponds to the allocation procedure actually used. If later assignments depend on earlier outcomes, a fixed-count formula may not capture all relevant uncertainty. In a design evaluation, simulate the complete allocation and analysis procedure under stated assumptions, or use an inferential method justified for that adaptive design. When possible, compare both objectives across a range of plausible response probabilities, since a policy’s patient-outcome advantage can depend on which treatment truly performs better.
| Evaluation approach | What it answers | Main caution |
|---|---|---|
| Expected participant outcomes | How many successes, or what average outcome, participants may experience under a policy | Depends on the assumed outcome model and does not measure precision |
| Estimator variance or standard error | How precisely the trial estimates its stated treatment contrast | Must account for the design and the estimator used |
| Power or probability of a decision | How often a specified test or decision rule succeeds under stated conditions | Depends on the effect, test, significance level, and design; it is not the same as patient benefit |
The main alternative to optimizing participant outcomes is not one single design rule. A trial may prioritize estimation, treatment selection, safety, or a combination of aims; fixed allocation provides a benchmark, while response-adaptive allocation changes assignments using accrued outcomes. A fair comparison makes the objective explicit and assesses each candidate against the same benchmark. No single metric captures all ethical and scientific considerations, and simple calculations cannot replace a complete design analysis.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- In a trial comparison, one allocation policy has a higher expected number of successes but a larger standard error for the treatment contrast. Which policy has greater patient benefit and which has greater inferential efficiency, according to those measures?
- For fixed group sizes \(n_A=40\) and \(n_B=60\), with hypothetical success probabilities \(p_A=0.50\) and \(p_B=0.40\), calculate the fixed-count variance and standard error of the difference in proportions.
- Why might the fixed-count variance formula be inadequate when group sizes are determined by earlier trial outcomes?
- A report gives only the average number of successes under two allocation policies. What can and cannot be concluded about inferential efficiency?
Solutions. (1) The policy with more expected successes has greater patient benefit by that measure; the policy with the smaller standard error has greater inferential efficiency. (2) The variance is \(0.50(0.50)/40+0.40(0.60)/60=0.00625+0.00400=0.01025\), and the standard error is \(\sqrt{0.01025}=0.1012\). This calculation assumes fixed counts and independent Bernoulli outcomes. (3) Adaptive group sizes can be random and dependent on earlier responses, so the fixed-count calculation may omit design-induced uncertainty or dependence. The analysis must reflect the allocation procedure. (4) The report permits a comparison of expected participant outcomes under its assumptions, but it does not show which policy estimates the treatment contrast more precisely; a suitable variance or standard error is also needed.