Why Vary the Assumed Treatment Effect?
Power and patient-benefit comparisons are not properties of a randomization rule alone. They depend on the treatment effect used to evaluate the design. A design that appears attractive under one assumed effect may look different when the effect is smaller, larger, or close to zero. Sensitivity analysis makes this dependence visible rather than hiding it inside a single planning value.
The aim is not to guess the true effect by trying several numbers. It is to ask whether the design comparison is stable across a defensible range of effects. This matters especially when an adaptive design is being compared with fixed randomization: allocation may change both how many patients receive each treatment and the precision of the final comparison. The earlier material on benefit-power trade-off curves provides a way to display those competing outcomes; here, the emphasis is on recalculating them as the assumed treatment effect changes.
The calculations below use a deliberately simple model to show the mechanics. They are a screening approximation, not a substitute for evaluating the full operating characteristics of an outcome-dependent randomization rule.
Model and Sensitivity Calculations
For the worked calculations, assume a two-arm trial with a fixed total sample size \(N=200\), independent outcomes, normally distributed outcomes with a common known standard deviation \(\sigma=1\) outcome unit, no missing outcomes, and a two-sided z test at \(\alpha=0.05\). Suppose \(Nq\) patients are assigned to treatment 1 and \(N(1-q)\) to control, where \(q\) is the treatment allocation fraction. For the allocation-summary calculations, these expected counts may be fractional.
Under this fixed-allocation model, the estimated difference in sample means has standard error \(\sigma\sqrt{1/n_1+1/n_0}\). Substituting \(n_1=Nq\) and \(n_0=N(1-q)\) gives the standard error used in the power calculation:
The test rejects when the absolute z statistic exceeds \(z_{1-\alpha/2}\), which is approximately 1.96 here. Under a particular effect \(\delta\), the test statistic has a normal distribution with mean \(\lambda=\delta/SE\) and variance 1. Thus the two-sided power is
where \(Z\sim N(\lambda,1)\) under the alternative and \(\Phi\) is the standard normal cumulative distribution function. This expression shows directly why the assumed effect matters: changing \(\delta\) changes the noncentrality parameter and therefore changes power, even if sample size and allocation are held fixed.
For a response-adaptive design, \(q\) is generally random and depends on trial outcomes. Plugging an expected allocation fraction into this formula is an approximation: an average allocation does not, in general, capture the full distribution of allocations or its dependence on outcomes. In practice, the full adaptive rule should be evaluated under each assumed effect, usually by simulation. The examples use the formula as a transparent allocation-summary calculation and state its limitation explicitly.
Worked Example: Recalculating Power at One Assumed Effect
Suppose the fixed design allocates equally, \(q=0.50\). For an illustrative adaptive policy, assume an operating-characteristic summary gives an expected treatment fraction \(q=0.679\) when \(\delta=0.50\). Calculate and compare the approximate power under the model above.
For fixed randomization, \(Nq(1-q)=200(0.50)(0.50)=50\), so \(\lambda=0.50\sqrt{50}=3.536\). Therefore,
For the allocation-summary calculation, \(Nq(1-q)=200(0.679)(0.321)=43.5918\), giving \(\lambda=0.50\sqrt{43.5918}=3.301\). Hence,
At this assumed effect, the displayed power difference is \(0.942-0.910=0.032\): the approximation gives about 3.2 percentage points less power for the adaptive allocation summary. This is not a universal penalty for adaptive randomization. It follows from the particular allocation fraction and assumptions in this example. A full adaptive-design assessment could differ because the allocation fraction varies across trial paths.
Expected Patient Benefit and the Same Scenarios
Power concerns the chance of statistical success; expected patient benefit concerns outcomes received by participants. Under the homogeneous-effect assumption in this example, each treated patient has mean outcome \(\mu_0+\delta\), and each control patient has mean outcome \(\mu_0\). If the expected treatment fraction is \(q\), the mean outcome across \(N\) trial participants is \(\mu_0+q\delta\). Relative to assigning everyone to control, the expected total gain is therefore \(Nq\delta\) outcome units. Relative to 50/50 fixed allocation, the difference is \(N(q-0.50)\delta\).
This calculation assumes a common treatment effect for all participants and that the outcome scale has a meaningful direction. It describes an expected difference in outcomes, not a guarantee for any individual and not an estimate of benefit for a wider population unless the trial participants and target population justify that interpretation.
Worked Example: Calculating the Benefit Difference
Continue with \(N=200\), \(\delta=0.50\) outcome units, and the expected adaptive treatment fraction \(q=0.679\). Compare the expected total benefit with 50/50 allocation, assuming higher outcomes are better and the same treatment effect applies to every participant.
The adaptive allocation summary gives an expected total gain over assigning everyone to control of \(Nq\delta=200(0.679)(0.50)=67.9\) outcome units. The 50/50 design gives \(200(0.50)(0.50)=50.0\) outcome units. The difference is \(67.9-50.0=17.9\) outcome units, equivalently \(200(0.679-0.50)(0.50)=17.9\).
Thus, under these assumptions, the adaptive allocation summary yields greater expected participant benefit than 50/50 allocation, while the previous example showed lower approximate power. The two results can coexist: allocating more participants to the beneficial treatment can improve the trial's average outcome while reducing the balance that supports precision in a treatment comparison.
Reading a Range of Effects
A sensitivity analysis should show more than the result at a single planning value. The following illustrative calculations vary \(\delta\) and use a scenario-specific expected treatment fraction for the allocation-summary approximation. The values for \(q\) are hypothetical operating summaries for one candidate adaptive policy; they are not asserted to follow from a particular randomization algorithm. Power values are rounded to three decimals, and differences are calculated from the displayed values.
| Assumed \(\delta\) (outcome units) | Expected adaptive \(q\) | Fixed power, \(q=0.50\) | Adaptive-summary power | Adaptive minus fixed power | Benefit difference, total outcome units |
|---|---|---|---|---|---|
| 0.25 | 0.550 | 0.424 | 0.420 | -0.004 | 2.50 |
| 0.50 | 0.679 | 0.942 | 0.910 | -0.032 | 17.9 |
| 0.75 | 0.750 | 0.9996 | 0.9958 | -0.0038 | 37.5 |
For example, at \(\delta=0.25\), the benefit difference is \(200(0.550-0.50)(0.25)=2.50\) outcome units. At \(\delta=0.75\), it is \(200(0.750-0.50)(0.75)=37.5\) outcome units. The fixed-allocation power in the final row is close to its maximum of 1, so a difference in allocation has little room to create a large absolute power difference. This is a ceiling effect, not evidence that the allocations have identical precision.
Worked Example: Identifying an Effect-Dependent Conclusion
Use the table to assess the statement: “The adaptive allocation has a major power cost, but its benefit advantage is negligible.”
At \(\delta=0.25\), the power difference is only \(-0.004\), or four-tenths of a percentage point, and the benefit difference is 2.50 total outcome units. At \(\delta=0.50\), the power difference is \(-0.032\), or 3.2 percentage points, while the benefit difference is 17.9 units. At \(\delta=0.75\), the power difference is \(-0.0038\), but the benefit difference is 37.5 units.
The quoted statement is therefore not stable across the evaluated effects. The power cost is appreciable only in the middle scenario in this table, and the benefit advantage is not negligible at the largest effect under the chosen outcome scale. Neither conclusion should be generalized beyond the scenarios and assumptions shown. In particular, the table does not establish that an actual response-adaptive design will have these power values; its allocation-summary calculations are approximate.
How to Conduct and Diagnose the Sensitivity Analysis
Specify a central planning value and smaller and larger values that are clinically or scientifically plausible. Include zero when useful for checking behavior under the null.
Keep sample size, outcome scale, significance level, analysis rule, and allocation policy the same across effect scenarios. This isolates sensitivity to \(\delta\).
For each effect, calculate power and the chosen patient-benefit measure. State units, the comparison baseline, and whether results are exact, approximated, or estimated by simulation.
Identify effect ranges where rankings, practical differences, or trade-offs change. Report ranges and assumptions rather than selecting only the most favorable scenario.
Several diagnostics help reveal an unreliable analysis. First, under the stated two-sided test, power should approach \(\alpha\) as \(\delta\) approaches zero; a marked discrepancy suggests a formula, coding, or calibration error. Second, power should generally rise as the magnitude of a fixed-direction effect increases, although rounding and Monte Carlo error can obscure small changes. Third, check that \(q\) stays strictly between 0 and 1 and that allocation counts sum to \(N\). Fourth, if a simulation is used for an adaptive rule, report the number of simulated trials and Monte Carlo uncertainty; small apparent differences may be simulation noise. Finally, verify benefit arithmetic independently and confirm the outcome's direction: if higher values are worse, the sign interpretation must be reversed.
Alternatives, Limitations, and a Common Pitfall
The simple plug-in calculation is useful for showing how allocation balance and effect size interact under a normal model. It is not the only approach. A fixed-allocation analytic calculation is appropriate when allocation counts are fixed and the model assumptions apply. For a genuinely response-adaptive rule, simulation can reproduce the sequential allocation and outcome process under each assumed effect, then estimate power as the proportion of simulated trials that reject \(H_0\). The same simulated paths can estimate expected allocation fractions and patient-benefit measures. These approaches answer related questions, but the analytic approximation is not a replacement for simulating an outcome-dependent design.
The analysis is also conditional on the specified outcome model, variance, treatment effect structure, sample size, and test. It does not address whether those assumptions are correct, whether effects differ among participants, or whether treatment and control variances differ. Those issues can change both power and benefit and require separate evaluation. Nor should a single assumed effect be treated as known merely because it was used for planning.
A further check is the null scenario. For a correctly calibrated test at the stated level, power when \(\delta=0\) is its Type I error rate, approximately \(\alpha=0.05\) here. If a simulated adaptive design has a substantially different rejection probability under the null, the issue is calibration or implementation, not sensitivity to a beneficial treatment effect. Keeping this check distinct helps prevent a power comparison from masking an invalid test.
Check Your Understanding
Answer each question using the assumptions and notation stated above. Brief solutions follow the questions.
- With \(N=200\), \(\sigma=1\), and \(q=0.50\), calculate the noncentrality parameter when \(\delta=0.25\). Why is power lower than at \(\delta=0.50\)?
- For \(N=200\), \(q=0.60\), and \(\delta=0.40\), calculate the expected total benefit relative to assigning everyone to control, and the difference relative to 50/50 allocation.
- If two designs have similar power at a large assumed effect, why might that fail to show they have similar power at a smaller effect?
- State one reason the average treatment fraction alone may not give exact power for a response-adaptive randomization rule.
- What should the rejection probability be at \(\delta=0\) for a test correctly calibrated at \(\alpha=0.05\)?
Solutions. (1) \(\lambda=0.25\sqrt{200(0.50)(0.50)}=0.25\sqrt{50}=1.768\). The smaller effect produces a smaller noncentrality parameter, so the sampling distribution under the alternative is less separated from the null distribution and power is lower. (2) Relative to all control, expected total benefit is \(Nq\delta=200(0.60)(0.40)=48.0\) outcome units. Relative to 50/50 allocation, the difference is \(200(0.60-0.50)(0.40)=8.00\) outcome units. (3) At a large effect, both powers may be near 1, leaving little room to see a difference; at a smaller effect, changes in allocation and standard error may produce a more noticeable power difference. (4) The allocation fraction varies across trial paths and is dependent on observed outcomes, so the distribution of the final test statistic is not determined by the average fraction alone. (5) The rejection probability should be approximately \(\alpha=0.05\), subject to the test's calibration and any simulation error.