Why Power Must Be Read Across Effect Sizes
A single power value answers a narrow question: how often would a specified analysis reject the null hypothesis if one particular treatment effect were true? It does not describe performance for every plausible treatment effect. In a response-adaptive randomized trial, this distinction matters because the same design can have low power for a small effect and high power for a larger effect, while random allocation histories can add another source of variation.
The goal is to read these two features separately. Low power near the null is expected when the true effect produces little signal relative to outcome noise. By contrast, a design-specific allocation problem is suggested when the adaptive procedure often produces imbalanced or highly variable allocation counts, leading to less precise treatment contrasts or unstable operating characteristics. A power curve across effects helps reveal both patterns, but it does not by itself explain their causes.
In an adaptive trial, the probability in this definition includes the randomness in patient outcomes and in the response-adaptive assignments. It is therefore an unconditional operating characteristic, averaged over the allocation paths the design can generate. Power conditional on one allocation path can help explain performance, but it is not the trial’s overall power. This distinction connects to the earlier material on conditional and unconditional performance.
A Normal-Approximation Power Curve
To make the calculation explicit, suppose outcomes are independent across patients, treatment assignment is randomized according to the trial’s specified mechanism, and outcomes in each arm follow normal distributions with a common standard deviation \(\sigma\). Assume the analysis uses a two-sided test of \(H_0:\Delta=0\), calibrated to have Type I error \(\alpha\), and that its standardized treatment contrast is approximately normal with mean \(\Delta/SE\) and standard deviation 1 under the alternative. For fixed arm sizes \(n_1\) and \(n_0\), the usual standard error under this model is \(\sigma\sqrt{1/n_1+1/n_0}\).
The approximation requires more than normal outcomes: in an adaptive trial, the test must also be valid for the actual assignment mechanism. If adaptation makes the standardized statistic’s distribution differ materially from the assumed normal distribution, the formula below is not a substitute for a properly calibrated analysis. It is a useful calculation when its assumptions are reasonable, and a benchmark for understanding how effect size and allocation precision interact.
The expression follows by standardizing the test statistic under the alternative. It has distribution \(N(\lambda,1)\), and the two-sided test rejects when the statistic is below \(-z_{1-\alpha/2}\) or above \(z_{1-\alpha/2}\). Adding the probabilities of these disjoint rejection regions gives the formula. When \(\Delta=0\), \(\lambda=0\), and the two tails sum to \(\alpha\); away from zero, the probability of rejection generally rises as the magnitude of the standardized effect increases under this location-shift model.
Worked Example: A Fixed-Allocation Power Curve
Consider a hypothetical trial with 100 patients, 50 assigned to each arm, a common outcome standard deviation of 10 points, and a two-sided test with \(\alpha=0.05\). Assume the normal approximation above is valid. The standard error is \[ SE=10\sqrt{\frac{1}{50}+\frac{1}{50}}=10\sqrt{0.04}=2.00\text{ points}. \] The critical value is approximately 1.96.
For a true treatment effect of 2 points, \(\lambda=2/2=1\). Thus \[ \pi(2)=\Phi(-1.96-1)+1-\Phi(1.96-1) =\Phi(-2.96)+1-\Phi(0.96) \approx 0.0015+0.1685=0.170. \] Power is about 17.0%, so the Type II error probability at this effect is \(1-0.170=0.830\), or 83.0%.
For a true effect of 4 points, \(\lambda=4/2=2\): \[ \pi(4)=\Phi(-3.96)+1-\Phi(-0.04)\approx 0.00004+0.5160=0.516. \] The corresponding Type II error probability is about 48.4%.
For a true effect of 6 points, \(\lambda=6/2=3\): \[ \pi(6)=\Phi(-4.96)+1-\Phi(-1.04)\approx 0+0.8508=0.851. \] The Type II error probability is about 14.9%. All effects here are in points. This curve shows low power for a 2-point effect because it is small relative to the standard error, and much higher power for a 6-point effect. It does not, by itself, show a failure of randomization.
How Allocation Variability Enters the Curve
In a response-adaptive trial, arm sizes may be random. Denote the final numbers assigned by \(N_1\) and \(N_0\), with \(N_1+N_0=n\). Even when outcomes have the same variance in both arms, the standard error depends on the realized counts: \[ SE(N_1,N_0)=\sigma\sqrt{\frac{1}{N_1}+\frac{1}{N_0}}. \] An allocation away from balance generally increases this standard error. But power for the adaptive design is not obtained by silently replacing the random counts with their average. Under a valid conditional approximation, it is the average of conditional rejection probabilities over the allocation paths.
Worked Example: Averaging Over Hypothetical Allocation Patterns
Continue with the hypothetical model \(\sigma=10\), \(\alpha=0.05\), and a true effect of 4 points. To isolate allocation variability, suppose a design produces allocations of 50/50, 70/30, or 85/15 with probabilities 0.25, 0.50, and 0.25, respectively. Assume, for this illustration, that the conditional normal approximation is valid for each allocation pattern. These are specified hypothetical probabilities, not empirical findings.
With 50/50, \(SE=2.00\), so the power from the previous calculation is approximately 0.516. With 70/30, \[ SE=10\sqrt{\frac{1}{70}+\frac{1}{30}} =10\sqrt{0.04762}\approx 2.182, \qquad \lambda=\frac{4}{2.182}\approx1.833. \] The resulting power is \[ \Phi(-1.96-1.833)+1-\Phi(1.96-1.833) \approx \Phi(-3.793)+1-\Phi(0.127)\approx 0.450. \] With 85/15, \[ SE=10\sqrt{\frac{1}{85}+\frac{1}{15}} =10\sqrt{0.07843}\approx2.801, \qquad \lambda=\frac{4}{2.801}\approx1.428, \] giving power \[ \Phi(-3.388)+1-\Phi(0.532)\approx0.298. \] Therefore the unconditional power under these assumed allocation probabilities is approximately \[ 0.25(0.516)+0.50(0.450)+0.25(0.298)=0.428. \]
This calculation illustrates how allocation variability can reduce average power, even at the same treatment effect. It does not establish that every adaptive design has this loss: a real design’s allocation probabilities and conditional test distributions must be evaluated under its own randomization and analysis procedures. In particular, adaptation based on observed outcomes can make a conditional normal approximation inadequate.
A useful diagnostic is to compare power at a fixed effect under the adaptive design with power under a clearly specified reference design, using the same outcome model, sample size, hypothesis test, and Type I error calibration. Also examine the distribution of \(N_1\), \(N_0\), and conditional power across simulated trials, not just the average counts. A wide spread in conditional power can reveal allocation-path sensitivity that a single average conceals. These comparisons concern operating characteristics; they should not be interpreted as evidence that a treatment causes better patient outcomes merely because the design allocates more patients to it.
Estimating a Power Curve by Simulation
Simulation is often needed when the adaptive assignment rule and analysis do not yield a trustworthy closed-form power calculation. For each chosen effect size, generate many hypothetical trials under a fully specified outcome model, run the complete response-adaptive randomization procedure, apply the planned test, and record whether it rejects \(H_0\). The estimated power is the proportion of simulated trials that reject. To estimate Type I error, simulate at \(\Delta=0\); to estimate Type II error at a nonzero \(\Delta\), use one minus the estimated power at that effect.
Worked Example: Reading Simulated Power Estimates
Suppose a hypothetical simulation runs 2,000 trials at each effect size. The numbers of rejections are 101 at \(\Delta=0\), 340 at \(\Delta=2\), 1,032 at \(\Delta=4\), and 1,702 at \(\Delta=6\), with effects measured in points. Each simulated trial uses the full adaptive assignment procedure and the same prespecified test.
The estimated rejection proportions are \(101/2000=0.0505\), \(340/2000=0.170\), \(1032/2000=0.516\), and \(1702/2000=0.851\). The estimate at \(\Delta=0\) is an estimated Type I error rate. At \(\Delta=2\), estimated power is 17.0% and estimated Type II error is \(1-0.170=0.830\). At \(\Delta=4\), estimated power is 51.6%; at \(\Delta=6\), it is 85.1%.
The Monte Carlo standard error of the estimated power at \(\Delta=4\) is approximately \[ SE_{\mathrm{MC}}=\sqrt{\frac{0.516(1-0.516)}{2000}} =\sqrt{0.0001249}\approx0.0112. \] Thus simulation noise alone is about 1.12 percentage points in standard-error units; an approximate 95% Monte Carlo margin is \(1.96(0.0112)\approx0.022\), or 2.2 percentage points. This uncertainty describes the finite number of simulated trials, not uncertainty in the treatment effect or variation among patients in an actual trial.
Separating Weak Signal From Design-Specific Problems
A power curve that is low near \(\Delta=0\) and increases steadily as \(|\Delta|\) grows may simply reflect limited information for distinguishing small effects from the null. To assess whether the adaptive design adds a separate performance concern, compare it with a suitable nonadaptive reference and inspect allocation behavior across simulated trials. A sharp difference between designs at the same effect size, or high variability in power across allocation paths, points to design-specific allocation variability rather than only a weak signal.
The comparison must hold the analysis and error-rate target constant. A design that appears more powerful only because its test rejects more often under \(H_0\) has not demonstrated a genuine power advantage at the intended Type I error rate. Likewise, comparing conditional power for a favorable allocation path with unconditional power from another design is not an even comparison. The introductory material on power under response-adaptive randomization provides the basic operating-characteristic context; here the essential extension is to interpret performance over a range of effects and allocation paths.
| Pattern in results | Likely interpretation | What to check |
|---|---|---|
| Low power for effects close to zero; higher power as effects grow | Weak signal relative to outcome variability and sample size | Whether the effects are scientifically important and the curve is correctly calibrated |
| Lower average power than a matched reference design at the same effect | Possible loss of precision or another design-specific difference | Type I error, allocation counts, analysis, and simulation assumptions |
| Large spread in conditional power across allocation paths | Performance may depend strongly on random allocation history | The full distribution of arm sizes and rejection rates, not only mean arm sizes |
| Jagged or implausibly irregular simulated curve | Monte Carlo error, coding problems, or an invalid approximation may be present | More simulation replicates, implementation checks, and test calibration under the null |
Common Mistake and Limitations
Common mistake: “Power is low at the smallest effect, so the adaptive allocation rule must be performing badly.” This conclusion does not follow. Small effects are difficult to detect in many designs. The correction is to compare designs at the same effect sizes and error-rate target, then investigate allocation distributions and conditional performance to determine whether adaptation adds variability or reduces precision.
Another mistake is to calculate power using the average arm sizes and treat the result as the adaptive design’s unconditional power. Rejection probability is generally nonlinear in the arm sizes and standard error, so the average of conditional powers need not equal power evaluated at the average counts. Moreover, for outcome-driven allocation, allocation counts can be associated with observed outcomes. A calculation based only on realized counts may then fail to represent the full adaptive experiment. Simulation can address this only if it faithfully reproduces the randomization, outcomes, stopping rules, and planned analysis.
Power curves also inherit the assumptions used to create them. A curve based on a common outcome variance or a particular effect model may not apply when those assumptions fail. The curve describes rejection probabilities under specified conditions; it does not guarantee the result of a future trial, identify the true effect, or quantify patient benefit by itself. Report the model, test, significance level, effect sizes, and whether probabilities are conditional or unconditional whenever presenting a curve.
Check Your Understanding
Answer each question, then compare with the solutions.
- If power at a nonzero effect is 0.30, what is the Type II error probability at that effect? Why is this not a statement about the probability that \(H_0\) is true?
- In the normal approximation with \(\alpha=0.05\), \(SE=2\) points, and \(\Delta=4\) points, calculate \(\lambda\) and state the approximate power.
- Why can power averaged over random allocation patterns differ from power calculated at the average arm sizes?
- A simulated curve has lower power than a reference design at a nonzero effect, but also a higher rejection rate at \(\Delta=0\). What should be checked before calling it a power advantage or disadvantage?
Solutions. (1) The Type II error probability is \(1-0.30=0.70\). It is the probability of failing to reject under the specified effect, not the probability that the null hypothesis is true. (2) \(\lambda=\Delta/SE=4/2=2\); the two-sided normal-approximation power is approximately 0.516, or 51.6%. (3) Conditional rejection probabilities vary with the allocation counts, and averaging a nonlinear function of those counts does not generally equal evaluating it at average counts. (4) Check whether both tests are calibrated to the same Type I error target under the actual designs; a higher null rejection rate can make apparent power differences misleading.