Diagnosing Apparent Power Gains In Adaptive Trials

What an Apparent Power Gain Must Survive

A response-adaptive design may report more rejections than a fixed-randomization design in a simulation. That difference is not, by itself, evidence that adaptation improved power. The designs may have different total sample sizes, different type I error rates, different effect assumptions, or different variability in the numbers assigned to each arm. A valid comparison checks these features before interpreting the rejection rates.

The relevant quantity is power under a specified alternative: the probability that the complete design and its prespecified analysis reject \(H_0\) when that alternative is true. The phrase “complete design and analysis” matters. It includes the allocation rule, stopping and sample-size rules, outcome model, and test used to make the decision. In a simulation, power is estimated by repeatedly generating trials under the alternative and applying the same planned analysis to each trial.

For a fair comparison, match the total sample size \(N\), outcome distribution and treatment effect, significance level \(\alpha\), analysis, and rules for missing outcomes or stopping. If the goal is to compare allocation rules, vary the allocation rule while holding those other features fixed. If a design intentionally changes more than allocation, the result compares the packages of design choices, not the allocation rule alone.

A Reference Calculation for Fixed Allocation

A simple calculation helps identify a sample-size mismatch. Suppose outcomes in two arms are independent and normally distributed, with common known standard deviation \(\sigma\), and the arm sample sizes \(n_A\) and \(n_B\) are fixed. Let \(\delta=\mu_A-\mu_B\). For the difference in sample means, the standard error under this model is

For a two-sided \(z\) test at level \(\alpha\), the test rejects when the absolute standardized difference exceeds \(z_{1-\alpha/2}\), the \(1-\alpha/2\) quantile of the standard normal distribution. Under a particular alternative \(\delta\), the standardized statistic has mean \(\lambda=\delta/SE\) and variance 1. Thus its power is

This is a reference calculation, not a formula that automatically gives the power of an outcome-adaptive trial. It assumes fixed arm sizes and the stated outcome and testing model. When the allocation depends on accrued outcomes, the arm sizes and outcomes can be related; the reference calculation may then fail to describe the actual test’s operating characteristics. Simulating the full adaptive process is often necessary.

Check Type I Error Before Comparing Power

Power estimates can mislead when one design rejects too often under the null. If an adaptive procedure has type I error above its target \(\alpha\), its higher rejection rate under an alternative may partly reflect a more permissive decision rule rather than a more efficient design. Conversely, a procedure with type I error below target may appear to have lower power because it is more conservative.

Estimate type I error by simulating trials under \(H_0\), running the full allocation and analysis procedures, and recording the proportion that reject. If \(R\) of \(M\) simulated trials reject, the estimated rate is \(\hat{p}=R/M\). Its approximate Monte Carlo standard error is \(\sqrt{\hat{p}(1-\hat{p})/M}\). This quantifies simulation noise; it does not repair a test whose operating characteristics are genuinely miscalibrated.

Separate Effect Estimates From Rejection Rates

Power is the probability of rejection at a specified true effect; it is not the size of the estimated effect. Designs can yield different distributions of effect estimates even when evaluated under the same true \(\delta\). An outcome-dependent allocation can also make an estimator’s behavior differ from the fixed-design case, so the analysis and estimator must be appropriate for the allocation mechanism. The earlier material on estimator bias and bias–variance distinctions is directly relevant here.

A useful diagnostic is to report, for each design, the mean estimated treatment difference across all simulated trials, its variability, and the rejection rate. Do not report only the estimates from trials that rejected: selecting on statistical significance tends to select extreme estimates. Nor should a simulation give one design a larger assumed effect than the other and then attribute the resulting rejection difference to allocation.

Allocation Variability and a Variance Benchmark

Even with the same total \(N\), an adaptive design may produce variable arm sizes. Under the fixed-count, independent, common-variance model, the variance of the difference in means is \(\sigma^2(1/n_A+1/n_B)\). This gives a useful benchmark for asking whether allocation imbalance could offset a potential benefit. It does not, on its own, calculate unconditional adaptive-trial power.

For a fixed total \(N\), write \(n_B=N-n_A\) and define \(f(n_A)=1/n_A+1/(N-n_A)\). On \(0<n_A<N\),

Thus \(f\) is convex, and Jensen’s inequality gives \(E[f(n_A)]\geq f(E[n_A])\). In this variance benchmark, variability in allocation counts increases the expected variance factor relative to using the mean allocation count. The argument requires valid arm counts and applies to the stated function; it does not account for outcome-dependent selection, estimator bias, or the full distribution of a test statistic. Those features require design-specific evaluation.

A Practical Diagnostic Sequence

A useful comparison proceeds in an order that can expose misleading gains before they are interpreted. Use the same null and alternative models for each design, and preserve the planned allocation and analysis rules in every simulated replicate.

The main alternative to full design simulation is an analytic power calculation under a fixed-allocation model. It is valuable as a benchmark when its assumptions hold, but it does not represent outcome-dependent allocation automatically. Conditional power, which conditions on information available at a particular point in a trial, answers a different question from unconditional power across repetitions of the complete design. For adaptive-versus-fixed comparisons, unconditional simulation is often the clearest approach because it includes variation in allocation and outcomes together.

Common Mistakes and Limits

• Attributing a gain when total sample sizes differ. More observations can increase power. Match \(N\) or clearly identify sample size as part of the design contrast.
• Comparing rejection rates without checking the null. A higher type I error can inflate apparent power. Estimate both designs’ null rejection rates using the same target \(\alpha\).
• Changing the assumed treatment effect. Power depends on the alternative. Use the same true effect for both designs, then examine sensitivity to other plausible effects separately.
• Using only significant estimates. Rejection-selected estimates are not representative of estimates across all trials. Report all-trial summaries for estimation diagnostics.
• Treating a variance benchmark as the adaptive trial’s exact power. Variable, outcome-dependent counts can affect the joint behavior of assignment and outcomes. Simulate the actual allocation and planned analysis when the fixed-count assumptions do not hold.
• Overinterpreting simulation noise. A small difference in rejection rates may result from finite simulation replicates. Report the number of replicates and Monte Carlo uncertainty.

These checks diagnose a power claim; they do not establish that one allocation rule is universally preferable. The result remains conditional on the chosen outcome model, effect assumptions, estimator, test, and trial constraints. In particular, a design comparison should not be generalized beyond the scenarios evaluated without additional sensitivity analysis.