Power Under Clinically Relevant Alternatives

Why Examine Power Across Alternatives?

A design can have an acceptable rejection rate at one assumed treatment effect and perform poorly at effects that matter to patients or clinicians. Reporting power at only a large, optimistic effect can therefore give an incomplete picture of the design. A more informative analysis examines a range of plausible effects, including the smallest effect considered clinically worthwhile.

For an adaptive trial, the probability of rejection depends on the complete trial algorithm: interim decisions, sample-size changes, treatment selection, stopping rules, and the final analysis. Power is consequently a property of the design and the data-generating scenario. It is not a fixed property of the test statistic alone.

Specify the Alternatives Before Simulating

A scenario must say more than “the treatment works.” It specifies the treatment effect and the other quantities needed to generate trial data, such as outcome variability, event rates, accrual, missingness, and any association between efficacy and safety. These additional quantities are often called nuisance parameters: they affect the trial’s behavior but are not the treatment effect being varied.

For a simple illustration, let \(\delta\) denote the mean difference in a continuous outcome, defined as the experimental-treatment mean minus the control-treatment mean. Suppose higher values are better, the null hypothesis is \(H_0:\delta\leq 0\), and the alternative is \(H_a:\delta>0\). If clinicians regard a difference of 5 points as the smallest worthwhile benefit, \(\delta_{\mathrm{clin}}=5\) points is a useful anchor for the scenario grid. Smaller positive effects may also matter for understanding performance near the decision boundary; larger effects show how rejection probability behaves when benefit is more substantial.

The scenario grid should be clinically interpretable, not chosen merely because it produces a favorable power estimate. State the effect scale and units, specify which nuisance assumptions are held fixed as \(\delta\) varies, and include plausible departures when those assumptions are uncertain. A single grid in which only \(\delta\) changes answers a focused question about sensitivity to effect size. Additional grids can answer whether that pattern changes under different event rates, variability, dropout, or safety conditions.

Estimating Rejection Probability

For one scenario \(S\), run \(R\) simulated replicates. Define \(I_r=1\) if replicate \(r\) ends with rejection of \(H_0\), and \(I_r=0\) otherwise. Apply the trial’s prespecified rules in every replicate, including adaptive rules and the final analysis rule. The estimate is the mean of these indicators:

This estimate targets the probability that the specified algorithm rejects under \(S\). It does not target the probability of completing the trial, selecting the correct dose, or obtaining a particular estimate. Those are different operating characteristics. When a trial stops early for futility and does not reject, that replicate contributes \(I_r=0\). A replicate should not disappear from the denominator merely because it stopped early or produced an inconvenient outcome.

The justification for the estimator is direct. Under independent, identically generated replicates from the same scenario, each \(I_r\) is a Bernoulli random variable with success probability \(p_S=P(\text{reject }H_0\mid S)\). Therefore \(E(I_r)=p_S\), and

Thus the simulated proportion estimates the target rejection probability without systematic bias under this simulation setup. As the number of independent replicates increases, the law of large numbers implies that the proportion converges to \(p_S\). This does not make a finite simulation exact: the estimate varies across simulation runs, so its precision should be considered when deciding how firmly to interpret small differences between scenarios.

Read the Pattern, Not Just One Number

A plot or table of estimated power against treatment effect makes several design questions visible. Does the estimated rejection probability become substantial by the smallest worthwhile effect? Is it already high only at effects larger than clinicians consider plausible? Does it change sharply over a narrow range, or gradually? These questions help assess whether the design is aligned with the clinical decision it is meant to support.

For a fixed, well-behaved test of a simple location effect, power often increases as the true effect moves farther from the null in the favorable direction. But an adaptive design’s simulated estimates are not required to form a perfectly increasing sequence. Interim stopping, treatment-selection rules, discrete outcomes, and finite simulation variability can produce small reversals. A reversal should prompt investigation, not an automatic conclusion that the design is defective or that the effect has become less detectable.

Comparison With Other Design Assessments

Power under alternatives and type I error calibration answer related but distinct questions. Type I error is the rejection probability under the null, such as \(\delta=0\) for the hypotheses above. Power is the rejection probability under a specified alternative scenario, such as \(\delta=5\) points. A design that has calibrated type I error is not thereby guaranteed to have adequate power at clinically worthwhile effects. Conversely, a high simulated rejection rate at an alternative does not show that the null rejection rate is controlled.

An analytic power calculation can be efficient when the design is fixed and its model and test have a tractable distribution. Simulation is useful when the full adaptive algorithm makes that calculation difficult or when the goal is to represent realistic features such as interim stopping and changing allocation. Simulation does not remove the need for assumptions: it calculates the implications of the implemented generator and algorithm. If either is misspecified or coded incorrectly, a precise estimate can still answer the wrong question.

Power also differs from a confidence interval’s coverage, estimator bias, treatment-selection accuracy, sample size, and duration. Those quantities may be important to design assurance, but they are not interchangeable with the probability of rejecting \(H_0\). Define the decision event first, then count that event consistently.

Diagnostics and Common Pitfalls

• Using an implausible effect grid. A large effect may yield impressive power but be clinically unrealistic. Anchor the grid to the smallest worthwhile effect and to a defensible range of alternatives.
• Changing more than the effect without saying so. If outcome variability, dropout, or safety risk also changes between rows, the differences cannot be attributed to treatment effect alone. Label such rows as distinct combined scenarios.
• Conditioning on trial completion. Calculating rejection only among trials that reach the final analysis can overstate the probability of rejection by excluding non-rejections after early stopping. Use all initiated replicates unless the target is explicitly a conditional probability.
• Confusing power with observed efficacy. A simulated power of 80% does not mean an 80% chance the treatment is effective, nor does it predict the result of one future trial. It is a rejection probability conditional on the scenario and design.
• Assuming the curve must be monotone. A jagged curve may reflect simulation variability, an implementation issue, or genuine features of the adaptive rules. Check the generator, decision logs, and precision before interpreting it.

A useful audit records, for every scenario, the effect and units, the nuisance assumptions, the number of attempted replicates, the number of rejections, and the exact rejection rule. Examine whether early stops and other terminal outcomes were classified according to protocol. If the resulting power pattern is unexpected, inspect replicate-level decisions rather than relying only on a summary table.