Estimating Power Across Effect Scenarios

Power Is a Design-Specific Rejection Probability

A group sequential design can reject the null hypothesis at one of several planned analyses, subject to its stopping rules and decision boundaries. Its power therefore depends not only on the treatment effect, but also on the full design: the analysis schedule, maximum information, boundary rules, and analysis procedure. In planning, a single power value may hide important differences between effects that are clinically plausible. Estimating power across a range of such effects shows how the design’s performance changes as the true effect changes.

A simulation estimates this operating characteristic by generating complete trial paths under a specified treatment effect and applying the planned analysis and stopping rules to each path. The simulated result for a path is whether the design rejects \(H_0\), not whether a particular interim statistic crosses a boundary considered in isolation.

The design and data-generating assumptions must be specified before interpreting this estimate. For example, state the effect scale and units, the allocation ratio, the outcome model and nuisance parameters, the planned information at each analysis, the decision boundaries, and the rule for handling events such as nonconvergence or missing data. A simulated estimate answers a conditional question: what rejection probability results under these particular assumptions and this particular design?

For a common normal model, consider a balanced two-arm trial with total sample size \(n\), common known standard deviation \(\sigma\), and treatment-control mean difference \(\delta\). The estimated difference has variance \(4\sigma^2/n\), so its standard error is \(2\sigma/\sqrt{n}\). If \(Z\) is the standardized difference, then under the assumed true effect its mean is \(\delta\sqrt{n}/(2\sigma)\). Equivalently, if information is defined as the inverse variance of the estimated difference, \(I=n/(4\sigma^2)\), then \(E_{\delta}(Z)=\delta\sqrt{I}\). These expressions use inverse-variance information; information should not be substituted into a formula intended for sample size.

Estimating Power and Its Monte Carlo Uncertainty

Under independent simulation replicates generated from the same design and effect scenario, each \(Y_r\) is a Bernoulli random variable with success probability \(\beta(\delta)\). Thus \(\sum_rY_r\) has a binomial distribution with parameters \(R\) and \(\beta(\delta)\). The sample proportion is unbiased for power, and its variance follows directly from the variance of a Bernoulli variable:

Replacing the unknown power in the variance formula by its estimate gives the usual estimated Monte Carlo standard error:

This standard error measures simulation noise, not uncertainty about the treatment effect, uncertainty in the assumed nuisance parameters, or uncertainty about whether the model represents a future trial. A normal-approximation interval is \(\hat{\beta}\pm1.96\widehat{SE}_{MC}\), but it can be unsuitable when the estimate is near 0 or 1 or when \(R\) is small. In those settings, a binomial interval such as a Wilson interval is preferable. Report the replication count alongside the estimate and interval.

Effect Scenarios and Analysis Timing

Choose effects that answer a planning question. A useful set might include no effect as a reference, a minimally important effect, and larger plausible effects. Keep units explicit, and distinguish a raw treatment difference from a standardized effect. For a continuous outcome, for instance, an effect of 2 mmHg is not interchangeable with a standardized difference of 0.2 unless the standard deviation used to standardize it is stated.

A power curve plots estimated rejection probability against the true effect, with Monte Carlo uncertainty shown or otherwise reported. Under a conventional one-sided efficacy design, power will often increase as a beneficial effect increases. A visibly irregular curve can result from simulation noise, too few replicates, an error in the scenario inputs, or an implementation problem. Do not silently smooth away irregularities: first determine whether they are larger than expected from Monte Carlo error and check the simulation setup.

Analysis timing can also change power. Compare schedules in information units, not just calendar time, because calendar timing depends on accrual and event rates. If two designs have the same maximum information but different interim information fractions, their rejection probabilities can differ: an earlier analysis offers a chance to stop earlier but has less information at that look. Boundaries may also differ because each design must use its own prespecified rule. There is no general guarantee that moving an interim analysis earlier or later increases power.

For a fair simulation comparison, specify what is held fixed, such as maximum information, the target error rate, and the true effect. Generate paths consistently under both schedules. When possible, apply both schedules to the same underlying simulated trial paths. This common-random-number approach makes the comparison more precise because variation shared by the two results can cancel in their difference. It does not make the two designs identical; it makes the simulation comparison paired.

Choosing Replications and Reading Power Curves

Replication counts should be chosen to make Monte Carlo error small enough for the planning decision. If power is expected to be near \(p\), the approximate standard error is \(\sqrt{p(1-p)/R}\). For a target 95% half-width \(h\), a useful planning calculation is:

Because \(p\) is not known in advance, use a pilot estimate or use \(p=0.5\) for a conservative calculation when no reliable estimate is available; the Bernoulli variance is largest at 0.5. A target precision should reflect the question. Distinguishing power of 0.79 from 0.80 requires more simulation than deciding whether power is roughly 0.6 or roughly 0.9.

Simulation is particularly useful when operating characteristics are awkward to calculate analytically under the planned data model, stopping rules, or implementation details. An analytical calculation, when available under matching assumptions, can be faster and can serve as a benchmark. Simulation has the advantage of accommodating a richer trial-path model, but it is only as trustworthy as its inputs and code. A grid of effect scenarios is also more informative than a single power value, but it does not justify precise claims at effect values that were never simulated. If interpolation is used, report it as an approximation and inspect whether the simulated curve supports it.

Diagnostics, Failure Modes, and Limits

Check that every simulated path uses the intended effect scenario, information schedule, boundaries, and stopping rule. Confirm that rejection is counted once per trial if any planned analysis triggers it, and that paths that do not reject by the end are coded as nonrejections rather than omitted. Verify the direction of the effect and the alternative hypothesis. In a balanced two-arm normal model, check the standardized mean against \(\delta\sqrt{n}/(2\sigma)\); confusing total sample size with inverse-variance information can create a substantial power error.

• Overreading small differences. A few percentage points may be within Monte Carlo error. Compare the difference with its uncertainty, preferably using a paired calculation when paths are matched.
• Changing multiple design features at once. If effect scenarios or analysis timing are compared while maximum information, boundaries, or nuisance assumptions also change, the result cannot be attributed to timing alone.
• Assuming a smooth curve is automatically correct. Power often changes smoothly with effect under regular models, but a smooth plot can still conceal an incorrect simulation. Inspect raw estimates and Monte Carlo uncertainty.
• Treating simulation precision as model certainty. A narrow Monte Carlo interval only indicates that repeated simulation under the specified assumptions would give a similar estimate. It does not cover misspecified outcome distributions, incorrect nuisance values, or operational departures.

Compared with a single-effect calculation, a scenario curve reveals how robust the design’s power is to uncertainty about the true effect. Compared with a purely analytical power calculation, simulation can represent complex trial paths but requires more computational work and explicit diagnostics. Neither approach removes the need to justify the assumed model. The simulated power curve is an operating-characteristic summary under stated conditions, not a forecast guaranteed to hold in an actual trial.