Evaluating Interim Efficacy Boundaries

Why Evaluate Interim Efficacy Boundaries?

An interim efficacy boundary is a rule for stopping a trial early when accumulating evidence is sufficiently strong to declare efficacy. Looking at the data more than once creates more than one opportunity to cross a threshold. Consequently, a nominal per-look significance level does not by itself describe the probability of at least one false efficacy declaration over the trial. Boundary evaluation asks how the whole stopping rule behaves across repeated hypothetical trials, not just what happens at a single analysis.

The key comparison is not simply “which boundary is lower?” A boundary that is easier to cross early may increase early stopping, but it must still control the trial-level type I error under the null. Under plausible treatment effects, the same design should also be assessed for power and for how much information or follow-up it uses on average. These quantities can be compared by simulating complete trial paths under specified data-generating assumptions and applying the planned analysis and stopping rule to each path.

Specify the Repeated-Look Design

Before simulating, specify the estimand and test, the maximum planned information, the interim information fractions, the efficacy boundaries, and what constitutes a declaration of efficacy. For example, a design may have two analyses at 50% and 100% of planned information. The second analysis is the final analysis; it is still part of the repeated-look rule. This tutorial considers one-sided efficacy boundaries for a single primary test. A two-sided test requires a rule that allocates error to both directions, and multiple endpoints or treatment comparisons require additional design specifications.

A useful reference model is the canonical group-sequential model. Let \(t_i\) be the fraction of maximum information available at look \(i\), with \(0<t_1<\cdots<t_K=1\), and let \(Z_i\) be the standardized test statistic at that look. Under the null, suppose the information increments are independent and the score process has the usual large-sample normal approximation. Then the joint null distribution is approximately multivariate normal, with

The statistics are correlated because later looks include information from earlier participants. They are not independent tests. In simulation, this dependence must be preserved. One way to generate a canonical path is to generate independent standard normal increments \(U_i\) and form

Under an alternative with final-information noncentrality parameter \(\delta\), the corresponding reference model has \(E(Z_i)=\delta\sqrt{t_i}\), with the same covariance structure. In a full trial simulation, it is usually better to generate the trial data under the alternative and apply the planned analysis, especially when the endpoint, missingness, variance, or analysis model makes the canonical approximation questionable.

Boundary Choices and Their Consequences

Two familiar shapes illustrate the trade-off. An O’Brien–Fleming-like design uses a stringent early boundary and a less stringent later boundary. A Pocock-like design uses a more nearly constant boundary across looks. The names describe boundary shapes, not universal numerical cutoffs: the actual values depend on the number and timing of looks, the sidedness of the test, and the target trial-level type I error.

For a two-look design at information fractions 0.5 and 1, consider the following illustrative one-sided boundaries targeting a trial-level type I error of about 0.025: an O’Brien–Fleming-like pair of 2.797 and 1.977, and a Pocock-like pair of 2.178 and 2.178. These numbers are specific to this illustrative design. A design team should calculate or validate boundaries for its exact analysis and information schedule rather than transplanting values from a different setting.

What the Simulation Estimates

For each simulated trial, apply the stopping rule in order. If a statistic crosses its efficacy boundary, record the look of the first crossing and do not treat later analyses as if they occurred. If no boundary is crossed, record that the trial continues to the maximum planned information. Under \(H_0\), the fraction of paths with any crossing estimates trial-level type I error. Under an alternative, the fraction with any crossing estimates power for this particular sequential design, assuming that crossing leads to the specified efficacy conclusion.

The early stopping fraction is the proportion of paths that stop before the final analysis. It should be reported alongside the probability of crossing at each look, since a total early-stop rate alone can conceal when stopping occurs. If stopping is only for efficacy, and a trial that does not stop early reaches information fraction 1, the mean information fraction used is

where \(R\) is the number of simulated trials and \(T_r\) is the information fraction at which trial \(r\) stops. This quantity is an average, not a guarantee about the duration of an individual trial. Information use can differ from calendar duration: event-driven trials, recruitment rates, and follow-up may make the relation between information and elapsed time nonlinear.

Why the Simulation Result Is Trustworthy—or Not

A short justification explains why repeated-look calibration is necessary. The trial-level rejection event is the union of the look-specific crossing events, so its probability is not simply the nominal error rate at one look. For two looks with upper boundaries \(b_1\) and \(b_2\), for example,

The second term is conditional on not having crossed at the first look; its value depends on the joint distribution of \(Z_1\) and \(Z_2\). Treating the two looks as independent, or simply assigning the full nominal level to both, changes this probability. Simulation estimates the union probability by counting paths in which the first crossing occurs at any scheduled analysis. This is also why a boundary comparison should use whole trial paths rather than separately simulated samples at each look.

Boundary evaluation is especially useful when the planned test statistic or data structure makes simple canonical calculations less dependable. However, simulation is not a substitute for checking the design assumptions. At minimum, verify the null calibration, the alternative scenarios, the information fractions, the rule for analysis failures, and consistency between the simulated and planned analysis. Check whether simulated estimates stabilize as the number of trials increases; report Monte Carlo standard errors for error rates and power. If a result is close to a decision threshold, increase the simulation count rather than treating a noisy estimate as decisive.

Also inspect failures by look and by scenario. A high rate of nonconvergence, missing analysis results, or unexpected information shortfalls can change the effective stopping rule. Do not silently count a failed analysis as “no efficacy crossing” unless that is the prespecified operational rule and the consequences are understood. If nuisance parameters are estimated, the simulated analysis should estimate them as planned. A well-calibrated result under a simplified normal model may not carry over to a small sample, a skewed endpoint, a changing variance, or a design with irregular information accumulation.

Comparisons, Limitations, and a Common Pitfall

When the test and information structure meet the canonical assumptions, analytic group-sequential calculations can provide boundary values and error probabilities efficiently. Simulation is more flexible: it can evaluate the actual planned data generation and analysis, including features for which the canonical approximation is inadequate. A strong approach is often to use analytic calculations as a reference and simulation as a validation under realistic scenarios. Simulation-based calibration can also be used when necessary, but it requires enough replicates and an explicit rule for how boundary adjustments are made.

A fixed nominal threshold at each look is another possible rule, but it does not automatically preserve the desired trial-level type I error. Nor does an early boundary that appears “conservative” at the final analysis prove that the full design is calibrated. The relevant quantity is the probability of any false crossing across all planned looks under \(H_0\). Boundary choice also cannot resolve questions about the scientific estimand, the credibility of the assumed treatment effects, operational feasibility, or other sources of multiplicity; those require their own design decisions.

Common mistake: “Each look uses a 2.5% test, so the trial’s type I error is 2.5%.” This reasoning ignores repeated opportunities to cross and the dependence between analyses. Instead, simulate under \(H_0\), apply the complete stopping rule to each path, and estimate the proportion of trials with at least one efficacy crossing. Use the resulting estimate and its Monte Carlo uncertainty to judge calibration.