Classical Efficacy Boundaries And Effect Size

Why the Same Boundary Has Different Roles

A classical efficacy boundary is chosen to limit the chance of a false efficacy conclusion when the null hypothesis is true. Its calibration therefore uses the distribution of the test statistics under \(H_0\). Once the boundary is fixed, however, the chance of crossing it in a trial with a real treatment effect is a different quantity: it depends on how far the alternative distribution is shifted toward the boundary. That probability is part of the design’s power.

This distinction matters because a critical value is not a promise that a trial will have a particular probability of success. It is a decision threshold. The null distribution determines how often a sequence of statistics crosses the threshold when there is no effect; the alternative distribution determines how often it crosses when an effect of a specified size is present.

Model, Assumptions, and the Null Calibration

A standard setting has a prespecified sequence of looks, with information \(I_1<\cdots<I_K\). For a treatment effect \(\Delta\), the statistic at look \(k\) is approximately normal with unit variance and mean \(\Delta\sqrt{I_k}\). Thus, under the null effect \(\Delta=0\), \(Z_k\) has mean zero; under a positive alternative, its mean is shifted upward. Information is measured on the scale that makes \(\Delta\sqrt{I_k}\) dimensionless.

For cumulative statistics based on nested data, the standard canonical model also has \(\operatorname{Corr}(Z_j,Z_k)=\sqrt{I_j/I_k}\) for \(j<k\). The statistics are therefore generally dependent: observations contributing to an earlier statistic also contribute to later ones. In applications, a normal or asymptotic normal model is used when appropriate, and the information sequence and dependence structure used for calibration must represent the planned design.

For upper boundaries, define \(A_k\) as the event that the trial first crosses at look \(k\): it has not crossed at any earlier look, and \(Z_k\ge c_k\). The events \(A_1,\ldots,A_K\) are disjoint. Therefore the null error rate is the sum of their null probabilities:

Classical boundaries are calibrated by choosing the \(c_k\)'s so that this joint probability equals the target significance level \(\alpha\), or is no greater than it. Because the statistics are dependent, adding the marginal null tail probabilities \(P_0(Z_k\ge c_k)\) generally double-counts paths that cross at more than one look. Calibration instead accounts for the joint path and, in particular, for the first look at which a crossing occurs.

How Effect Size Changes Crossing Probability

Under the canonical normal model, an alternative effect shifts the mean of each statistic by \(\Delta\sqrt{I_k}\). For a fixed look, the marginal probability of crossing its upper boundary is \(P_\Delta(Z_k\ge c_k)=1-\Phi(c_k-\Delta\sqrt{I_k})\), where \(\Phi\) is the standard normal cumulative distribution function. Increasing a positive \(\Delta\), or increasing information while holding \(\Delta\) fixed, increases this marginal probability.

Overall power is not just the marginal probability at the final look. It is the probability of crossing at any look, with earlier crossing paths counted once. In full form,

The joint alternative distribution is needed to calculate this probability. For example, an early crossing and a later crossing are not independent events. The first-crossing probabilities must use the joint distribution of the sequence, just as they do under the null.

Worked Calculations

Why Power Increases With a Positive Effect

For upper efficacy boundaries, the dependence of crossing probability on a positive effect can be justified directly. Couple the statistics under different effects by using the same null-centered random path \(Z_k(0)\), and write \(Z_k(\Delta)=Z_k(0)+\Delta\sqrt{I_k}\). If \(\Delta_2\ge\Delta_1\ge0\), then for every look \(k\),

Consequently, any path that crosses an upper boundary under \(\Delta_1\) also crosses an upper boundary under \(\Delta_2\), using this coupling. The crossing event for the smaller effect is contained in the crossing event for the larger effect, so its probability cannot be greater. This monotonicity concerns positive effects and upper boundaries; it does not mean that every individual trial becomes more likely to stop at a particular look.

Interpreting the Design and Diagnosing Problems

A null-calibrated boundary and an alternative power calculation answer different design questions. Calibration asks how often the planned rule rejects \(H_0\) when the null model holds. Power asks how often that same rule rejects \(H_0\) when a specified effect and information sequence hold. It is therefore useful to report the assumed effect size and information alongside a power estimate; “the design has power” is incomplete without them.

A common alternative to group sequential monitoring is a fixed-sample design with only a final analysis. In a one-look design, a marginal normal tail probability is the full type I error or power. In a group sequential design, repeated looks create multiple opportunities to cross, so the nominal cutoff for a single test cannot simply be used at every look while claiming the same overall error rate. Calibrated classical boundaries account for those repeated opportunities under the null. Under an alternative, one must then calculate power for the entire sequence rather than treating the looks as independent tests.

Several checks help identify a mistaken calculation or an unsuitable model:

• Check which distribution is being used. The boundary calibration uses the null distribution. Power uses an alternative with a stated effect size.
• Check the statistic’s mean shift. Confirm that \(\Delta\sqrt{I_k}\) is dimensionless and uses the effect and information on compatible scales.
• Check the whole path. Summing marginal crossing probabilities is generally wrong because crossings at different looks can occur on the same path.
• Check the planned design. The information times, look schedule, and dependence structure used for calibration should match the analysis plan. Material departures may invalidate the stated error rate.
• Check the direction and rule. The monotonicity argument above is for positive effects and upper boundaries. A two-sided design uses a different crossing event.

The normal model and canonical dependence are approximations in many applications. If the test statistic is poorly approximated by its assumed distribution, the information estimates are inaccurate, or the actual analysis departs from the planned rule, the calculated type I error and power may not describe the trial well. Calibration does not remove these modeling requirements.