Stopping Rules With Separate Efficacy Limits

Why Separate Efficacy Limits Need an Explicit Rule

A two-sided group sequential design monitors accumulating evidence at a sequence of planned analyses. At each look, the trial may stop for evidence favoring an effect in either direction, or it may continue to the next look. When the upper and lower efficacy limits are specified separately, the decision cannot be reduced to checking one absolute-value cutoff unless the limits happen to be exact mirror images. The rule must say which statistic is compared with which limit, and what happens after the first crossing.

The essential operational principle is first crossing while the trial is still active. Once an efficacy limit is crossed, the efficacy stopping decision is made at that look. Later statistics are not used to reverse that decision, because the trial would no longer be continuing under the rule.

Definition: Let \(K\) be the number of planned analyses, indexed in chronological order by \(k=1,\ldots,K\). Let \(Z_k\) be the signed, standardized test statistic at look \(k\), oriented so that larger values favor effects in the positive direction. Let \(l_k\) and \(u_k\) be the lower and upper efficacy limits at look \(k\), with \(l_k<u_k\). At an active look \(k\), stop for positive-direction efficacy if \(Z_k\geq u_k\), and stop for negative-direction efficacy if \(Z_k\leq l_k\). Continue to look \(k+1\) if \(l_k<Z_k<u_k\) and \(k<K\). At the final look, stop the trial in all cases: a boundary crossing gives the corresponding efficacy decision, while a statistic between the limits gives no efficacy rejection under this rule.

Assumptions and the Analysis Scale

The rule is meaningful only when the monitoring plan and the statistic are defined consistently. Assume the planned looks are ordered, the statistic is available at every scheduled analysis, and the signed statistic uses the same direction throughout: positive values favor the positive effect and negative values favor the negative effect. For example, a design may use \(Z_k=(\hat{\theta}_k-\theta_0)/SE_k\), where \(\hat{\theta}_k\) estimates an effect \(\theta\), \(\theta_0\) is the null value, and \(SE_k\) is its standard error. This statistic is dimensionless; its sign is determined by the effect estimate relative to the null value.

The limits must be stated on the same scale as \(Z_k\), and each pair must satisfy \(l_k<u_k\). The limits may vary across looks, and the lower limit need not equal the negative of the upper limit. A monitoring plan should specify the scheduled looks, the applicable limits at each look, and the action for crossing either limit. It should also make clear which analysis statistic and effect direction are intended.

In a group sequential design, the looks are generally based on accumulating information, and the statistics across looks are dependent. Thus, a list of plausible-looking cutoffs is not by itself evidence that a design has a desired overall Type I error rate. The limits used in practice must be justified for the design’s intended error control under its analysis assumptions. This tutorial applies a specified set of limits; it does not calculate or allocate the error rate that would justify them.

Worked Example: Crossing the Lower Limit at an Interim Look

Suppose a hypothetical trial has three planned looks. Its stated efficacy limits are \(u_1=2.8,\ u_2=2.4,\ u_3=2.0\) and \(l_1=-2.6,\ l_2=-2.3,\ l_3=-1.9\). Assume these are the limits specified for this design and that the statistic is positive in the direction of benefit. At the first look, the estimated treatment effect relative to the null is \(0.8\) units with \(SE_1=0.5\) units. At the second look, it is \(-1.15\) units with \(SE_2=0.5\) units.

At look 1, the standardized statistic is

Z_1=\frac{0.8-0}{0.5}=1.6.

Since \(-2.6<1.6<2.8\), neither efficacy limit is crossed. The trial continues to look 2.

At look 2,

Z_2=\frac{-1.15-0}{0.5}=-2.30.

The lower-limit condition is \(Z_2\leq l_2=-2.3\), and it holds with equality. The trial therefore stops at look 2 for evidence in the negative direction under the specified rule. The planned third-look statistic is not consulted to undo that decision. The estimates are in the effect’s units and the standard errors have the same units, so the resulting \(Z\) statistics can be compared with the dimensionless limits.

First Crossing and the Stopping Index

The rule can be written compactly by defining a crossing at look \(k\) as either \(Z_k\geq u_k\) or \(Z_k\leq l_k\). The stopping look is the first crossing, if one occurs, and otherwise the final planned look. This definition makes explicit that a later statistic cannot change an earlier stopping decision.

T=\min\{k: Z_k\geq u_k\ \text{or}\ Z_k\leq l_k\},

when the set is nonempty; if no crossing occurs at any planned look, set \(T=K\). At \(T\), classify the result as positive-direction efficacy if \(Z_T\geq u_T\), negative-direction efficacy if \(Z_T\leq l_T\), and no efficacy rejection if \(T=K\) and \(l_K<Z_K<u_K\).

The final-look clause matters: “no crossing” at an interim look means continue, but “no crossing” at the final look means stop without rejecting for efficacy. It does not mean that the trial continues indefinitely, nor does it establish that the null hypothesis is true.

Justification: At each look, the two crossing conditions cannot both hold because \(l_k<u_k\). If \(Z_k\geq u_k\), then \(Z_k>l_k\); if \(Z_k\leq l_k\), then \(Z_k<u_k\). Thus, each active look has at most one efficacy direction. If neither condition holds, the statistic lies strictly between the limits, so the rule continues when \(k<K\). Since the plan has only \(K\) looks, a crossing stops the trial at its first crossing, and a path with no crossing reaches look \(K\) and stops there. These clauses cover every possible sequence of statistics.

Worked Example: Crossing the Upper Limit Only at the Final Look

Use the same hypothetical limits: upper limits \(2.8,\ 2.4,\ 2.0\) and lower limits \(-2.6,\ -2.3,\ -1.9\). Suppose the estimated effects and standard errors at the three looks are, respectively, \(0.55\) and \(0.5\) units, \(0.95\) and \(0.5\) units, and \(1.025\) and \(0.5\) units.

The statistics are calculated using \(Z_k=(\hat{\theta}_k-0)/SE_k\):

Z_1=\frac{0.55}{0.5}=1.10,\qquad Z_2=\frac{0.95}{0.5}=1.90,\qquad Z_3=\frac{1.025}{0.5}=2.05.

At look 1, \(-2.6<1.10<2.8\), so the trial continues. At look 2, \(-2.3<1.90<2.4\), so it continues again. At the final look, \(Z_3=2.05\geq u_3=2.0\). The trial stops at look 3 for positive-direction efficacy. It would be incorrect to compare \(Z_3\) with the earlier upper limit \(2.8\): the applicable upper limit is the one assigned to look 3.

When No Efficacy Limit Is Crossed

A two-sided efficacy rule is not a promise that every trial will reject in one direction or the other. A trial may remain inside the continuation region through the final analysis. The operational decision is then to stop at the final look without an efficacy rejection. Depending on the broader protocol, other rules or outcomes may also be relevant, but they should not be silently substituted for the efficacy rule being applied here.

Worked Example: Reaching the Final Look Without a Crossing

For a third hypothetical trial with the same limits, suppose the treatment-effect estimates at the three looks are \(0.20,\ 0.30,\) and \(0.35\) units, and the standard error is \(0.5\) units at each look. The statistic at each look is the estimate divided by its standard error, since \(\theta_0=0\):

Z_1=\frac{0.20}{0.5}=0.40,\qquad Z_2=\frac{0.30}{0.5}=0.60,\qquad Z_3=\frac{0.35}{0.5}=0.70.

At look 1, \(-2.6<0.40<2.8\); at look 2, \(-2.3<0.60<2.4\); and at look 3, \(-1.9<0.70<2.0\). No efficacy limit is crossed. The trial continues after the first two looks and stops at the final look with no efficacy rejection. This conclusion describes the rule’s decision; it is not proof that the treatment effect is zero.

Implementation Checks, Alternatives, and Limits

Before applying a set of separate limits to a monitoring path, check the protocol and analysis output for the following:

Direction: Confirm that positive and negative values of \(Z_k\) have the intended interpretation. Reversing the statistic’s sign without reversing the directional labels can reverse the conclusion.
Look matching: Use \(l_k\) and \(u_k\) for the current look, not limits from another look. The final look has its own pair of limits.
Scale: Compare standardized statistics with standardized limits, not raw effect estimates with \(Z\)-scale cutoffs. Check that an estimate and its \(SE\) use compatible units before forming \(Z_k\).
First crossing: Record the first look that crosses a limit. Do not continue the efficacy decision using later statistics after the stopping rule has triggered.
Plan integrity: Check that every look has both limits, that \(l_k<u_k\), and that the analysis follows the planned look schedule and statistic definition.

The main alternative for a two-sided decision is to use symmetric limits, where \(l_k=-u_k\) at every look. That is a special case of the separate-limit rule, not a requirement of it. A single fixed-sample analysis is another alternative, but it does not use the sequence of interim efficacy decisions described here. Futility stopping is also distinct: a futility rule can stop for lack of promise, whereas the limits in this tutorial specify efficacy stopping in the two directions. A full monitoring plan may contain more than one kind of rule, but each rule’s role and precedence must be stated.

A common mistake is to assume that because the rule has upper and lower cutoffs, crossing either one automatically gives a valid overall two-sided significance level. That reasoning is incomplete. Repeated looks are dependent, and the probability of stopping across the whole sequence depends on the joint behavior of the statistics and on all the limits. The correction is to treat the limits as a complete, design-specific sequence and verify their operating characteristics using the appropriate group sequential methods. Applying the stopping rule tells you what decision follows a path; it does not, by itself, establish how often that path or decision occurs under a specified null model.

Separate limits are useful because a monitoring plan can express distinct stopping thresholds in the two effect directions and at different looks. Their flexibility also makes careful specification essential: the statistic’s sign, the look index, the applicable pair of limits, and the consequence of a first crossing all belong in the written rule.

Check Your Understanding

For each question, apply the rule as stated. Solutions follow the questions.

At an interim look, \(l_k=-2.4\), \(u_k=2.5\), and \(Z_k=-2.1\). Should the trial stop for efficacy or continue?
At the final look, \(l_K=-1.8\), \(u_K=2.1\), and \(Z_K=2.1\). What is the decision?
Suppose a path crosses the lower limit at look 2, but a statistic hypothetically reported for look 3 is above its upper limit. Which look determines the efficacy stopping decision?
Why is it not enough to know that both upper and lower limits are specified in order to claim a particular overall Type I error rate?

Solutions. (1) Since \(-2.4<-2.1<2.5\), neither limit is crossed, so continue if this is not the final look. (2) Since \(Z_K=u_K=2.1\), equality counts as crossing: stop at the final look for positive-direction efficacy. (3) Look 2 determines the decision because it is the first crossing; the trial would stop there under the rule. (4) The overall error rate depends on the joint behavior of the statistics across looks and on the full boundary sequence, so the sequence must be justified under the design’s assumptions rather than inferred from the existence of two limits.