Power Loss From Enforcing A Nonbinding Rule

Why a Nonbinding Rule Can Still Cost Power

A futility boundary may be called nonbinding because ignoring it does not invalidate the design’s type I error control. That does not mean enforcing it has no consequence. Under a design alternative, some trajectories can cross a lower futility boundary early and, if followed onward, cross an upper efficacy boundary later. Enforcing futility stops those trajectories before the later rejection can occur.

The relevant comparison is between two versions of the same design: one that continues after a futility crossing and one that stops for futility. Their type I error properties are not the question here. The focus is the change in rejection probability under the design alternative when the optional rule is actually followed.

Definition: The power loss from enforcing a nonbinding futility rule, under a specified design alternative, is the rejection probability when the rule is ignored minus the rejection probability when it is enforced. A lost-power path is one that would eventually cross an efficacy boundary if continued, but crosses a futility boundary first.

Pathwise Definition of the Power Difference

Consider a one-sided group sequential design with \(K\) analyses. Let \(Z_j\) be the test statistic at look \(j\), \(u_j\) its upper efficacy boundary, and \(l_j\) its lower futility boundary at an interim look. Assume \(l_j<u_j\), so a statistic cannot cross both boundaries at the same look. Futility is considered at looks \(j=1,\ldots,K-1\); the final look has an efficacy boundary but no futility stop.

To compare the rules, imagine the complete sequence of statistics that would be available if the trial continued to the final analysis. Define \(T_E=\min\{j:Z_j\ge u_j\}\), with \(T_E=\infty\) if no efficacy boundary is crossed. Define \(T_F=\min\{j<K:Z_j\le l_j\}\), with \(T_F=\infty\) if no futility boundary is crossed. Under the rule that ignores futility, rejection occurs when \(T_E<\infty\). Under the rule that enforces it, rejection occurs when efficacy is crossed before futility: \(T_E<T_F\).

\begin{aligned} \text{Power}_{\text{ignore}} &= P_{\mathrm{alt}}(T_E<\infty),\\ \text{Power}_{\text{enforce}} &= P_{\mathrm{alt}}(T_E<T_F),\\ \text{Power loss} &= P_{\mathrm{alt}}(T_F<T_E<\infty). \end{aligned}

The last equality follows by partitioning paths that would reject if futility were ignored. Each such path either crosses efficacy before futility, so it still rejects, or crosses futility first, so it is stopped and does not reject. Because \(l_j<u_j\), the two events cannot occur at the same look. Thus the power difference is exactly the probability of a futility crossing before a later efficacy crossing, under the specified alternative.

Worked Example: Classifying Hypothetical Sample Paths

Suppose a hypothetical two-look design has \(u_1=2.5\), \(l_1=0.5\), and \(u_2=2.0\). The following values are hypothetical complete paths, including the second-look value even if a trial would have stopped at the first look. Classify each path.

Path	\(Z_1\)	\(Z_2\)	Classification
A	0.2	2.2	Futility first, then later efficacy: lost rejection
B	2.6	2.3	Efficacy first: rejection retained
C	0.8	2.1	No futility crossing, then efficacy: rejection retained
D	-0.1	1.7	Futility first, with no later efficacy: no rejection either way

For path A, \(0.2\le0.5\) gives \(T_F=1\), while \(2.2\ge2.0\) gives \(T_E=2\). Hence \(T_F<T_E<\infty\), so this path contributes to power loss. For path B, \(2.6\ge2.5\), giving \(T_E=1\); a later futility crossing cannot undo an efficacy rejection. Path C has \(0.8>0.5\) and \(2.1\ge2.0\), so it rejects under either rule. Path D is stopped for futility but would not later reject, so it does not contribute to the difference in rejection probability.

Justification Under the Canonical Normal Model

For a numerical calculation, suppose the standardized statistics follow the canonical joint normal model. At information fraction \(t_j\), with \(0<t_1<\cdots<t_K=1\), assume \(Z_j\sim N(\theta\sqrt{t_j},1)\) under the specified design alternative, where \(\theta\) is the standardized effect on the final-analysis scale. The joint distribution has \(\operatorname{Corr}(Z_i,Z_j)=\sqrt{t_i/t_j}\) for \(i<j\). These assumptions describe the usual normal-increment approximation: information accumulates as planned, the effect is represented by a fixed \(\theta\), and the standardized increments are independent. The calculation below uses this model; other data structures may require a different joint distribution.

With two looks, futility can be crossed only at look 1. If \(l_1<u_1\), a lost-power path must have \(Z_1\le l_1\) and \(Z_2\ge u_2\). The first condition also rules out an interim efficacy crossing. Under the joint normal model, the conditional distribution needed to calculate this probability is \(Z_2\mid Z_1=x\sim N\bigl(\theta\sqrt{t_2}+\rho[x-\theta\sqrt{t_1}],\,1-\rho^2\bigr)\), where \(\rho=\sqrt{t_1/t_2}\). This is the conditional-normal formula applied to the specified means, variances, and correlation.

\begin{aligned} P_{\mathrm{alt}}(T_F<T_E<\infty) &=P_{\mathrm{alt}}(Z_1\le l_1,\ Z_2\ge u_2)\\ &=\int_{-\infty}^{l_1} \frac{1}{\sqrt{2\pi}} \exp\!\left[-\frac{(x-\theta\sqrt{t_1})^2}{2}\right] \left[ 1-\Phi\!\left( \frac{u_2-\theta\sqrt{t_2}-\rho(x-\theta\sqrt{t_1})} {\sqrt{1-\rho^2}} \right) \right]\,dx . \end{aligned}

Here \(\Phi\) is the standard normal cumulative distribution function. The integrand is the density of the first-look statistic at \(x\), multiplied by the conditional probability that the second-look statistic reaches efficacy given that first-look value. Integrating only through \(l_1\) restricts the calculation to trajectories stopped for futility.

Worked Example: Calculating a Two-Look Power Loss

Consider a hypothetical two-look design with \(t_1=0.5\), \(t_2=1\), \(l_1=0.5\), \(u_1=2.5\), and \(u_2=2.0\). Under a specified hypothetical alternative, let \(\theta=1.5\). Assume the canonical normal model just stated, including its information-based correlation structure. Then \(\theta\sqrt{t_1}=1.0607\), \(\theta\sqrt{t_2}=1.500\), and \(\rho=\sqrt{0.5}=0.7071\). The conditional standard deviation is \(\sqrt{1-\rho^2}=\sqrt{0.5}=0.7071\).

Since \(0.5<2.5\), a first-look futility crossing excludes an interim efficacy crossing. The lost-power probability is therefore \(P(Z_1\le0.5,\ Z_2\ge2.0)\). Substituting the values into the conditional-normal integral gives

\begin{aligned} P_{\mathrm{alt}}(T_F<T_E<\infty) &=\int_{-\infty}^{0.5} \phi(x-1.0607) \left[ 1-\Phi\!\left( \frac{2.0-1.500-0.7071(x-1.0607)}{0.7071} \right) \right]dx\\ &\approx 0.0122 . \end{aligned}

Thus, under this model and alternative, enforcing futility lowers the rejection probability by about \(0.0122\), or 1.22 percentage points. This is the probability of the specific lost-path event, not the overall futility-crossing probability and not the probability that any particular trial will reject. The calculation depends on the stated alternative and joint model.

Estimating the Loss by Simulation

The same pathwise distinction can be used in simulation when a joint probability is inconvenient to evaluate analytically. Simulate complete trajectories under the design alternative, apply both rules to each trajectory, and count trajectories with \(T_F<T_E<\infty\). To estimate the difference, it is important to use the same complete paths for both rules; then the estimated difference is exactly the proportion of simulated paths that would reject if continued but are stopped first for futility.

Worked Example: Paired Simulation Counts

Suppose 10,000 hypothetical complete trajectories are simulated under a specified alternative and design model. Of these, 840 cross an efficacy boundary when futility is ignored. Among those 840 trajectories, 95 cross futility first and efficacy later. The remaining \(840-95=745\) reject under the enforced rule.

The estimated rejection probabilities are \(\widehat{\text{Power}}_{\text{ignore}}=840/10{,}000=0.0840\) and \(\widehat{\text{Power}}_{\text{enforce}}=745/10{,}000=0.0745\). Their difference is \(0.0840-0.0745=0.0095=95/10{,}000\). In this hypothetical simulation, enforcing the rule reduces estimated power by 0.95 percentage points. These are simulation estimates under the chosen alternative and model, not observed results from a clinical study.

Interpretation, Diagnostics, and Limitations

A nonbinding rule is optional for the purpose of type I error control, but it is not consequence-free when enforced. The earlier material on nonbinding futility and type I error protection concerns control under the null; the power difference here is evaluated under a specified alternative. A rule can preserve type I error protection when ignored and still reduce power when followed.

Several checks help detect mistakes in a calculation or implementation:

Check event order. A path is lost only when futility occurs before efficacy and a later efficacy crossing would have occurred. A futility crossing with no later rejection does not reduce power relative to ignoring the rule.
Check the boundaries and indices. Verify \(l_j<u_j\), apply futility only at the planned interim looks, and do not treat a later hypothetical statistic as observed after an enforced stop.
Check the probability model. In the canonical model, use the correct means and correlation for each information fraction. Incorrectly treating cumulative \(Z\)-statistics as independent changes the joint probability.
Check the simulation denominator. The loss estimate is the number of paths satisfying \(T_F<T_E<\infty\), divided by all simulated paths under the alternative—not by only the paths that crossed futility or only those that rejected.
Check the direction and scale. Enforcing an additional stop cannot increase rejection probability on the same paths. The difference is a probability and must be between 0 and 1; report it as a probability or in percentage points.

The main alternative to enforcing the rule is to leave it advisory and continue according to the original analysis plan. That avoids these futility-induced lost rejections, but it also gives up the practical stopping decision the rule was intended to support. The pathwise calculation does not by itself decide whether the reduction is acceptable: it quantifies the trade-off under one specified alternative and model. It also does not guarantee the same loss under a different alternative, information pattern, or departure from the assumed joint distribution.

Common mistake: “Because futility is nonbinding, enforcing it cannot change power.” Nonbinding describes the rule’s optional status for type I error control; it does not make the stopping action invisible to the rejection event. The correction is to compare the two rejection events under the alternative and count paths with \(T_F<T_E<\infty\).

Check Your Understanding

Answer each question using the pathwise definition. The solutions are included so that each calculation can be checked.

In a two-look design with \(l_1=0.4\) and \(u_2=2.1\), a hypothetical complete path has \(Z_1=0.3\) and \(Z_2=2.4\). Does it contribute to power loss if the first-look efficacy boundary is above \(0.4\)?
A path crosses futility at the first interim look and never crosses any later efficacy boundary. Does it contribute to the power difference? Explain.
If the probability of rejection is 0.62 when futility is ignored and 0.59 when it is enforced, what is the lost-power probability under the specified alternative?
Why should the two rules be compared on the same simulated complete trajectories?
For the two-look canonical model, state the event whose probability is the power loss when \(l_1<u_1\).

Solutions. (1) Yes. Since \(Z_1=0.3\le0.4\), futility occurs first; since \(Z_2=2.4\ge2.1\), the continued path later rejects. It satisfies \(T_F<T_E<\infty\). (2) No. It is stopped under the enforced rule, but it would not reject under the rule that ignores futility, so it does not change the rejection indicator. (3) The difference is \(0.62-0.59=0.03\), or 3 percentage points. (4) Paired paths make the difference equal to the fraction of trajectories that reject under continuation but not under enforcement, without adding variation from comparing separate simulated samples. (5) The event is \(\{Z_1\le l_1,\ Z_2\ge u_2\}\): futility at look 1 followed by efficacy at look 2 if the trial were continued.