P-Values After Futility Stopping

Why a Futility Stop Still Has a P-Value

A futility stop means that the trial will not continue because the interim result does not meet the design’s criterion for continuing. It does not erase the observed outcome from the sample space. In a group sequential analysis, the data are the stopped outcome: both the look at which the trial stopped and the statistic observed at that look. A p-value must therefore be calculated over outcomes that could actually have been observed under the stopping rule.

This matters especially when the observed statistic is low. For a one-sided alternative in which larger values of the test statistic favor the alternative, a low observed value ordinarily has a large upper-tail probability. After early stopping, however, the stagewise p-value is determined by the ordering of stopped outcomes, not by the final-look normal tail alone. In the stagewise ordering used here, earlier stopping looks precede later stopping looks; within a look, larger values of the statistic are ordered as stronger evidence for the upper-sided alternative.

Consequently, an outcome that stops at an early look can have a p-value that differs substantially from the ordinary one-look tail probability. This is a property of the ordering, not evidence that a futility stop supports efficacy. Interpret the p-value together with the stopping decision and the design.

Model and Stopped-Outcome Definition

Consider a planned design with \(J\) analyses. At look \(j\), let \(Z_j\) be the standardized test statistic and let \(C_{j-1}\) denote the event that the trial reached look \(j\). Under the one-sided null hypothesis \(H_0\), assume the planned statistics have the specified joint null distribution. In the canonical group sequential model, they are jointly normal with mean zero, unit variances, and correlation \(\operatorname{corr}(Z_j,Z_k)=\sqrt{t_j/t_k}\) for \(j<k\), where \(t_j\) is the information fraction at look \(j\). The calculations also require that the boundaries and look schedule are fixed as specified, and that the statistic and null distribution used for calculation match the design.

At an interim look \(j<J\), suppose the design has a lower futility boundary \(l_j\) and an upper efficacy boundary \(u_j\), with \(l_j<u_j\). A path reaching that look stops if it crosses either boundary. Otherwise it continues. At the final look, every path reaching the analysis is terminal.

Definition: The stopping look is \(T\). At an interim look \(j<J\), the terminal event is \(T=j\) when \(C_{j-1}\) occurs and \(Z_j\le l_j\) or \(Z_j\ge u_j\). The continuation event at that look is \(C_{j-1}\) and \(l_j<Z_j<u_j\). At the final look, \(T=J\) for every path that reaches it.

For an observed futility outcome \((T=j,Z_j=z_{\mathrm{obs}})\), we have \(z_{\mathrm{obs}}\le l_j\), and no efficacy boundary was crossed at that look. Under the stagewise ordering, all outcomes stopping before \(j\) are at least as extreme in the ordering as the observed outcome. Among outcomes stopping at \(j\), those with \(Z_j\ge z_{\mathrm{obs}}\) are at least as extreme within that look. Outcomes stopping later are not included for this ordering. Thus the one-sided stagewise p-value is

p_{\mathrm{SW}}(j,z_{\mathrm{obs}}) = P_0(T<j) + P_0(T=j,\ Z_j\ge z_{\mathrm{obs}}).

The notation \(P_0\) denotes probability under \(H_0\). At an interim futility look, the same-look term must include only outcomes that stop at that look:

P_0(T=j,\ Z_j\ge z_{\mathrm{obs}}) = P_0\!\left( C_{j-1},\ Z_j\ge z_{\mathrm{obs}},\ \{Z_j\le l_j\ \text{or}\ Z_j\ge u_j\} \right).

At the final look, all paths reaching it stop, so the same-look term simplifies to \(P_0(C_{J-1},Z_J\ge z_{\mathrm{obs}})\). This distinction between an interim look and the final look is essential: an interim value between the futility and efficacy boundaries is not a stopped outcome.

Why the Probability Calculation Is Valid

A stagewise p-value is a null probability assigned to the set of stopped outcomes that are at least as extreme as the observation under the stated ordering. Its validity is a property of the ordering and the null distribution. To see the basic reason, arrange the possible stopped outcomes from most to least extreme. For each outcome, its p-value is the null probability accumulated up to and including that outcome. The null probability of outcomes assigned p-values no greater than a threshold \(a\) cannot exceed \(a\); if there are ties or a discrete outcome space, it can be smaller. Therefore,

P_0(p_{\mathrm{SW}}\le a)\le a,\qquad 0\le a\le1.

This is the superuniformity criterion for a valid p-value. In a continuous statistic model, the same principle applies to the probability accumulated along the ordered stopping stages and within-stage statistic values. It does not say that the p-value is the probability that \(H_0\) is true, nor that it measures the probability of futility.

Calculating a Futility-Stop P-Value

Worked Example: A Futility Stop at the First Look

Hypothetical design and observation. Consider a two-look design under the canonical null model. At the first look, set \(l_1=-1.2\) and \(u_1=2.5\). Suppose the observed statistic is \(z_{\mathrm{obs}}=-1.3\). It is a futility stop because \(-1.3\le-1.2\), and it is not an efficacy crossing. There are no earlier looks, so \(P_0(T<1)=0\).

The same-look contribution includes terminal values at least as large as \(-1.3\). Within the lower futility region, these are values from \(-1.3\) to \(-1.2\); upper-boundary crossings are also terminal and included. Values strictly between \(-1.2\) and \(2.5\) would continue and must be excluded. Under \(H_0\), \(Z_1\sim N(0,1)\), so

\begin{aligned} p_{\mathrm{SW}} &=P_0(-1.3\le Z_1\le-1.2)+P_0(Z_1\ge2.5)\\ &=\{\Phi(-1.2)-\Phi(-1.3)\}+\{1-\Phi(2.5)\}\\ &=(0.1151-0.0968)+(1-0.9938)\\ &=0.0183+0.0062\\ &\approx0.0245. \end{aligned}

The calculation uses the standard normal null distribution and the specified stopping boundaries. The result is a stagewise p-value for this stopped outcome, not a conclusion that the trial demonstrated efficacy. In particular, the observed statistic is below zero and the futility boundary; the p-value reflects the stagewise ordering, which also ranks outcomes at this early stopping look.

Final-Look Outcome After an Earlier Futility Opportunity

At the final look, the terminal region is different: every path that reached the final analysis stops, regardless of whether the statistic lies above or below a boundary. The p-value for a low final statistic therefore includes the probability of all earlier terminal outcomes, plus the null probability of reaching the final look with a statistic at least as large as the observed value.

Worked Example: A Low Statistic at the Final Look

Hypothetical design and observation. Suppose there are two looks, with \(t_1=0.5\) and \(t_2=1\). At look 1, let the futility and efficacy boundaries be \(l_1=-1.5\) and \(u_1=2.5\). Suppose the trial reaches look 2 and the observed final statistic is \(z_{\mathrm{obs}}=-1.30\). Under \(H_0\), \((Z_1,Z_2)\) is bivariate standard normal with correlation \(1/\sqrt{2}\).

The probability of stopping at look 1 is the probability of either boundary crossing:

\begin{aligned} P_0(T=1) &=P_0(Z_1\le-1.5)+P_0(Z_1\ge2.5)\\ &=\Phi(-1.5)+1-\Phi(2.5)\\ &=0.06681+0.00621\\ &=0.07302\approx0.0730. \end{aligned}

For the final-look term, the trial must have continued at look 1, so \(-1.5<Z_1<2.5\). Since look 2 is final, there is no additional terminal-region restriction on \(Z_2\). Conditional on \(Z_1=x\), the canonical model gives \(Z_2\) mean \(x/\sqrt{2}\) and variance \(1/2\). Therefore

\begin{aligned} P_0(T=2,Z_2\ge-1.30) &=P_0(-1.5<Z_1<2.5,\ Z_2\ge-1.30)\\ &=\int_{-1.5}^{2.5}\phi(x)\, \Phi\!\left(x+1.30\sqrt{2}\right)\,dx\\ &\approx0.8658. \end{aligned}

The integral is a bivariate normal probability; its integrand follows by standardizing \(Z_2\) conditional on \(Z_1=x\). Adding the earlier stopped outcomes gives

p_{\mathrm{SW}} = P_0(T=1)+P_0(T=2,Z_2\ge-1.30) \approx0.0730+0.8658 =0.9388.

Thus the stagewise p-value is approximately \(0.9388\). It is large, consistent with the low observed final statistic under a one-sided upper alternative. Notice that the earlier futility and efficacy stops both contribute to \(P_0(T<2)\), because both are earlier stopping outcomes in this stagewise ordering.

Diagnosing an Incorrect Tail Calculation

Worked Example: Separating Terminal and Continuing Paths

Hypothetical calculation. Use the first-look boundaries \(l_1=-1.5\), \(u_1=2.5\), and suppose the observed futility statistic is \(z_{\mathrm{obs}}=-1.8\). Since this is the first look, the stagewise p-value is the same-look terminal probability above \(-1.8\). The relevant pieces are the lower futility interval from \(-1.8\) to \(-1.5\) and the upper efficacy region from \(2.5\) upward:

\begin{aligned} p_{\mathrm{SW}} &=\{\Phi(-1.5)-\Phi(-1.8)\}+\{1-\Phi(2.5)\}\\ &=(0.06681-0.03593)+(1-0.99379)\\ &=0.03088+0.00621\\ &\approx0.03709. \end{aligned}

The ordinary normal upper tail \(P_0(Z_1\ge-1.8)\approx0.9641\) is not the same-look terminal contribution. It also includes continuation outcomes. In this example, the probability of continuing at the first look is

P_0(-1.5<Z_1<2.5) = \Phi(2.5)-\Phi(-1.5) = 0.99379-0.06681 = 0.92698.

Indeed, \(0.03709+0.92698\approx0.96407\), which recovers the ordinary upper tail. The calculation is a useful diagnostic: if the interim same-look term equals the unrestricted tail, check whether continuation paths were mistakenly counted as stopped outcomes.

Practical Checks, Comparisons, and Limitations

The main alternatives are not interchangeable. The ordinary upper-tail probability \(P_0(Z_j\ge z_{\mathrm{obs}})\) treats the statistic as if it were analyzed at that look without a stopping rule. At an interim look it can include continuation paths and omit the probability of earlier stopping. A final-look tail conditional on reaching the final look answers a conditional question, not the stagewise question defined here. The stagewise p-value uses the full null stopped-outcome distribution and the specified stagewise ordering.

Before reporting a result, check the following:

Confirm the observed stopping status. A futility observation must satisfy \(z_{\mathrm{obs}}\le l_j\), and the design must not also have crossed the efficacy boundary.
Apply the correct terminal set. At an interim look, restrict the same-look event to \(Z_j\le l_j\) or \(Z_j\ge u_j\). At the final look, all paths that reach the analysis are terminal.
Include earlier stops. The stagewise p-value includes \(P_0(T<j)\), calculated under the same null model and boundaries.
Check the joint distribution. For multiple looks, dependence between statistics matters. Marginal normal probabilities alone generally do not provide the probability of reaching a later look with a specified statistic.
Check probability-mass accounting. The probabilities of stopping at each look and reaching the next look should sum to one, up to numerical error.

Numerical integration can be sensitive to truncation, boundary conventions, and rounding. Keep enough precision in intermediate calculations, verify the probability of reaching each look, and report the final p-value to a sensible number of significant figures. If boundaries or timing were changed in response to unplanned information, the fixed-design null distribution may no longer describe the actual procedure.

Key takeaway: For an interim futility stop, the same-look stagewise tail is the probability of terminal outcomes at that look with \(Z_j\ge z_{\mathrm{obs}}\), not the unrestricted event \(Z_j\ge z_{\mathrm{obs}}\). Exclude values in the continuation region, add the probability of earlier stops, and use the full terminal tail only at the final look.

Check Your Understanding

Answer each question using the stagewise ordering and stopping definitions above. Brief solutions follow the questions.

At an interim look with \(l_j=-1.0\), \(u_j=2.0\), and \(z_{\mathrm{obs}}=-1.4\), which same-look terminal statistic values contribute to the p-value?
Why is \(P_0(C_{j-1},Z_j\ge z_{\mathrm{obs}})\) generally too large as the same-look contribution at an interim futility stop?
At the final look, why can the same-look term be written \(P_0(C_{J-1},Z_J\ge z_{\mathrm{obs}})\)?
For the first-look example with \(l_1=-1.5\), \(u_1=2.5\), and \(z_{\mathrm{obs}}=-1.8\), what is the approximate contribution from the lower futility interval alone?
What probability property should a correctly calibrated p-value satisfy under \(H_0\)?

Solutions. (1) Values in \([-1.4,-1.0]\) and values at least \(2.0\) contribute; values between \(-1.0\) and \(2.0\) continue and are excluded. (2) It includes values in the continuation region that do not stop at that look. (3) Every path reaching the final look is terminal, so no futility-or-efficacy restriction is needed there. (4) \(\Phi(-1.5)-\Phi(-1.8)=0.06681-0.03593\approx0.03088\). (5) Superuniformity: \(P_0(p\le a)\le a\) for every \(0\le a\le1\).