P-Values After an Early Efficacy Stop

Why an Early Efficacy Stop Needs a Stopped-Outcome p-Value

When a group sequential trial stops early for efficacy, the reported result is not just a test statistic observed at a planned final analysis. The stopping look is part of the observed outcome. A p-value must therefore be calculated from the null distribution of outcomes under the stopping rule and the ordering used to compare those outcomes with the one observed.

This tutorial uses the stagewise ordering: outcomes that stop at an earlier efficacy look rank ahead of outcomes stopping at later looks; among outcomes stopping at the same look, a larger one-sided \(Z\)-statistic is more extreme. Under this ordering, an early stop’s p-value includes the null probability of more extreme outcomes at its look and, if any, the probability of outcomes stopping at earlier looks. Later-look outcomes are not counted as more extreme merely because they would have a large statistic; they rank after an outcome that already stopped earlier.

Definition: Let \(T\) be the first look at which the efficacy stopping rule is met, and let \(Z_T\) be the one-sided test statistic at that look. If efficacy boundaries are \(b_1,\ldots,b_K\), stopping at look \(j\) means \(Z_j\ge b_j\) and \(Z_i<b_i\) for every earlier look \(i<j\). Under stagewise ordering, a stopped outcome is more extreme than the observed outcome \((T=k,Z_k=z)\) if it stops at a look \(j<k\), or if it stops at look \(k\) with \(Z_k\ge z\). The stagewise p-value is the null probability of this set of outcomes.

Model and Calculation

For the calculations below, assume a one-sided test with a prespecified, nonrandom look schedule and efficacy boundaries. Under \(H_0\), the standardized statistics have a joint multivariate normal distribution, with \(Z_j\sim N(0,1)\). For the canonical group sequential model, their correlations are \(\operatorname{Corr}(Z_i,Z_j)=\sqrt{t_i/t_j}\) when \(i<j\), where \(t_i\) and \(t_j\) are the information levels. The trial stops at the first efficacy-boundary crossing. The examples have no futility stopping or other data-dependent changes to the design.

For an observed stop at look \(k\), the stagewise p-value is the sum of probabilities over disjoint stopping stages: all efficacy stops at earlier looks, plus outcomes that survive to look \(k\) and have \(Z_k\ge z\). Writing \(T\) for the stopping look gives

p_{\mathrm{stage}}(k,z) = \sum_{j=1}^{k-1}P_0(T=j) + P_0(T=k,\ Z_k\ge z).

The subscript \(0\) indicates probability under \(H_0\). Each probability is computed with the stopping rule applied, so a term such as \(P_0(T=k,Z_k\ge z)\) includes the requirement that no earlier boundary was crossed. The sum is over mutually exclusive stopping stages; it is not a sum of marginal tail probabilities that may overlap.

At the first look, there are no earlier stopping stages. If the trial stops there with \(Z_1=z\), and all later outcomes rank below a first-look stop, the formula reduces to \(P_0(Z_1\ge z)\). At a later early look, the earlier-stop probabilities matter: those outcomes are ranked ahead of the observed one even if their statistics were only just above their boundaries.

Worked Calculations

Worked Example: Stop at the First Efficacy Look

Suppose a hypothetical trial has a first-look efficacy boundary \(b_1=2.00\). It stops at that look with \(Z_1=2.35\). Under \(H_0\), \(Z_1\sim N(0,1)\), and the later planned look ranks after a first-look stop under the stagewise ordering.

There are no earlier looks to contribute. The outcomes at the observed look that rank at least as extreme have \(Z_1\ge2.35\), so

p_{\mathrm{stage}}(1,2.35) =P_0(Z_1\ge2.35) =1-\Phi(2.35) \approx 0.009386.

The observed statistic exceeds the efficacy boundary, as required for this stop. The stagewise p-value is about 0.00939: under the null model and this ordering, about 0.939% of outcomes are at least as extreme as the observed stopped outcome. This is not the probability that \(H_0\) is true.

Worked Example: An Early Stop at the Second Look

Consider a hypothetical two-look trial with information levels \(t_1=0.5\) and \(t_2=1\). Under \(H_0\), the canonical model gives \(\operatorname{Corr}(Z_1,Z_2)=\sqrt{0.5}=0.7071\). Let the first-look boundary be \(b_1=2.00\), and suppose the trial survives that look and stops at look 2 with \(Z_2=2.00\), crossing a second-look boundary of \(1.90\).

By stagewise ordering, every outcome stopping at look 1 ranks ahead of this observed look-2 outcome. At look 2, outcomes at least as extreme have \(Z_2\ge2.00\), but they count only if the trial survived look 1. Therefore

p_{\mathrm{stage}}(2,2.00) =P_0(Z_1\ge2.00) +P_0(Z_1<2.00,\ Z_2\ge2.00).

The first term is \(1-\Phi(2)=0.0227501\). For the second term, subtract from the marginal final-look tail the outcomes that crossed the first boundary:

P_0(Z_1<2,\ Z_2\ge2) =P_0(Z_2\ge2)-P_0(Z_1\ge2,\ Z_2\ge2).

With correlation \(0.7071\), the bivariate-normal joint upper tail is approximately \(0.007449\). Thus the second term is \(0.0227501-0.007449\approx0.015301\), and

p_{\mathrm{stage}}(2,2.00) \approx0.0227501+0.015301 =0.038051.

The earlier-look contribution and the look-2 contribution describe disjoint stopping outcomes. The calculation also checks the design conditions: \(Z_1<2\) encodes survival to look 2, and \(Z_2=2.00\) exceeds the specified look-2 boundary of \(1.90\). The p-value is larger than the same-look marginal tail \(0.02275\) because the stagewise ordering counts all null outcomes that stopped at look 1 as more extreme.

Joint Probabilities, Ordering, and Double-Counting

Computing a probability involving multiple looks requires their joint null distribution. For two canonical looks with correlation \(\rho\), the conditional distribution is \(Z_2\mid Z_1=x\sim N(\rho x,1-\rho^2)\). For example, if \(\rho=1/\sqrt{2}\), then

P_0(Z_1\ge4,\ Z_2\le2) = \int_4^\infty \phi(x)\, \Phi\!\left(\frac{2-\rho x}{\sqrt{1-\rho^2}}\right)\,dx.

Here \(\phi\) and \(\Phi\) are the standard normal density and distribution functions. Substituting \(x=4+s\) gives

\phi(4)\int_0^\infty e^{-4s-s^2/2}\, \Phi(-1.1716-s)\,ds \approx 0.00000271.

The factor \(e^{-s^2/2}\) comes from the exact normal density ratio \(\phi(4+s)/\phi(4)\). This small value is a joint probability, not an extra stagewise contribution by itself: whether it belongs in a p-value depends on the stopped-outcome ordering and the event being evaluated.

Worked Example: Recognizing an Overlapping Tail Calculation

Use the same two-look null model. Let \(A=\{Z_1\ge2\}\) and \(B=\{Z_1\ge4,\ Z_2\le2\}\). The marginal probability of \(A\) is \(P_0(A)=1-\Phi(2)\approx0.0227501\). From the conditional-normal integral above, \(P_0(B)\approx0.00000271\).

Adding these probabilities gives \(0.0227501+0.00000271=0.0227528\). That arithmetic is correct as a sum, but it is not the exact probability of \(A\cup B\): \(B\) is a subset of \(A\), since \(Z_1\ge4\) implies \(Z_1\ge2\). Consequently \(P_0(A\cup B)=P_0(A)\approx0.0227501\). The sum \(0.0227528\) is only the loose union bound \(P_0(A\cup B)\le P_0(A)+P_0(B)\).

This is a useful numerical audit. A joint probability can be calculated accurately and still be added incorrectly. Before summing terms for a stagewise p-value, verify that the events are disjoint stopped-outcome categories, or explicitly subtract their overlap.

Derivation, Interpretation, and Diagnostics

The formula follows by partitioning the stagewise tail event according to the stopping look. For the observed outcome \((k,z)\), the tail contains (i) all outcomes stopping at any \(j<k\), and (ii) outcomes stopping at \(k\) with \(Z_k\ge z\). These sets are disjoint because a trial has only one first stopping look. Adding their null probabilities gives the stated formula. Outcomes stopping after \(k\) are not in this tail under the stagewise ordering, which places later stopping stages below the observed stage.

This construction differs from reporting the ordinary fixed-look quantity \(1-\Phi(z)\) at every analysis. At the first look, the two coincide under the stated model and ordering. At a later early stop, the ordinary tail omits the null probability of earlier efficacy stops, so it does not calculate the stagewise tail. Nor should one simply add marginal probabilities \(P_0(Z_j\ge z)\): the statistics across looks are correlated, and marginal exceedance events may overlap or include paths that would already have stopped.

When a calculation seems inconsistent, check the stopped-outcome partition before adjusting the arithmetic. Confirm that earlier boundary crossings are included exactly once, that the current-look tail also requires survival to that look, and that boundaries and one-sided direction match the design. For numerical integration, compare the sum of stopping-stage probabilities with the overall probability of stopping, and check that conditional normal means, variances, and tail directions are correct. A large discrepancy may indicate probability-mass leakage, a missed survival condition, or double-counting.

The result depends on the prespecified design, null model, and ordering. Changing the ordering can change the p-value even when the observed trial outcome is unchanged. The stagewise p-value is not a posterior probability for \(H_0\), and it does not quantify the effect size. It summarizes how much null probability is assigned to outcomes ranked at least as extreme under this particular stopped-outcome ordering.

Key takeaway: For an early efficacy stop at look \(k\), the stagewise p-value is the null probability of all outcomes stopping at earlier looks plus the null probability of outcomes surviving to look \(k\) and reaching at least the observed statistic there. Keep the stopping stages disjoint, and do not add later-look or overlapping marginal tails without a justification from the ordering.

Check Your Understanding

Answer each question using the stagewise ordering and assumptions stated above. Brief solutions follow.

A trial stops at its first look with \(Z_1=2.10\). What is the stagewise p-value formula under \(H_0\), and why is there no earlier-look term?
A trial stops at look 3. Which earlier outcomes contribute to its stagewise p-value?
At look 2, why must the event \(Z_2\ge z\) be paired with a survival condition on look 1?
In the overlapping-events example, what is the exact probability of \(A\cup B\), and why is the sum of marginal probabilities larger?

Solutions. (1) The p-value is \(P_0(Z_1\ge2.10)=1-\Phi(2.10)\approx0.01786\); no look precedes look 1. (2) All outcomes stopping at look 1 or look 2 contribute, regardless of their within-look statistic, because earlier stopping stages rank ahead of a look-3 outcome. (3) Without survival, the event includes paths that stopped earlier and therefore were not observed at look 2; the correct event is \(T=2,\ Z_2\ge z\). (4) Since \(B\subset A\), \(P_0(A\cup B)=P_0(A)\approx0.0227501\). Adding \(P_0(B)\) double-counts the portion of \(B\) already contained in \(A\).