From a Stopped Trial to a Sequential p-Value
A test statistic observed at the final look does not, by itself, describe the full evidence from a group sequential trial. The trial could have stopped earlier, and a sequential p-value must account for the stopping rule and for how stopped outcomes are ordered. This worked analysis follows one hypothetical trial from its observed stopping look through the null-tail calculation.
We consider a one-sided test of \(H_0:\theta=0\) against \(H_a:\theta>0\). At look \(i\), let \(Z_i\) be the standardized statistic and \(I_i\) the cumulative information. Assume the canonical normal model: under \(H_0\), each \(Z_i\) has a standard normal marginal distribution and, for \(i<j\), \( \operatorname{Corr}(Z_i,Z_j)=\sqrt{I_i/I_j} \). This model is appropriate when the information is cumulative, the statistic has the stated joint-normal distribution (exactly or as a suitable approximation), and the prespecified look schedule and stopping boundaries are followed. The calculations below use no futility stopping.
Design and Observed Outcome
Suppose the three information levels are \(I_1=0.40I_3\), \(I_2=0.70I_3\), and \(I_3\). The trial stops for efficacy at the first look with \(Z_i\ge c_i\), using boundaries \(c_1=2.80\), \(c_2=2.30\), and \(c_3=2.00\). Thus a trial reaching look 3 has \(Z_1<2.80\) and \(Z_2<2.30\). Under the null, the correlations are \( \rho_{12}=\sqrt{0.40/0.70}\approx0.7559 \), \( \rho_{13}=\sqrt{0.40}\approx0.6325 \), and \( \rho_{23}=\sqrt{0.70}\approx0.8367 \).
In the hypothetical observed trial, \(Z_1=1.10\), \(Z_2=1.85\), and \(Z_3=2.45\). The first two statistics do not cross their boundaries; the third crosses \(c_3=2.00\). The trial therefore stops at look 3 with observed statistic \(2.45\). The next decision is not to test \(2.45\) against a standard normal distribution and stop there. We must first state how outcomes at different looks are ranked.
This ordering makes a distinction that matters numerically. A look-2 path with \(2.30\le Z_2<2.45\) stops for efficacy under the design, but it does not enter the specified tail event: its statistic is below the observed value at that look. A path with \(Z_2\ge2.45\), by contrast, is included. The ordering must be stated before computing the p-value.
Worked Example: Identify the Stopping Stage and Tail Event
Use the observed values \(Z_1=1.10\), \(Z_2=1.85\), and \(Z_3=2.45\), and the boundaries \(2.80, 2.30, 2.00\). Determine the stopping stage and write the event whose null probability is the sequential p-value for the ordering above.
At look 1, \(1.10<2.80\), so the trial continues. At look 2, \(1.85<2.30\), so it continues again. At look 3, \(2.45\ge2.00\), so efficacy is declared and stopping occurs at look 3.
A null path is at least as extreme as this observed outcome if it crosses the first boundary, or reaches look 2 without a first-look crossing and has \(Z_2\ge2.45\), or reaches look 3 without crossing either earlier boundary and has \(Z_3\ge2.45\). The event is \( \{Z_1\ge2.80\}\cup\{Z_1<2.80,Z_2\ge2.45\}\cup\{Z_1<2.80,Z_2<2.30,Z_3\ge2.45\} \). The three events are disjoint, because each later event explicitly requires that the path did not stop at an earlier look.
Calculating the Null Tail Probability
Write \(\Phi\) and \(\phi\) for the standard normal cumulative distribution and density functions. The first contribution is a one-dimensional normal tail. For the second, the conditional distribution under \(H_0\) is \(Z_2\mid Z_1=z_1\sim N(\rho_{12}z_1,1-\rho_{12}^2)\). For the third, the joint distribution of the first two statistics conditional on \(Z_3=z_3\) is bivariate normal. Its conditional means and covariance can be obtained from the multivariate-normal conditioning formula. For example, \(E(Z_1\mid Z_3=z_3)=\rho_{13}z_3\) and \(E(Z_2\mid Z_3=z_3)=\rho_{23}z_3\).
The first contribution is \(1-\Phi(2.80)\approx0.002555\). For the second contribution, condition on \(Z_1\) and integrate its density over values that did not stop at look 1:
For the third contribution, integrate over \(z_3\ge2.45\), weighting by the conditional probability that neither earlier stopping condition was met. In symbols, with the conditional distribution determined by the correlation matrix above,
These integrals can be evaluated by numerical quadrature using the conditional normal distributions. Adding the disjoint contributions gives
The resulting sequential p-value is approximately \(0.0127\). Under the specified null model and ordering, this is the probability of a stopped outcome at least as extreme as the observed outcome. It is not the probability that \(H_0\) is true, nor the probability that the observed treatment effect is due to chance.
Worked Example: Why the Fixed-Look Tail Is Not the Sequential p-Value
At the observed look, the one-sided standard normal tail is \(1-\Phi(2.45)\approx0.00714\). Compare it with the sequential p-value and explain what each probability describes.
The value \(0.00714\) is the probability under a standard normal model that a single statistic at a prespecified look is at least \(2.45\). The sequential value \(0.0127\) includes null paths that were more extreme under the stated ordering at earlier looks, as well as the look-3 tail among paths that reached look 3.
The two answers differ because their events differ. Reporting \(0.00714\) as the sequential p-value would omit the earlier-look contributions \(0.002555+0.00635\) and would not represent the ordering used for the stopped trial.
Why the Calculation Is Calibrated
The basic justification is a partition argument. Under \(H_0\), every path in the specified tail event belongs to exactly one of the included stopping-stage regions: it either crosses the first boundary, reaches and qualifies at the second look, or reaches and qualifies at the third look. Disjointness means their probabilities add. Paths that stop at look 2 with \(2.30\le Z_2<2.45\) are deliberately outside this particular tail event because the stated ordering ranks them below the observed outcome.
Use the prespecified joint distribution of the statistics under \(H_0\), including their correlation from cumulative information.
Translate “at least as extreme” into inequalities at each possible stopping look, accounting for the earlier boundaries.
Compute each disjoint region’s null probability and sum them to obtain the sequential p-value.
For a prespecified ordering, a tail probability formed from the null distribution is a calibrated p-value: under \(H_0\), the probability that it is at most a chosen level \(a\) is no greater than \(a\). With a continuous outcome ranking and no ties, the null p-value is uniform; ties or discrete outcomes generally make it conservative. Calibration depends on using the actual look schedule, null model, stopping rule, and ordering. It does not follow merely from calling a number a sequential p-value.
As a quick numerical check, the second contribution cannot exceed \(P_0(Z_2\ge2.45)=1-\Phi(2.45)\approx0.00714\), and the third cannot exceed that same marginal tail. The calculated values \(0.00635\) and \(0.00380\) satisfy those bounds. The total is larger than the look-3 marginal tail, as expected here because earlier qualifying paths are added.
Diagnostics, Alternatives, and Limitations
A useful diagnostic is to check each region against its marginal probability and to verify that the regions are disjoint and match the verbal ordering. Numerical integration should also be checked by tightening the integration tolerance or comparing it with a simulation under the same joint-normal null model. Simulation is a check, not a substitute for stating the event: a simulation based on a different ordering or stopping rule will estimate a different probability.
The stagewise ordering used here is not the only possible ordering. A likelihood-ratio ordering ranks stopped outcomes by their likelihood evidence, while a conditional p-value conditions on the observed stopping look and uses a conditional reference distribution. These choices answer related but distinct questions and can give different numerical values. A fixed-look p-value instead treats the observed look as if it had been selected in advance; it does not generally account for the sequential path. The reporting practice is to name the ordering and the null probability being computed, not to present an ordering-dependent p-value as if it were unique.
This analysis relies on a one-sided normal model, known information fractions, the given boundaries, and the stated no-futility stopping rule. Model misspecification, unrecorded changes to the look schedule, or using a different stopping boundary invalidates the numerical calibration unless the null distribution is recomputed. The p-value also does not correct or interpret the treatment-effect estimate selected by stopping; estimation after data-dependent stopping is a separate issue.
Common Mistakes
- Using only \(1-\Phi(Z_3)\). This discards earlier qualifying paths. Include them whenever the chosen ordering places them in the tail event.
- Confusing the efficacy boundary with the ordering threshold. Here \(2.30\) is the look-2 stopping boundary, while \(2.45\) is the look-2 threshold in the specified comparison with the observed outcome. Replacing one by the other changes the event and the p-value.
- Adding overlapping probabilities. Write the earlier non-crossing conditions in every later-stage region before summing.
- Changing the ordering after seeing the result. The tail event must follow a stated ordering; selecting it after observing the statistics can compromise calibration.
- Calling the sequential p-value the probability the null is true. It is a probability calculated assuming the null model, not a posterior probability about the hypothesis.
Check Your Understanding
Answer each question using the boundaries and ordering stated above. Solutions follow each question.
- Suppose the observed statistics are \(Z_1=2.10\), \(Z_2=2.40\), and \(Z_3\) is not observed because the trial stops at look 2. Which stopping boundary is crossed, and does this look-2 outcome meet the specified comparison threshold of \(2.45\)?
Solution: The first look does not cross \(2.80\), and \(2.40\ge2.30\) crosses the second-look boundary, so stopping is at look 2. But \(2.40<2.45\), so this outcome does not meet the specified “at least as extreme as the observed look-3 outcome” condition at look 2. - Why does the look-3 contribution require both \(Z_1<2.80\) and \(Z_2<2.30\)?
Solution: The path can reach look 3 only if it did not cross either earlier efficacy boundary. Omitting either condition would include paths that had already stopped, causing the regions to overlap and misrepresenting the trial. - What would change if the ordering used \(Z_2\ge2.30\) rather than \(Z_2\ge2.45\) for the look-2 contribution?
Solution: The tail event would include additional paths with \(2.30\le Z_2<2.45\) that stop at look 2. Its probability, and therefore the sequential p-value, would increase. It would be a different ordering-based p-value. - Does the value \(0.0127\) mean there is a \(1.27\%\) probability that \(H_0\) is true?
Solution: No. It means that under \(H_0\), the specified joint-normal model, stopping rule, and ordering, the probability of an outcome at least as extreme as the observed stopped outcome is about \(1.27\%\).