Why Sequential P-Values Need an Audit
After an interim analysis, the data available for inference are not simply the data that would have been observed at a fixed final sample size. The trial may already have stopped, and the observed stopping look is part of the outcome. A p-value calculated as if every trial had continued to its planned end can therefore answer a different question from the one intended.
The main diagnostic is to ask whether the calculation assigns probabilities to the same outcomes that the design could actually have produced. That requires the null model, the look schedule, the stopping boundaries, and the ordering used to define “at least as extreme” as the observed result. A sequential p-value is not a unique number until that ordering is specified.
For examples below, assume the canonical group-sequential normal model under \(H_0\): each \(Z_k\) has a standard normal marginal distribution, and for \(j\leq k\), \(\operatorname{Corr}(Z_j,Z_k)=\sqrt{I_j/I_k}\), where \(I_k\) is the information at look \(k\). Equivalently, standardized increments over nonoverlapping information intervals are independent under the null. The information levels and stopping rule are fixed in advance, the statistics follow this null model, and the ordering is specified before the p-value is calculated. These assumptions should be checked against the actual design; adaptations or different test-statistic distributions require corresponding null calibration.
Four Errors That Change the Answer
Using the terminal Z-test p-value. The fixed-sample calculation \(1-\Phi(Z_J)\) for a one-sided test uses the null distribution at the terminal look. It does not generally account for the opportunity to stop earlier after an unusually large statistic. If the trial stops early, \(Z_J\) may not even be observed. Even if the trial reaches the final look, a p-value based only on \(Z_J\) ignores the earlier statistics that helped determine whether the trial continued.
Conditioning without saying what is conditioned on. A conditional tail probability, such as \(P_0(Z_2\geq z\mid Z_1=x)\), describes future variation given a specified interim statistic. It is not automatically the unconditional p-value for the entire sequential experiment. Conditioning can be appropriate for a clearly stated conditional question, but the conditioning event and its role in the calibration must be explicit. In particular, a conditional probability given the stopping look is not interchangeable with an unconditional probability over all stopping looks.
Omitting a stopping stage. A calculation based only on the final look leaves out possible earlier stopping outcomes. That omission changes the sample space and, in general, the null distribution of the reported statistic. The p-value must include every way the design could have stopped that meets the chosen extremeness criterion, while excluding outcomes that could not have been observed after an earlier stop.
Applying the wrong two-sided ordering. “Two-sided” does not by itself specify how sequential outcomes are ordered. A maximum-absolute-statistic ordering, a two-sided stagewise ordering, and a likelihood-ratio ordering can produce different tail areas. A calculation that doubles a one-sided tail or uses a full-look maximum without accounting for stopping may not correspond to the chosen ordering or to the reachable outcomes.
Worked Examples: Trace the Reference Sample Space
Worked Example: A Terminal Z-Test After Reaching the Final Look
Suppose a hypothetical two-look trial has information levels \(I_1=100\) and \(I_2=200\), an upper efficacy boundary \(Z_1\geq2.0\), and otherwise continues to look 2. The observed statistics are \(Z_1=0.8\) and \(Z_2=2.2\), so the trial reaches the final look. For a one-sided maximum-statistic ordering, the observed threshold is \(2.2\). Find the null probability of an outcome at least this extreme under the actual stopping rule, and compare it with the terminal Z-test p-value.
The canonical correlation is \(\rho=\sqrt{100/200}=0.7071\). A path is extreme at the observed threshold if it crosses \(2.2\) at look 1, or if it does not stop at look 1 and reaches \(2.2\) at look 2. Thus
Using \(Z_2\mid Z_1=x\sim N(\rho x,1-\rho^2)\), this becomes
The terminal-only calculation is \(1-\Phi(2.2)\approx0.0139\). It includes only the first term in the sequential probability and omits qualifying look-2 outcomes among paths that continued. The integral is the additional probability for those paths. A valid report should evaluate this expression using the stated null model and stopping rule, rather than report \(0.0139\) as though it were the sequential p-value.
Worked Example: A Conditional Tail Is Not an Unconditional Sequential P-Value
In a hypothetical two-look trial with \(I_1=100\), \(I_2=200\), and the same upper boundary \(Z_1\geq2.0\), suppose the observed values are \(Z_1=1.2\) and \(Z_2=2.0\). Calculate the conditional probability of a final statistic at least \(2.0\), given \(Z_1=1.2\), and identify the corresponding unconditional sequential tail event under a one-sided maximum ordering.
Here \(\rho=0.7071\), so the conditional mean of \(Z_2\) is \(0.7071(1.2)=0.8485\), and its conditional standard deviation is \(\sqrt{1-0.7071^2}=0.7071\). Therefore,
This is a conditional future-tail probability for a fixed interim value. The unconditional sequential probability for the maximum-statistic threshold \(2.0\) instead includes early crossings and qualifying continuations:
The two calculations condition on different information and describe different probability statements. The value \(0.0518\) should not be presented as the unconditional sequential p-value. It could be reported as a conditional quantity only if that conditional target and its calibration are intended.
Worked Example: A Two-Sided Tail Must Exclude Unreachable Outcomes
Consider a hypothetical two-look trial with \(I_1=100\), \(I_2=200\), and an upper efficacy stop at look 1 when \(Z_1\geq2.0\). Otherwise, the trial continues. The observed path is \(Z_1=0.4\), \(Z_2=-2.4\). Under a maximum-absolute-statistic ordering, the observed threshold is \(c=\max(|0.4|,|-2.4|)=2.4\). Write the correctly calibrated two-sided p-value.
The observable outcome is extreme if \(|Z_1|\geq2.4\), or if the trial continues (\(Z_1<2.0\)) and the observed second-look statistic satisfies \(|Z_2|\geq2.4\). Hence
An equivalent complement makes the reachability restriction especially clear:
The two subtracted regions are disjoint. The first consists of look-1 stops that do not meet the observed absolute threshold; \(Z_2\) is unobserved on those paths. The second consists of continuations with neither observed statistic reaching the threshold. A full bivariate maximum calculation would incorrectly count some paths with \(2.0\leq Z_1<2.4\) as extreme because their hypothetical, unobserved \(Z_2\) is extreme. Those paths cannot contribute a look-2 statistic under this stopping rule.
A Short Justification and a Practical Audit
The event definition follows by partitioning the sample space according to the stopping look. Write \(A_k\) for the event that the design stops at look \(k\), and \(E_k\) for the event that the observed data at that look meet the chosen extremeness criterion. Since the stopping events are disjoint, the sequential tail probability is
This identity is the reason both omissions and unreachable paths matter. Leaving out a possible \(A_k\cap E_k\) removes probability mass from the tail. Evaluating a statistic after the process has stopped adds mass from outcomes that are not in the observed-data sample space. For a continuous, fully specified null model and ordering, the resulting tail-area p-value is calibrated under \(H_0\); for discrete outcomes, including ties in the tail generally makes the p-value conservative.
When reviewing a calculation, trace it in this order:
Identify the null hypothesis, whether the p-value is unconditional or conditional, and the ordering used to define extremeness.
List planned looks, information levels, and stopping boundaries. Confirm that the null distribution reflects the actual look timing and any design adaptations.
For each possible stopping stage, identify the outcomes that are both reachable under the rule and at least as extreme as observed.
Compare the calculation with fixed-look, conditional, and other ordering-based quantities only after labeling what each one measures. Agreement is not required; the probability statements must be.
The main alternatives are useful for different purposes but are not interchangeable:
| Quantity | What it conditions on or orders | Common reporting risk |
|---|---|---|
| Terminal Z-test p-value | One statistic at the planned final look | Ignores early stopping and earlier observed statistics |
| Conditional tail probability | Future data given a specified interim history | Misreported as an unconditional p-value |
| Sequential p-value | Outcomes reachable under the stopping rule, under a stated ordering | Incorrect if stages or reachable regions are omitted |
| Different sequential ordering | Same stopped data, but a different definition of extremeness | Presented as though all two-sided orderings give the same value |
Diagnostics, Limitations, and Common Mistakes
A p-value calculation deserves further checking if its formula refers to a statistic after the trial could have stopped, uses a terminal normal tail without describing sequential calibration, or conditions on the observed stopping look without explaining the conditional target. Likewise, a two-sided expression should state its ordering and make clear how paths that stop before a later look are treated. Numerical integration or simulation can check an implementation, but neither repairs an incorrectly defined probability event.
A common mistake is to reason, “The final observed \(Z\) is 2.2, so the p-value is \(1-\Phi(2.2)\).” The correction is to ask which outcomes the design could have produced under \(H_0\) that are at least as extreme under the selected ordering. That calculation must include earlier qualifying stops and qualifying continuations, but never use unobserved statistics from paths that already stopped.
There is no ordering-free sequential p-value. A maximum-statistic ordering is one explicit choice; stagewise and likelihood-ratio orderings are other choices and need not yield the same tail area. This tutorial’s examples use the maximum ordering only to make the reachable-event accounting explicit. The reported ordering and the design’s actual stopping rule remain essential limitations on interpretation.
Check Your Understanding
For each question, identify the relevant probability statement or correction.
- A trial stops for efficacy at look 1 when \(Z_1\geq2.1\), otherwise continues. Under a one-sided maximum ordering with observed threshold \(c=2.5\), write the tail event.
- Why is \(1-\Phi(Z_J)\) not generally a sequential p-value after a design allowed early stopping?
- A report gives \(P_0(Z_3\geq z\mid Z_1=x_1,Z_2=x_2)\) and labels it an unconditional sequential p-value. What is missing?
- In a two-sided analysis, why can \(P_0(\max(|Z_1|,|Z_2|)\geq c)\) be wrong when the trial can stop at look 1?
Solutions. (1) The event is \(\{Z_1\geq2.5\}\cup\{Z_1<2.1,\ Z_2\geq2.5\}\); the second-look statistic is counted only on paths that continue. (2) The terminal tail uses a fixed-look reference distribution and omits the effect of earlier opportunities to stop and the earlier statistics used by the stopping rule. (3) The report must state that it is conditional, identify the conditioning history and conditional target, and must not label it an unconditional probability over all stopped outcomes. (4) The full maximum can count an extreme, hypothetical \(Z_2\) on paths with \(Z_1\geq2.1\), where the trial already stopped and \(Z_2\) was never observed.