Why Validity Depends on the Whole Stopped Outcome
A trial that may stop at an interim look does not produce just a final test statistic. Its observed outcome includes the look at which it stopped and the data available at that look. A stagewise p-value evaluates that complete outcome against the null distribution of outcomes from the same monitoring plan. The key validity fact is simple to state: the null probability of obtaining a p-value no larger than \(\alpha\) cannot exceed \(\alpha\).
This guarantee is a property of a fixed ordering of possible outcomes and a correctly specified null distribution. It does not require the ordering to be a likelihood ratio or any particular statistic. It does require that the ordering and stopping plan be set independently of the observed result, and that the null probability assigned to an outcome include the possible stopping looks.
For a numerical representation, let \(T(X)\) be an evidence score, with larger values defined in advance to be more extreme in the direction being tested. Then $$ p(x)=P_0\{T(X^*)\geq T(x)\}. $$ This expression is an unconditional probability over the stopped-trial outcome. It averages over the stopping looks according to their null probabilities; it is not generally the tail probability from a fixed sample size or from the observed look alone.
The Superuniformity Result
A valid p-value is superuniform under the null: for every \(0\leq\alpha\leq1\), $$ P_0\{p(X)\leq\alpha\}\leq\alpha. $$ The probability on the left is the chance, if the null model is true, of obtaining a result that would lead to rejection at level \(\alpha\). The inequality is the type I error guarantee. Equality is not required: discreteness and ties can make the procedure conservative.
Here is a justification that allows ties. Put \(S=T(X)\), and write \(q(s)=P_0(S\geq s)\). Since \(q(s)\) cannot increase as \(s\) becomes more extreme, the set of score values for which \(q(s)\leq\alpha\) is an upper tail. If this set has a least value \(s_0\), then $$ P_0\{q(S)\leq\alpha\}=P_0(S\geq s_0)=q(s_0)\leq\alpha. $$ If there is no least value, let \(c\) be the lower endpoint of that upper tail. Choose values \(s_n\) in the tail decreasing to \(c\). The event \(\{q(S)\leq\alpha\}\) is contained in \(\{S>c\}\), and $$ P_0(S>c)=\lim_{n\to\infty}P_0(S\geq s_n) =\lim_{n\to\infty}q(s_n)\leq\alpha. $$ Thus the inequality holds whether or not the score distribution has atoms. This argument uses the inclusive tail: for a score value with positive probability, its entire probability mass is counted among outcomes at least as extreme as that value.
If the null distribution of \(T(X)\) is continuous, there are no ties with positive probability. Let \(F\) be its null cumulative distribution function. Then \(p(X)=1-F(T(X))\), and the probability integral transform gives \(F(T(X))\sim\text{Uniform}(0,1)\). Consequently \(p(X)\) is also uniform under \(H_0\), and the superuniform inequality holds with equality for every \(\alpha\). Continuity here is an assumption about the distribution of the ordered score, not merely about the raw data. Continuous observations can still produce ties if the ordering maps them to the same score.
Worked Example: A Discrete Ordering Is Valid but Conservative
Suppose a hypothetical stopped trial has four possible evidence-score levels under \(H_0\). The probabilities include all possible stopping looks, and larger scores are more extreme.
| Score \(T\) | Null probability | Inclusive-tail p-value |
|---|---|---|
| 3 | 0.10 | 0.10 |
| 2 | 0.20 | 0.30 |
| 1 | 0.30 | 0.60 |
| 0 | 0.40 | 1.00 |
For example, if the observed score is 2, then $$ p=P_0(T\geq2)=0.10+0.20=0.30. $$ At nominal level \(\alpha=0.25\), only score 3 has a p-value no larger than 0.25, so the null rejection probability is \(0.10\), which is at most 0.25. At \(\alpha=0.30\), scores 3 and 2 qualify, giving rejection probability \(0.10+0.20=0.30\). At levels between 0.10 and 0.30, the rejection probability remains 0.10. It never exceeds the nominal level, but it does not increase smoothly because the possible p-values are discrete.
How Stopping Enters the Null Probability
For stagewise inference, \(X\) must represent the result after applying the stopping rule, not a hypothetical result from a fixed final sample size. For example, the possible outcomes might be partitioned by stopping look \(K\), with each component having its own data values. In that case the null tail probability sums or integrates the ordered outcomes over every possible \(K\), using the probability of each look under \(H_0\). The ordering determines which outcomes count as at least as extreme; the null model determines their probability.
Worked Example: A Continuous Score Across Two Possible Looks
Consider a hypothetical two-look monitoring plan under a simple null hypothesis. Suppose the trial stops at look 1 with probability 0.40 and reaches look 2 with probability 0.60. For illustration, assume the fixed ordering assigns a score \(T\) whose null distribution, conditional on either stopping look, is Uniform(0,1). This is a stipulated toy null model, not a general property of group sequential scores.
The unconditional score distribution is also Uniform(0,1), since for \(0\leq t\leq1\), $$ P_0(T\leq t)=0.40t+0.60t=t. $$ Thus an observed score \(t=0.82\), even if it occurred at look 2, has stagewise p-value $$ p=P_0(T\geq0.82)=1-0.82=0.18. $$ Across the complete monitored experiment, the null probability of \(p\leq0.05\) is \(P_0(T\geq0.95)=0.05\), and the probability of \(p\leq0.18\) is \(0.18\). The exact level guarantee follows from the unconditional distribution over both stopping looks. The conditional uniformity assumption in this example is deliberately strong; in an actual analysis, the score distribution must be derived or computed from the specified null model and stopping plan.
If a null hypothesis contains nuisance parameters, validity at one chosen parameter value is not automatically validity over the whole null hypothesis. A p-value calibrated at a particular null parameter has the stated guarantee at that parameter. To claim a guarantee for a composite null, the calibration must ensure superuniformity for every parameter value in the null, for example by using a valid worst-case null tail probability. Estimating a nuisance parameter and then treating it as known does not, by itself, establish the result.
Ties, Assumptions, and Practical Checks
When the score has atoms, the inclusive-tail p-value remains superuniform, but it is typically conservative. A tied score level contributes its full null probability to the tail. This is essential to the guarantee: excluding some outcomes tied with the observed outcome can reduce the reported tail probability and may lead to excess rejection. A randomized tie-break can recover a uniform null p-value in suitable discrete settings, but it adds external randomness and is a different reporting convention from the usual inclusive-tail p-value.
The result relies on a small set of conditions. The null law must describe the data-generating process under \(H_0\); the monitoring schedule and stopping rule must be included in that law; and the ordering must be fixed before seeing the outcome. The probability calculation must also be accurate enough for the claimed level. Numerical integration, truncation, or simulation error that understates a tail probability can undermine practical calibration even when the mathematical definition is valid.
Diagnostics should therefore examine whether probabilities over all stopping looks sum to one, whether the null tail probability decreases as outcomes become more extreme, and whether simulation under the null shows rejection rates at or below the nominal levels within Monte Carlo uncertainty. A mismatch between the implemented stopping rule and the planned rule, probability-mass leakage, or a score whose direction of extremeness is inconsistent across looks are signs that the computed p-values may not correspond to the intended ordering.
Worked Example: Why Inclusive Ties Matter
Return to the discrete null distribution in the first example. Suppose an observed outcome has score 1. The standard inclusive-tail p-value is $$ P_0(T\geq1)=0.10+0.20+0.30=0.60. $$ If someone instead reports only the probability of a strictly larger score, the value is \(P_0(T>1)=0.10+0.20=0.30\). At level 0.30, this strict-tail rule would reject for observed scores 1, 2, or 3, with null probability $$ 0.30+0.20+0.10=0.60, $$ which is twice the nominal level. The mistake is excluding the observed tied score level from the tail while still treating the result as an ordinary p-value. Including the tie gives the 0.60 value and preserves the superuniform guarantee.
What the Guarantee Does—and Does Not—Say
A stagewise p-value is valid because its tail probability is calculated under the null distribution of the same stopped experiment and against a fixed ordering. This differs from simply applying a final-look fixed-sample tail formula to data that may have stopped early. It also differs from a probability calculated conditional on the observed stopping look: conditioning changes the probability law and the inferential target, so the unconditional superuniformity proof above does not automatically establish validity for a conditional calculation.
The guarantee does not say that the null hypothesis has probability \(p\), nor that the parameter lies in a particular range with probability \(1-p\). It says that, under the null model, the chance of a p-value at or below a chosen threshold is no greater than that threshold. It also does not protect against an ordering chosen after inspecting the observed results unless that selection is incorporated into the null calibration. A data-driven ordering can make the observed outcome look unusually extreme by construction.
Check Your Understanding
Answer each question, then compare with the solutions below.
- Under a null distribution, an ordered score takes values 0, 1, and 2 with probabilities 0.50, 0.30, and 0.20, respectively. What is the inclusive-tail p-value for an observed score of 1?
- In that same example, what is the null rejection probability at level 0.25?
- Why is continuity of the raw observations not enough to guarantee exact uniformity of an ordering-based p-value?
- What must be included in the null distribution used for a stagewise p-value?
- Does a p-value calibrated at one nuisance-parameter value necessarily control type I error over a composite null? Explain.
Solutions. (1) \(P_0(T\geq1)=0.30+0.20=0.50\). (2) Only score 2 has a p-value no larger than 0.25, so the rejection probability is 0.20, at most 0.25. (3) The ordering may map different continuous observations to the same score, creating ties and atoms in the score distribution. Exact uniformity requires a continuous null distribution for the ordered score, not just for the raw data. (4) It must include the stopping rule and the probabilities of all possible stopping looks, as well as the outcomes and score ordering at those looks. (5) No. Validity at one null parameter value does not establish validity at all values in the composite null; the calibration must be valid throughout that null set.